Dow Jones Data News and Analytics (DNA)

By Scott M. Scher, Columbia University

At Dow Jones, the data team focuses on ensuring access to news and data and their goal is to leverage the cloud to accelerate innovation. They have moved their data archive to Google cloud and created a platform-agnostic solution allowing any enterprise client can facilitate text mining scenarios and AI analytics to innovate the way they look at their data. The idea behind DNA is to have a flexible solution that can handle any use case that arises. The main scenarios they have prepared for are: text mining, data argumentation, data integration, data extraction, data enrichment, and application development. DNA uses APIs and streams for all these scenarios.

Using data flow and BigQuery, DNA is able to index and warehouse a full archive of every data point they collect. According to Walsh, the biggest value ads of DNA is that they guarantee compliance with data rights. They validate that if they provide an enterprise with a snapshot data stream, that Dow Jones has the right to redistribute. This validated that DNA has trustworthy and usable data for its clients.

By using Google hub, Dow Jones are able to take a query and create deliverable real-time news streams. Walsh gave the example of an event planner having streams set up that will allow her to anticipate weather changes and be able to make decisions accordingly and react accordingly. This enables a user to use Dow Jones data with any platform to do back testing or historical analysis, predictive modeling to do things like trade and compliance, derived data sets, and weather modeling. The audience was walked through a use case DNA did for insurance risk due to hurricanes. They were able to figure out the impact on contractors, traveling, and tourism hits due to hurricanes. This data set and real-time news allowed them to generate a knowledge graph on the risk of hurricanes and their associated risks. Dow Jones now uses AI and machine learning with their coded data to identify potential risk. This has allowed them to identify crime, persons of interest, and adverse media. They have done the deep research with AI and in doing so are able to accept raw articles that can be coded and used for risk and compliance. This has resulted in a huge increase in this business, she said.

Walsh concluded with a few key lessons: moving to the cloud is not as easy as “lift and shift”, the truth is an organization will live in a hybrid state for longer than you realize and longer than you wanted to. The biggest value they have gained from living on the cloud has been the ability to experiment with new architecture with out disbanding existing architecture.