How New York Times produces faster, smarter analysis

Shane Murray, vice president/data analytics and platforms at The New York Times, gave the audience at the 2017 Big Data for Media Week conference in London a snapshot of how the media company revolutionised its marketing models using Big Data and algorithms.

“Data technology is foundational to the success of today’s digital leaders. The media landscape is transforming and data is instrumental,” he said.

Murray posed a couple of questions to show the thought process that went into how his team could make every operation within The New York Times work faster. He asked: “How can we balance things like original content, original ideas, editorial curation, and even innovative design?”

He explained that from the start of 2016, The New York Times merged its data analytics team of about 24 people with the data science team of five people and with the data engineering team of about 25 people: “Partly bringing those teams together was the No. 1 step to success.”

Murray showed the audience the hierarchy of data and data needs that The New York Times needed to address:

1. Access, accuracy, and reliability: “Without these, it’s nearly impossible to effectively deliver on the higher order needs and attempting to solve for all needs all at once, may result in a state close to paralysis.”
2. A bottom-up approach: “Identify the first order problems that must be solved, then move up the pyramid. We make sure we don’t get unclean data into the system, removing bots and really reducing unnecessarily analysis for our team.”
3. Organising analyst and data science teams: “Enables faster work across disciplines, with editors, product managers, and designers, developers, marketers. This helps facilitate data-hungry people across the company.”

Murray explained that The New York Times migrated from a very batch-based environment to a streaming data environment. He then gave insight into how The New York Times has built and used ABRA, an experiment platform. Since late 2015, The Times has used ABRA “for experiments so we can feed a consistent stream of data into our analytics platform.”

How does ABRA work in practice?

1. Factorial designs.
2. Simple configuration .
3. Bayesian methodology.
4. Customised reporting for end-users.

“Typical statistics around experiments are judged once only to see the success or lack of success. But a Bayesian look at the more frequent results in a more eloquent fashion,” he said.

Murray explained that customised reporting for end-users is a new development that has just started in recent weeks: “We are starting to build up very customised interfaces so people can see the results that matter to them.”

What are the next steps in creating even more innovation?

1. Access to a fast, reliable data environment is paramount: “This really was ground-breaking for us how much faster we could move and how much faster it made our team.”
2. This foundation paves the way for innovation and development of data products,
3. In turn, teams can deliver faster, smarter analysis: “It freed up time for deeper questions and not spend their time compiling dashboard and maintaining them.”

Murray finished the talk by describing the optimal environment that The New York Times strives for: “Faster. Happier. More productive. This is the fastest analytical environment I’ve worked with.”

Ella Wilks-Harper is an interactive journalism MA student at City, University of London. She can be reached at ellawilksharper@gmail.com