Step 1. Choose the Right Questions for the Right Team

An important strategy for success is to choose the right questions for the right team. One data scientist described three conditions that must be met before his data science team engages in a project: priority, actionability, and commitment.

“(a) Is it a priority for the organization (b) is it actionable, if I get an answer to this, is this something someone can do something with? and, (c), are you, as the feature team—if you're coming to me or if I'm going to you, telling you this is a good opportunity—are you committing resources to deliver a change? If those things are not true, then it's not worth us talking anymore.”

It is important to define actions in addition to generating insights from data.

You need to think about, “If you find this anomaly, then what?” Just finding an anomaly is not very actionable. What I do also involves thinking, “These are the anomalies I want them to detect. Based on these anomalies, I'm going to stop the build. I'm going to communicate to the customer and ask them to fix something on their side.”

For data scientists, there are many more questions to pursue than they have time and resources for. So it is important for them to choose questions that enable their stakeholders to achieve their goals.

Step 2. Work Closely With Your Consumers

Another strategy that was mentioned in several interviews was to interact closely and engage with the stakeholders who plan to consume the results from the data analysis. Data scientists often set up channels such as weekly data meet-ups and brown bag lunches to regularly deliver project outcomes. It is important for stakeholders to define the questions and scenarios addressed by data analysis early and often. To make their data science work adopted by stakeholders, data scientists iterate with them to interpret the data and refine important questions and scenarios.

“You begin to find out, you begin to ask questions, you begin to see things. And so you need that interaction with the people that own the code, if you will, or the feature, to be able to learn together as you go and refine your questions and refine your answers to get to the ultimate insights that you need.”

Step 3. Validate and Calibrate Your Data

The validation of quantitative data through qualitative channels was also mentioned as a criteria to ensure that measurements are meaningful and lead to correct actions.

“If you could survey everybody every ten minutes, you wouldn't need telemetry. The most accurate way to derive insights is to ask everybody all the time. The only reason we do telemetry is that (asking people all the time) it is slow, and by the time you get it, you're too late. So you can consider telemetry and data an optimization. So what we do typically, is 10% are surveyed and we get telemetry. And then we calibrate and infer what the other 90% have said.”

It is important for data scientists to triangulate multiple data resources to increase the confidence in their analysis results.

Step 4. Speak Plainly to Give Results Business Value

Many data scientists emphasize the need to explain findings in simple terms to non-experts. Talking to non-experts also requires the development of intuitive measurements. Data scientists need to “translate” findings into business values, such as dollars saved, customer calls prevented, and number of days that a product can be shipped early. Reporting precision, recall, and ROC curves, while convenient for the data scientists, are of less importance when presenting findings to analytics consumers.

Some data scientists argue that producing fancy results is bad, because fancy requires them to be there to explain in order for the person reading the results to figure out what's going on. So, it is better to produce easy-to-understand results than statistically valid, yet complex, results.

Step 5. Go the Last Mile—Operationalizing Predictive Models

A strategy that we noticed in several interviews was the willingness to go the last mile to actively contribute to the operationalization of predictive models.

“They accepted (the model) and they understood all the results and they were very excited about it. Then, there's a phase that comes in where the actual model has to go into production… You really need to have somebody who is confident enough to take this from a dev side of things.”

Many stakeholders want to deploy predictive models as a part of the product, so it is important to produce working software that leverages and integrates the predictive models. Producing accurate models that are not integrated or instantiated as new software features is not useful for the stakeholders. This implies that data scientists must be not only scientists who draw new findings, but also engineers who build systems based on their findings.