Outliers caused by outside forces

Be sure you check carefully for outliers before they influence your analysis. Outliers can distort both the data and data analysis. For example, any statistical analysis done with data that leaves outliers in place ends up skewing the means and variances. Some techniques, such as Trees, are more robust to outliers than others.

Unchecked or misinterpreted outliers may lead to false conclusions. Say your data that shows that a stock that was traded for a whole year at a price above $50 — but for only a few minutes out of that whole year the stock was traded at $20. The $20 price — an obvious exception — is the outlier in this dataset.

Now you have to decide whether to include the $20 stock price in your analysis; if you do, it has ramifications for the overall model. But what do you consider normal? Was the “flash crash” that took the stock market by surprise on May 6, 2010 a normal event or an exception? During that brief time, the stock market experienced a sharp decline in prices across the board — which knocked the sample stock price down from $50 to $20, but had less to do with the stock than with wider market conditions. Does your model need to take the larger fluctuations of the stock market into account?

Anyone who's lost money on brief moments of free-fall market considers those few minutes real and normal (even if they felt like an eternity to go through). A portfolio that diminishes in milliseconds due to a rapid decline, albeit short-lived, is clearly real. Yet the flash crash is an anomaly, an outlier that poses a problem for the model.

Regardless of what's considered normal (which can change anyway), data sometimes contains values that don't fit the expected values. This is especially true in the stock market, where virtually any event may send the market flying or plunging. You don't want your model to fail when the reality changes suddenly — but a model and a reality are two different things. As a data scientist, you should strive to build models that account for the unexpected and be able to address it in a way that strengthens the business.

Outliers caused by errors in the system

When we rely on technology or instrumentation to conduct a task, a glitch here or there can cause these instruments to register extreme or unusual values. If sensors register observational values that fail to meet basic quality-control standards, they can produce real disruptions that are reflected in data.

Someone performing data entry, for example, can easily add an extra 0 at the end of a value by mistake, taking the entry out of range and producing an outlier. If you're looking at observational data collected by a water sensor installed in Baltimore Harbor — and it reports a water height of 20 feet above mean sea level — you've got an outlier. The sensor is obviously wrong unless Baltimore is completely covered by water.

Data can end up having outliers because of external events or an error by a person or an instrument. If a real event such as a flash crash is traced to an error in the system, its consequences are still real — but if you know the source of the problem, you may conclude that a flaw in the data, not your model, was to blame if your model didn't predict the event. Such problems can be addressed outside the model, such as instituting risk management for your portfolio in case of financial data.

Knowing the source of the outlier will guide your decision on how to deal with it. Outliers that were the result of data-entry errors can easily be corrected after consulting the data source. Outliers that reflect a changed reality may prompt you to change your model.

There's no one-size-fits-all answer when you're deciding whether to include or disregard extreme data that isn't an error or glitch. Your response depends on the nature of the analysis you're doing — and on the type of the model you're building. In a few cases, the way to deal with those outliers is straightforward:

• If you trace your outlier to a data-entry error when you consult the data source, you can easily correct the data and (probably) keep the model intact.
• If that water sensor in Baltimore Harbor reports a water height of 20 feet above mean sea level, and you're in Baltimore, look out your window:
  • If Baltimore isn't completely covered by water, the sensor is obviously wrong.
  • If you see a fish looking in at you, the reality has changed; you may have to revise your model.
• The flash crash may have been a one-time event (over the short term, anyway), but its effects were real — and if you've studied the market over the longer term, you know that something similar may happen again. If your business is in finance and you deal with the stock market all the time, you want your model to account for such aberrations.

In general, if the outcome of an event normally considered an outlier can have a significant impact on your business, consider how to deal with those events in your analysis. Keep these general points in mind about outliers:

• The smaller dataset is, the more significant the impact outliers can have on the analysis.
• As you develop your model, be sure you also develop techniques to find outliers and to systematically understand their impact on your business.
• Detecting outliers can be a complex process; there is no simple way of identifying them.
• A domain expert (someone who knows the field you're modeling) is your best go-to person to verify whether a data point is valid, an outlier you can disregard, or an outlier you have to take into account. The domain expert should be able to explain what factors created the outlier, what its range of variability is, and its impact on the business.
• Visualization tools can help you spot outliers in the data. Also, if you know the expected range of values you can easily query for data that falls outside that range.

Keeping the outliers in the analysis — or not

Deciding to include outliers in the analysis — or to exclude them — will have implications for your model.

Keeping outliers as part of the data in your analysis may lead to a model that isn't applicable — either to the outliers or to the rest of the data. If you decide to keep an outlier, you'll need to choose techniques and statistical methods that excel at handling outliers without influencing the analysis. One such technique is to use mathematical functions such as natural algorithms and square root to reduce the gap between the outliers and the rest of the data. These functions, however, only work for numerical data that is greater than zero — and other issues may arise. For example, transforming the data may require interpretations of the relationship between variables in the newly transformed data that differ from the interpretation that governs those variables in the original data.

The mere presence of outliers in your data can provide insights into your business that can be very helpful in generating a robust model. Outliers may draw attention to a valid business case that illustrates an unusual but significant event.

Looking for outliers, identifying them, and assessing their impact should be part of data analysis and preprocessing. Business domain experts can provide insight and help you decide what to do with unusual cases in your analysis. Although sometimes common sense is all you need to deal with outliers, often it's helpful to ask someone who knows the ropes.

If you're in a business that benefits from rare events — say, an astronomical observatory with a grant to study Earth-orbit-crossing asteroids — you're more interested in the outliers than in the bulk of the data.

Outliers can be a great source of information. Deviating from the norm could be a signal of suspicious activity, breaking news, or an opportunistic or catastrophic event. You may need to develop models that help you identify outliers and asses the risks they signify.

It's prudent to conduct two analyses: one that includes outliers, and another that omits them. Then examine the differences, try to understand the implications of each method, and assess how adopting one method over the other would influence your business goals.

Turning down the noise

Data smoothing operates on several assumptions:

• That fluctuation in data is likeliest to be noise.
• That the noisy part of the data is of short duration.
• That the data's fluctuation, regardless of how varied it may be, won't affect the underlying trends represented by the core data points.

Noise in data tends to be random; its fluctuations shouldn't affect the overall trends drawn from examining the rest of the data. So reducing or eliminating noisy data points can clarify real trends and patterns in the data — in effect, improving the data's “signal-to-noise ratio.”

Provided you've identified the noise correctly and then reduced it, data smoothing can help you predict the next observed data point simply by following the major trends you've detected within the data. Data smoothing concerns itself with the majority of the data points, their positions in a graph, and what the resulting patterns predict about the general trend of (say) a stock price, whether its general direction is up, down, or sideways. This technique won't accurately predict the exact price of the next trade for a given stock — but predicting a general trend can yield more powerful insights than knowing the actual price or its fluctuations.

A forecast based on a general trend deduced from smoothed data assumes that whatever direction the data has followed thus far will continue into the future in a way consistent with the trend. In the stock market, for example, past performance is no definite indication of future performance, but it certainly can be a general guide to future movement of the stock price.

Methods, advantages, and downsides of data smoothing

Data smoothing focuses on establishing a fundamental direction for the core data points by

• Ignoring any noisy data points
• Drawing a smoother curve through the data points that skips the wriggling ones and emphasizes primary patterns — trends — in the data, no matter how slow their emergence

In a numerical time series, data smoothing serves as a form of filtering.

Data smoothing is not be confused with fitting a model, which is part of the data analysis consisting of two steps:

1. Find a suitable model that represents the data.

2. Make sure that the model fits the data effectively.

   For details of the model-fitting process, see Chapter 12.

Data smoothing can use any of the following methods:

• Random walk is based on the idea that the next outcome, or future data point, is a random deviation from the last known, or present, data point.
• Moving average is a running average of consecutive, equally spaced periods. An example would the calculation of a 200-day moving average of a stock price.
• Exponential smoothing assigns exponentially more weight, or importance, to recent data points than to older data points.
  • Simple: This method should be used when the time series data has no trend and no seasonality.
  • Linear: This method should be used when the time series data has a trend line.
  • Seasonal: This method should be used when the time series data has no trend but seasonality.

What these smoothing methods all have in common is that they carry out some kind of averaging process on several data points. Such averaging of adjacent data points is the essential way to zero in on underlying trends or patterns.

The advantages of data smoothing are

• It's easy to implement.
• It helps identify trends.
• It helps expose patterns in the data.
• It eliminates data points that you've decided aren't of interest.
• It helps predict the general direction of the next observed data points.
• It generates nice smooth graphs.

But everything has a downside. The disadvantages of data smoothing are

• It may eliminate valid data points that result from extreme events.
• It may lead to inaccurate predictions if the test data has only one season and isn't fully representative of the reality that generated the data points.
• It may shift or skew the data, especially the peaks, resulting in a distorted picture of what's going on.
• It may be vulnerable to significant disruption from outliers within the data.
• It may result in a major deviation from the original data.

If data smoothing does no more than give the data a mere facelift, it can lead to a fundamentally wrong path in the following ways:

• It can introduce errors through distortions that treat the smoothed data as if it were identical to the original data.
• It can skew interpretation by ignoring — and hiding — risks embedded within the data.
• It can lead to a loss of detail within your data — which is one way that a smoothed curve may deviate greatly from that of the original data.

How seriously data smoothing may affect your data depends on the nature of the data at hand, and which smoothing technique was implemented on that data. For example, if the original data has more peaks in it, then data smoothing will lead to major shifting of those peaks in the smoothed graphs — most likely a distortion.

Here are some cautionary points to keep in mind as you approach data smoothing:

• It's a good idea to compare smoothed graphs to untouched graphs that plot the original data.
• Data points removed during data smoothing may not be noise; they could be valid data points that result from rare, but real, events.
• Data smoothing can be helpful in moderation, but its overuse can lead to a misrepresentation of your data.

By applying your professional judgment and your business knowledge expertise, you can use data smoothing effectively. Removing noise from your data — without negatively affecting the accuracy and usefulness of the original data — is at least as much an art as a science.

The woes of overfitting

In essence, overfitting a model is what happens when you train the model to represent only your sample data — which isn't a good representation of the data as a whole. Without a more realistic dataset to go on, the model can then be plagued with errors and risks when it goes operational — and the consequences to your business can be serious.

Overfitting a model is a common trap because people want to create models that work — and so are tempted to keep tweaking variables and parameters until the model performs perfectly — on too little data. To err is human. Fortunately, it's also human to create realistic solutions.

To avoid overfitting your model to your sample dataset, be sure to have a body of test data available that's separate from your sample data. Then you can measure the performance of your model independently before making the model operational. Thus one general safeguard against overfitting is to divide your data to two parts: training data and test data. The model's performance against the test data will tell you a lot about whether the model is ready for the real world.

Another best practice is to make sure that your data represents the larger population of the domain you're modeling for. All that an overtrained model knows is the specific features of the sample dataset it's trained for. If you train the model only on (say) snowshoe sales in winter, don't be surprised if it fails miserably when it's run again on data from any other season.

Avoiding overfitting

It's worth repeating: Too much tweaking of the model is apt to result in overfitting. One such tweak is including too many variables in the analysis. Keep those variables to a minimum. Start by including variables that you see as absolutely necessary — those you believe will make a significant difference to the outcome. This insight comes from intimate knowledge of the business domain you're in. That's where the expertise of domain experts can help keep you from falling into the trap of overfitting. In addition, we can obtain such insight through the modeling process; the data more often than not manages to surprise us.

Here's a checklist of best practices to help you avoid overfitting your model:

• Chose a dataset to work with that is representative of the population as a whole.
• Divide your dataset into two parts: training data and test data.
• Keep the variables analyzed that have predictive value.
• Enlist the help of domain knowledge experts.

In the stock market, for example, a classic analytical technique is back-testing — running a model against historical data to look for the best trading strategy. Suppose that, after running his new model against data generated by a recent bull market, and tweaking the number of variables used in his analysis, the analyst creates what looks like an optimal trading strategy — one that would yield the highest returns if he could go back and trade only during the year that produced the test data. Unfortunately, he can't. If he tries to apply that model in a current bear market, look out below: He'll incur losses by applying a model too optimized for a narrow period of time and set of conditions that doesn't fit current realities. (So much for hypothetical profits.) The model worked only for that vanished bull market because it was overtrained, bearing the earmarks of the context that produced the sample data — complete with its specifics, outliers, and shortcomings. All the circumstances surrounding that dataset probably won't be repeated in the future, or in a true representation of the whole population — but they all showed up in the overfitted model.

If a model's output is too accurate, consider that a hint to take a closer look. Enlist the help of domain knowledge experts to see whether your results really are too good to be true, and run that model on more test data for further comparisons.

Testing and evaluating your model

No model can produce 100-percent accurate forecasts; any model has the potential to produce inaccurate results. Be on the lookout for any significant variation between the forecasts your model produces and the observed data — especially if the model's outputs contradict common sense. If it looks too good, bad, or extreme to be true, then it probably isn't true (to reality, anyway).

In the evaluation process, thoroughly examine the outputs of the models you're testing and compare them to the input variables. Your model's forecast capability should answer all stated business goals that drove its creation in the first place.

If errors or biases crop up in your model's output, try tracing them back to

• The validity, reliability, and relative seasonality of the data
• Assumptions used in the model
• Variables that were included or excluded in the analysis

The preceding factors will directly influence the accuracy of your model. If your data isn't appropriate for the model you're building, then your model is bound to fail to answer its business needs. If the data is good but flawed, like absent representation of a type of data due to seasonality, or that the data isn't valid or reliable, your data will not account for all cases that the model should be aware of and address when similar situations arise in the future.

As part of building the model, the data scientists or business stakeholders may make some assumptions either about the data used to create the model, or the business environment in which the model will run. If those assumptions are wrong, the model will be less or inaccurate.

The decision to include a variable or exclude it from the analysis has a direct impact on the outcome of the model. Some variables may only be effective and their predictive powers may only come in play in the presence of other variables. The decision to whether include or exclude a variable from an analysis is at the core of building predictive analytics models. Business stakeholders, data scientists’ experience, tools, and quality data should be of tremendous help in building successful predictive analytics models.

Work with business users to evaluate every step of your model's process; make sure that the model outputs can be easily interpreted and used in a real-world business situation. Balance the accuracy and reliability of the model with how easily the model's outputs can be interpreted and put to practical use.

The ability to easily interpret the model and for that interpretation to make sense to the business stakeholders is essential. That interpretation may be new, but you should be able to explain why this variable, or that variable, or a combination of variable will allow your business to make an accurate prediction.