What are Performance Variables?

Training records are passed through an analytics algorithm to build a model. Each record is made up of input variables. The model is then deployed and new records are passed so the model can provide a resultant. These records have the same input variables as the training set. Figure 4.1 is the example used in Chapter 1 where we had four input variables—age, income, student, and credit rating—and the predictive model was supposed to learn the relative impact of each input variable onto the prediction about a computer purchase.

The classification algorithm (decision trees) used in this case built a tree with the input variables with highest discriminatory power at the top of the tree and then subsequent variables in branches underneath, leading to a leaf of the tree that has a Yes or No as shown in Figure 4.2.

The data (input variables and historical records) is coming from a sales system that tracks the sales of computers and also records some customer information. There is a limit to how many types of variables are stored in the system. Even if the system is designed to handle a lot of information on a sales record, it is usually not properly filled in. Therefore, there can be a situation where the decision tree cannot be built. There are two possible reasons for this:

The first reason occurs when a sales system doesn’t track the lost sales or the customers who didn’t buy. If there are no records with the outcome No (the 0 records), it is not possible for the model to be trained for No and it cannot differentiate between Yes and No. Another case could be that the relative proportion of No is so small that it is not enough to build patterns against the dominant Yes (the 1 records) outcomes.

The second reason occurs when input variables are the same values for both 1 and 0 records. For example, 60–70% of the data for both 1 and 0 records of the data set have the same values for age range, or most of the customers have their input variable as Male for gender regardless if they bought or not. In these cases, the algorithm will discard age and gender as input variables because they have low discriminatory value. Here is a detailed example to explain the reasons why performance variables are essential for a model to perform well.

Designing Performance Variables

The design of performance variables uses the following techniques to convert detailed data into useful performance variables. These techniques can be used in any of the four analytics methods we have covered in the book.

Working Example

As described in Chapter 3, the art and science of analytics come together at the performance variable level. To understand their impact on the performance of an analytics model we have to be clear on the definition of grain. Analytics algorithms may be K-means algorithms for clustering, regression or neural network algorithms for prediction, or optimization or forecasting algorithms. They work with millions and millions of records at the same grain, meaning all records represent the same entity. For example, one record represents one customer or one record represents one account. They cannot work with one record being a customer and another being an account. So if we are trying to segment our customers into clusters, it would mean that the input data to the algorithm will be one point plotted per customer. Figure 4.5 shows an example of clustering to explain the concept of grain.

Each point represents one customer record with several variables, and the values of those variables have been resolved to result in one point. After plotting them, the clustering algorithm goes through numerous iterations to start to build clusters out of this scatter. The end state looks something like Figure 4.6 where the algorithm puts them into clusters and labels them. In this process one dot represents one customer and therefore it is at the customer grain. If we were using only one variable in the customer record, let’s say customer ID, the algorithm would not be able to find any similarities to bring the dots closer, and the end state of the model would still be the same scatter plot and the tool would probably label it as cluster 1. Therefore, the entire customer population would become one cluster, rendering the effort useless.

For the algorithm to figure out the “likeness” in these dots or points in a multidimensional Euclidean plane, it needs more than just the customer ID to establish likeness. Let’s say we add two more variables, age and gender, into the record and try again. If our customer base is more or less in the same age range—for example, a sports drink manufacturer will have the majority of their customers in the 17–37 age range and may be skewed in favor of males at 78% and females at 22%—then the algorithm will still struggle to find multiple clusters. It may find just three to four clusters with each having millions of customers, still rendering it useless for any effective campaigning. Increasing the variables will help ensure the variables are not skewed toward certain values within their range. But adding too many variables may contribute to losing likeness and very small clusters of 10–15 customers may end up clustered with some very large ones. So the variables, their value spread, and their ability to dissect the population are central to getting the clustering algorithm to produce good results. It is important to note that the same clustering algorithm is being used, but the variables and their quality are determining the quality of output from the model.

Using this as the premise, if we are to add interesting variables, since one point can only have one customer, those additional variables have to be at the customer record level. So a variable like Popular_Products_Purchased that can have multiple values per customer cannot be built just like that. It will have to be transposed to act like columns on a customer record and therefore limit it to a handful of popular products (maybe three or five). So the customer record will get three columns: POP_PROD_PURCHASE_1, POP_PROD_PURCHASE_2, and POP_PROD_PURCHASE_3. These are called performance variables since they exist in a different grain of data, but they have to be built to bring them to the grain of the record where the analytics algorithm is being applied. The definition of “popular” has to be agreed on and programmed on the detail-level data and then pick the top three that become performance variables.

Performance variables are needed because they can spread the data in a good mix that allows the algorithms to produce better results. This is why interesting variables can create very powerful models.

What is a Model?

As described in Chapter 1, it is important to differentiate between a model and the algorithm. Data passes through an analytics algorithm to produce a model. The model takes input records and then produces an output (forecasted, descriptive, predictive, or optimized). In this section we focus on descriptive and predictive models only. Forecasting and optimization models in their description and layout are very similar to predictive models and both techniques have been around for a long time. There is also not much of a mysterious fog on the magical powers of forecasting, since there is nothing hidden within a data mining algorithm or software like we have in predictive and descriptive analytics software. As you will see in the following, forecasting models can be a simple linear algebra equation not requiring the explanation needed of a predictive or descriptive model.

Model and Characteristics in Predictive Modeling

If you want to touch and feel a model, in the following example you can find what it would look like once produced. Some software packages do not allow for easy access to this level of detail on a model. There are good reasons for this, because if input and exact weights of characteristics within the model are known, when a model runs, input can be manipulated to get the desired output. Sometimes leaving a model as a black-box may not be a bad idea. Following is an example of a consumer risk model that takes input data from a loan applicant and tries to produce a risk score. A high score indicates lower risk and vice versa.

Similarly, this type of detailed definition is available for 12–16 characteristics in a typical risk scorecard. This is one example of what a model looks like, but different software may produce different outputs, visuals, actual mathematical equations, etc.

There are two important concepts here: the maximum weight and the weight distribution across the possible values of the variable. Let’s assume the total score for consumer risk is 1,000; the two variables we looked at had the maximum weight of 200 and 250, meaning they account for 450 out of 1,000, or 45% of the total score. The other variables will account for the remaining 55% of the score. So the first step of building a model is to identify the variables and their relative weights with respect to each other. The discriminatory power of a variable to distinguish good and bad records is used to identify the variables that will play a role in the model—that is, they will become characteristics. As input variables from the training set are converted into characteristics by the algorithm to form the model, the relative weight or importance of the variable also emerges with respect to other characteristics in the model. It can be numeric, as used in the preceding example (regression-based models), or it can have probabilities representing their discriminatory power (decision trees).

Once the relative weight of the characteristic is established within the model, we have something that looks like Table 4.5. Scorecard is a numeric representation of a model.

These are characteristics, and each of them could in itself be a more complex derived or aggregated performance variable. In this example the loan decision is being made on this data from the applicant (and from credit bureaus) to assess the risk. This takes care of the maximum weight that each characteristic gets.

The second concept is that of the weight distribution across the various possible values of each characteristic. That problem is broken down into two pieces: the possible values and the weight of each value. For the age characteristic, we used 5 different possible values:

We could use 6 values or 10 values so why these 5? The rule of thumb for identifying the possible values should follow the same approach as discussed in the “Designing Performance Variables” section. The possible values are tied closely into the distribution of the overall population. Once the values have been identified, the weight distribution is based on what role each value has played in determining the 1 or 0 records—that is, the discriminatory power. Data mining tools identify the weights themselves and provide as an output of the training activity as the model is built.

In regression-based models, this level of detail is abundantly clear—a statistician figures out the variables, their values, and their relative weights. Regression software packages also provide visibility into this level of detail but data mining predictive algorithms may not. It is alright to treat the data mining algorithm and the model it creates as a black-box. This level of detail is needed for tuning models, so if data mining software doesn’t provide the detail, how do we tune the predictive models? We will deal with that later in the “Model Validation and Tuning” section.

Model and Characteristics in Descriptive Modeling

The descriptive models are different in nature from predictive models since they don’t need to perform as accurately as the predictive models need to. Since predictions are for a potential future event and business wants to exploit that knowledge and take actions on the predictions, the reliability of the prediction matters a lot. A descriptive model, on the other hand, is describing the data in a form that allows for future action strategies, but it is not a precise event. Rather, it is a perspective into large quantities of data, so business can make sense of the data. It describes data in clusters or association rules so it doesn’t need to be accurate, just approximate. Descriptive analytics has input variables, but their values and weights function differently. When a descriptive analytics model like clustering is complete or built, here is what you find:

That makes the model and its characteristics in descriptive analytics much simpler to review, understand, and use. In predictive analytics, a future event is predicted and that has to be exploited favorably. The focus is the event and, therefore, the usage is tied to the event as well. In contrast, the descriptive model output is an explanation of the data using a structured form like clustering or social network analysis. Once the output is analyzed the question of exploiting this insight becomes wide open to interpretation and innovation. Therefore, the descriptive model’s output is not expected to meet any fixed criteria. This distinction is further explained in the next section.

Model Validation and Tuning

Model validation is essential because of the black-box nature of machine learning algorithms. Once the algorithm has been acquired and data is run through it to build the model, it is important to know what kind of model was built. Is it reliable? Is it a good model? Can business decisions be made on that model? What happens if the model is wrong? The answers to these questions are more important, since this book is encouraging increased use of data mining to solve problems across all functions of a business. It is unlikely that functions of an organization, from HR to finance to procurement and operations, all have access to people who can mathematically review and validate the models. Depending on the data mining software or package, it may not be possible to fully uncover the workings of the algorithm, reconfigure the settings, and ensure good output, even if data mining experts are available on staff. Therefore, with the spirit of this book, we will present a simplistic method of validating a model to satisfactorily answer the questions just posed.

The model validation for predictive models is more critical than the descriptive models since predictive models directly influence a business transaction. The tuning is simply the process of getting the model to pass the validation test. The validation of analytics models is an approximation based on a benchmark of potential usefulness. If the model’s use is not improving the results within the underlying problem statement, it needs tuning. For example, if a credit card default prediction model assigns “low risk” to customers who keep defaulting, then the model fails the validation. The same is true if the model keeps assigning “high risk” to customers who continue to be good customers. We will look at the predictive model validation first.