2.2.3.1 Defining the modeling population

The modeling population is comprised of the instances (customers) to be used for model training. Since the goal is to build a classification model for a targeted marketing campaign, the modeling population also corresponds to the customers who’ll be reached by the marketing action. The decision on who to include should be made with the final goal in mind. Thus, the question that mainly determines the modeling population is “how results will be used.” For instance, which customers do we want to contact to promote a product, or which customers do we want to prevent from leaving? Do we want to reach all customers or only customers of a specific group? For a voluntary churn prevention campaign, for example, we should decide if we want the model and the campaign to cover the entire customer base or just the group of valuable customers.

Usually, not-typical customers such as staff members of VIP customers should be identified and excluded from both the training and the scoring processes.

Quite often, the modeling population is based on the organization’s core segments. In retail banking, for example, customers are typically grouped in segments like affluent, mass, and inactive according to their balances, product ownership, and total number of transactions. These segments contain customers of different characteristics and are usually managed by different marketing teams or even departments. For operational as well as analytical reasons, a cross-segment classification model and marketing campaign would be unusual and probably ineffective. On the other hand, a separate approach and the development of distinct models for each segment may enhance the classification accuracy and the campaign’s effect.

Sometimes, it is worthwhile to go beyond the core business segments and build separate models for subsegments identified by cluster analysis. The cluster model in this approach is a preparatory step which can identify customer groups with different characteristics. The training of separate models for each cluster is an extra workload but might also significantly improve the performance of subsequent classification models due to the similar profiles of the analyzed instances.

In churn modeling, the “customer” status is taken into account (active, inactive, dormant) in the definition of the modeling population. A churn model is usually trained on cases that were active at the end of the observation period. These cases comprise the training population, and their behavior in the outcome period, after the observation period, is examined to define the target event.

In cross-selling, the modeling population is typically defined based on the ownership of products/services. According to the “product uptake” mining approach presented in Section 2.9.1.2, the cross-selling modeling population typically includes all customers not in possession of a product in the observation period who purchased or not the product in the outcome period that followed.

2.2.3.2 Determining the modeling (analysis) level

A decision tightly connected with the selection of the modeling population is the one concerning the level of the model. Apart from deciding whom to analyze, the project team should also agree on the appropriate analysis level which in turn defines the aggregation level and granularity of the modeling file—in plain words, what each record of the modeling file summarizes: customers, accounts, contracts, telephone lines, etc.

Once again, this choice depends on the type of the subsequent marketing campaign. Typically, a customer has a composite relationship with an organization, owning, for instance, more than one credit cards or telephone lines or insurance contracts. The modeling level depends on the specific campaign’s objective. For instance, if the plan is to target customers taking into account their entire relationship with the bank instead of cardholders of individual cards, then a customer-level approach should be followed.

Obviously, the selected modeling level and population delineate the form of the final modeling file and the relevant data preparation procedures that have to be carried out. For a customer-level campaign, all collected data, even if initially retrieved at lower levels, should be aggregated at the desired level for model training and scoring. Original information should be transformed at the granularity level necessary for answering the business question and addressing the specific business objective.

2.2.3.3 Definition of the target event and population

In classification modeling, the task is to assign new cases to known classes by using a model which had been trained on preclassified cases. The goal is to predict an event with known outcomes. Hence, in the design of the modeling process, it is necessary to define the target event and its outcomes in terms of available data—in other words, to reach a decision about the target, also known as the output or label or class attribute of the subsequent model. As explained in Chapter 2, the target attribute in classification modeling is a categorical/symbolic field recording class membership.

Although still at an early stage of the process, it is necessary to outline the target event to be predicted. Obviously, this selection should be made in close cooperation with the marketers of the organization that might already have their definitions for relative processes.

The target population includes the instances belonging to the target event outcome, the target class. These instances are typically the customers we are trying to identify since they belong in the class of special interest. Most of the times, the model’s target attribute is a flag (dichotomous or binary) field denoting membership (Yes/No or True/False) into the target population. In a churn model, for example, the target population is consisted of those that have voluntary churned within the event outcome period, as opposed to those who stayed loyal. In a cross-selling model, those that purchased a product or responded in a promotional campaign comprise the target population.

The definition of the target event should take into account the available data. The analysts involved should also ensure that the designed data model will allow the construction of the target field.

Although a disconnection event might be an obvious and well-defined target for most churn applications, in some cases, it is not applicable. In the case of prepaid customers in mobile telephony, for example, there isn’t a recorded disconnection event to be modeled. The separation between active and churned customers is not evident. In such cases, a target event could be defined in terms of specific customer behavior. This handling requires careful data exploration and cooperation between the data miners and the marketers. For instance, prepaid customers with no incoming or outgoing phone usage within a certain time period could be considered as churners. In a similar manner, certain behaviors or changes in behavior, for instance, substantial decrease in usage or a long period of inactivity, could be identified as signals of specific events and then be used for the definition of the respective target.

Moreover, the same approach could also be followed when analysts want to act proactively. For instance, even when a churn/disconnection event could be directly identified through a customer’s action, a proactive approach would be to analyze model behaviors before the typical attrition by identifying early signals of defection and not waiting for the official termination of the relationship with the customer.

When using behavioral information to build the target event, it is crucial to analyze relevant data and come up with a stable definition. For instance, preliminary analysis and descriptive statistics might reveal the inactivity time period which signifies permanent churn and trivial chances of “revival.” A simple graph will most probably show that the percentage of returning customers becomes insignificant after a specific number of inactivity months. This information can reveal the appropriate time window for defining a stable, behavioral churn event.

2.2.3.4 Deciding on time frames

Most classification models make use of historical, past data. Customer profiles are examined in the observation period before the occurrence of the target event, and input data patterns are associated with the event which follows, whether purchase of an add-on product or churn.

This approach requires the examination of the customer view in different points in time. Therefore, the analyzed data should cover different time frames. The time frame refers to the time period summarized by the relevant attributes. The general strategy is to have at least two distinct time frames. The first time frame, the observation window (also known as the historical period), is a past snapshot of the customer, used for constructing predictors which summarize the customer profile before the event occurrence. The second time frame, referred as event outcome period, is used for recording the event outcome and building the target attribute. Typically, an additional time period, in between the observation and the event outcome period, is taken into account, corresponding to the time needed to gather the data, prepare the model, score new cases, and roll out the targeted marketing campaign. Apparently, this approach requires that predictors should come from a time frame preceding the one used for examining the target event.

Let’s consider, for example, a typical classification model trained on historical data. The data setup is presented in Figure 2.1.

The respective time frames are:

1. Observation (historical) period: Used for building the customer view in a past time period, before the occurrence of the target event. It refers to the distant past, and corresponds to the time frame used for building the model inputs (predictors).
2. Latency period: It refers to the gap between the observation and the event outcome period, reserved for taking into account the time needed to collect all necessary information, score new cases, predict class assignment, and execute the relevant campaign. A latency period also ensures that the model is not trained to identify “immediate” event occurrence, for instance, immediate churners. Even if we manage to identify those customers, chances are that by the time they are contacted, they could already be gone or it will be too late to change their minds. The goal of the model should be long term: the recognition of early churn signals and the identification of customers with increased likelihood to churn in the near but not in the immediate future, since for them there is a chance of retention.
3. Event outcome period: Used for recording the target event and the class assignment. It follows the observation and the latency period, and it is only used for defining the target (output) attribute of the classification model. Predictors should not extend in this time period.

The model is trained by associating input data patterns of the observation period with the event outcomes recorded in the event outcome period.

Typically, in the validation phase, the model’s classification accuracy is evaluated in disjoint time periods. Yet again, the observation period, used for constructing predictors, should precede the event outcome period. In the deployment phase, the generated model scores new cases according to their present view. Now, the observation period covers the period right before the present, and the event outcome period corresponds to the future. The event outcome is unknown and its future value is predicted.

A more specific example of a voluntary churn model setup is illustrated in Figure 2.2.

In this example, the goal of a mobile telephony network operator is to set up a model for the early identification of potential voluntary churners. This model will be the base for a respective targeted retention campaign and predicts voluntary attrition 1 month ahead. We assume that the model is built in October:

• The model is trained on a 6-month observation window covering January to June. The inputs summarize all the aspects of the customer relationship with the organization in that period providing an integrated customer view also referred to as the customer signature. Yet, they come strictly from the historical period, and they do not overlap with the target event which is defined in the outcome period.
• The model is trained on customers that were active at the end of the observation window (end of the 6-month historical period). These customers comprise the modeling population.
• 1 month has been reserved in order to allow for scoring and campaign preparation. This month is shown as a grayed box in the respective illustrations and it corresponds to the latency period. No inputs were used from the latency period (July).
• The target event is attrition, recorded as application for disconnection in the event outcome period, that is, within August.
• The target population is consisted of those that have voluntary churned in August.
• The model is trained by identifying the input data patterns associated with voluntary churn.
• The generated model is validated on a disjoint data set of a different time period, before being deployed for classifying currently active customers.
• In the deployment phase, customers active in September, are scored according to the model. Remember that the model is built in October which is the current month.
• The profile of customers is built using information from the 6 most recent months, from April to September. The model predicts who will leave in November. If October churners become known in the meantime, they are filtered-out from the target list before the execution of the campaign.
• The presented time frames used in this example are purely indicative. A different time frame for the observation or the latency period could be used according to the specific task and business situation.

The length of the observation window should be long enough to capture and represent a stable customer profile as well as seasonal variations. However, it should not be extended that much to also capture outdated behaviors. Typically a time span of 12–24 months, depending on the specificities of the organization, is adequate.

For the length of the event outcome period, we should consider a possible periodicity of the target event as well as the size of the target population. Apparently, for a target event with seasonal patterns, this seasonality should be taken into account when selecting the event outcome period. In terms of the size of the target population, ideally, we’d prefer to have at least some hundreds of customers for model training. This might not be the case if the target event is rare. Although there are methods for boosting the number of target population cases and they are discussed later in this chapter, sometimes we can capture more positive cases by simply extending the time span of the event outcome period obviously at a cost of using and analyzing more data.

An advisable approach for training a classification model is to incorporate multiple time windows. With this approach, we can capture a more extended view of customer behaviors. Furthermore, we can also capture possible seasonal effects, avoiding the pitfall of using data from a particular season to train the model.

In the setup illustrated in Figure 2.3, 18 months of history are analyzed with three overlapping observation windows. In each setup, 6 months of historical data are used to predict churn 1 month ahead, and a single month is reserved as the latency period.

In the first time window, input data patterns recorded in past year’s July to December are used to predict February churners. The second time window is shifted 5 months ahead. This time, the customer profiles are based on the period from December to May, and the target population includes July’s churners. Finally, December’s churners are predicted from historical data covering May to October. With this approach, available data are partitioned to build the three data setups allowing a broader capturing of behaviors. The patterns captured by the model do not reflect a single period. Moreover, this approach increases the number of cases in the model set and the frequency of the target population, a desirable result, especially in the case of limited training instances.

2.3.6.1 Split or Holdout validation

A Split (Holdout) validation method works by partitioning the modeling file into two distinct, exhaustive, and mutually exclusive parts with random sampling. The training part usually contains the majority of training instances, and it is used for model training. The testing part is reserved for model evaluation. Its instances do not take part in model training. A common practice is to use a split ratio of about 70–75% for the training file and hence allocate approximately 70–75% of the cases in the training dataset and hold out about 25–30% of instances for evaluation.

Apparently, analysts should mainly focus on the examination of performance metrics in the testing dataset. A model underperforming in the testing dataset should be reexamined since this is a typical sign of overfitting and of memorizing the specific training data. Models with this behavior do not provide generalizable results. They provide solutions that only work for the particular data on which they were trained.

To better illustrate the logic of Split validation, let’s have a look at Table 2.2 which presents a simple modeling file for 20 customers. A pilot cross-selling campaign has been carried out, and an indicative list of inputs is used to classify customers to responders and nonresponders. The modeling file is partitioned in training and testing files through random sampling. The rightmost column of the table denotes allocation to the two partitions. Roughly 70% of the training instances are assigned to the training partition, leaving about 30% of the instances (six customers) for model evaluation.

The distribution of the target attribute should be analogous into both the training and the testing dataset. Normally, a partitioning based on simple random sampling will not distort the overall distribution of the target attribute. However, for even more accurate representation of all classes in the partitions, a Split validation with proportionate stratified sampling can be applied. With this approach, random samples are drawn independently from each class. The same sample ratios are used for each class, ensuring that a proportionate number of cases from all outcomes is allocated in the both the training and testing data files.

Some analysts split the modeling file into three distinct parts. Alongside the training and the testing files, an additional partition, the validation dataset is drawn. The training file is used for model training. The parameters of the generated model are refined on the validation file. Finally, the fine-tuned model is evaluated on the testing dataset.

2.3.6.2 Cross or n-fold validation

Although in most data mining applications we normally have enough training instances, there may be situations with data limitations. In such cases, we’d like to use all instances for model training without reserving a part for evaluation. This is what Cross- or n-fold validation does.

Initially, the cases are split into n distinct random subsamples or folds approximately of equal size. The process includes n iterations. In the first iteration, one of the folds is set apart, and the model is trained on the remaining training instances. The generated model is evaluated on the holdout fold. This process is repeated n times. In each iteration, a different fold is hold out for evaluation, and the model is trained on the other (n − 1) folds. In the end, all folds have been used (n − 1) times for model training and once for model evaluation.

Finally, the number of correct classifications and misclassified cases is counted across all the testing folds, and they are divided by the total number of cases to calculate the accuracy (percentage of correct classifications) and the error rate (percentage of misclassifications) of the full model. In general, the individual evaluation measures calculated on each fold are combined, typically averaged, to assess the performance of the full model.

Cross validation is the preferred validation in cases with data limitations. But it also has another advantage. The model evaluation is not based on a single random subsample as in Split validation. It is repeated n times on n different folds, providing a more reliable and unbiased evaluation of the model.

The standard approach is to use a 10-fold Cross validation. Table 2.3 presents a modeling file of 20 instances on which a fourfold Cross validation has been applied. Each fold contains five customers. In the first iteration, customers 1, 11, 14, and 18 of Fold 1 are hold out for assessing the performance of the classification model trained on Folds 2, 3, and 4. The procedure is repeated for three more times, each time with a different fold holdout for validation.

2.3.6.3 Bootstrap validation

Bootstrap validation uses sampling with replacement; hence, a training case can be included more than once in the resulting training file.

More specifically, the initial modeling file of size n is sampled n times with replacement to give a training file of size n. Due to the sampling with replacement, the bootstrap sample will include more than one instance of some of the original n cases, while other cases will not be picked at all. In fact, it turns out (and it can be shown if we calculate the probabilities of selection) that with the method described here, referred to as the 0.632 bootstrap, on average 63.2% of the original cases, will be included in the bootstrap sample and around 36.8% will not be picked at all. These cases comprise the testing file for model evaluation.

Although the number of total cases of the resulting training file (the bootstrap sample) remains equal to the original size n, the fact is that it contains less “unique” cases compared, for instance, with a 10-fold cross validation training file. Using only the bootstrap testing file for evaluation would lead to pessimistic measures of classification accuracy. To compensate for that, the estimated error rate (misclassification percentage) of the full model is also taking into account the training cases of the bootstrap sample, and it is calculated as follows:

The overall bootstrap procedure can be repeated for k times and the results can be averaged over iterations. The Bootstrap validation method works well with very small datasets.

2.3.7.1 Balancing

Balancing is the application of disproportionate stratified sampling on the training file to adjust class imbalances. Balancing changes the distribution of the training file. The frequent classes, typically denoting nonmembers of the target population, are undersampled. A random percentage of these instances is included in the balanced training file by applying a sample ratio, also referred to as balance factor, less than 1.0. Therefore, the frequency of the common cases is reduced. In the end, the training file will include all target class cases and a fraction of the other cases. Additionally or alternatively, the density of the rare class can be “inflated” by oversampling with a sample ratio higher than 1.0. Target class cases are resampled and may be included in the training file with more than one instance.

Either way, the rare class is “boosted in the balanced training file and its frequency is increased. Therefore, models should be evaluated on unbalanced data. Additionally, when scoring with a model trained on balanced data, we should be cautious with the generated scores. When a model is applied to new cases, along with the class prediction, it estimates its confidence and hence the propensity, the likelihood of the target outcome. If balanced data are used for training, the resulting propensities do not correspond to the actual probabilities of belonging to the target class. However, propensities are comparable and can be used to rank customers according to their likelihood of belonging to the target class.

The calculated probabilities for a binary target field can be adjusted for oversampling using the formula below:

where is the adjusted probability estimate of class i, the unadjusted probability estimate, π_i the actual population proportion of class i in the original training dataset, and ρ_i the observed proportion of class i in the balanced sample. The adjusted probabilities can be used as estimates of the actual class probabilities.

IBM SPSS Modeler offers an interesting solution to this issue. A generated model estimates adjusted propensities. These propensities are based on the unbalanced testing file (partition), and they are not affected by the balance.

The recommended balancing approach is to reduce the common outcomes through undersampling instead of boosting the rare ones with oversampling. The scope is to attain a target class percentage of about 25–50%.

Table 2.4 presents a class-imbalanced dataset for cross-selling. Three out of 30 customers have responded to the pilot campaign and were classified in the target class. Since the target class percentage is a poor 10%, the file was adjusted with balancing.

The training file after balancing is listed in Table 2.5. A balance factor of 1.0 was used for the target class, and all responders were retained. On the other hand, nonresponders were undersampled with a sample ratio of 0.26, and a random sample of seven nonresponders was selected for model training. After balancing, the proportion of the target class has been raised to 30% (3 out of 10).

2.3.7.2 Applying class weights

An alternative approach to deal with class imbalances is with the application of class weights. The contribution of each case to the model training is defined by its weighting factor. This method has the advantage of using all of the available training data, provided of course that the modeling algorithm supports case weights. In the end, estimated propensities should be adjusted to correspond to the actual probabilities of the target class.

A recommended approach is to assign a weight of 1.0 to the rare class and a weight lower than 1.0 to the frequent class to give extra importance to the underrepresented cases. This weighting is shown in Table 2.6. A weighting factor of 1.0 is assigned to responders of the pilot cross-selling campaign and a weight of 0.26 to the 27 nonresponders, adjusting the distribution of the target class to 30–70%.

Class weights should be ignored in the model evaluation procedure, and the model should be tested on unweighted data.

2.4.2.1 Bagging

With Bagging (Bootstrap Aggregation), multiple models of the same type are trained on different replicates of the original training file. A set of n bags, n bootstrap samples, typically of the same size as the original dataset, are drawn with replacement. Due to the resampling applied to the original training data, some training instances may be omitted, while others replicated. This technique involves n iterations. In each iteration, a different model is built on each bag. The set of n models generated are combined and vote to score new cases.

Bagging can improve the classification accuracy compared to individual models (although not as dramatically as Boosting which is presented immediately after). Additionally, since the ensemble model is based on more than one datasets, which unfortunately are not independent, it can be more robust to noisy data and hence more reliable and stable.

2.4.2.2 Boosting

Boosting involves the training of a sequence of models of the same type which complement one another. In the first iteration, a classifier is built the usual way. The second model though depends on the first one as it focuses on the instances which the first model failed to classify correctly. This procedure is continued for n iterations with each model aiming at the “hard cases” of the previous models.

Adaptive Boosting (AdaBoost), a popular Boosting technique, works as follows: Each record is assigned a weight denoting its “hardness” to be classified. In the first iteration, all records are assigned equal weights. After the first model run, these weights are adjusted so that misclassified instances yield higher weights and correctly classified ones lower weights. The subsequent base model is trained on the weighted instances. Assuming the base learner supports case weights, the contribution of each case to the model training is defined by its weight. That’s why each base model starts from where the previous models stopped.

Even in the case of a type of classifier which does not support case weights, the AdaBoost technique can be employed by using samples with replacement for each iteration. On the contrary to Bagging though, the inclusion probability of each instance should be proportional to its estimated weight.

A weighted voting procedure is finally applied for scoring with the boosted model. The weight of the vote of each base model depends on its error rate. Hence, more accurate classifiers have stronger influence on the prediction. The Boosting can substantially improve predictive accuracy; however, it is demanding in terms of training time and prone to overfitting.

2.4.2.3 Random Forests

Random Forests use Bagging and bootstrap samples to generate n different Decision Trees in n iterations. The Decision Tree models differ not only on their training instances but also on the used predictors. A random subset of predictors is considered for partition at each node. The predictor for the best split is chosen, according to the tree’s attribute selection method, on the random list of attributes. A voting procedure is applied for scoring with the ensemble classifier.

Random Forests are faster than Bagging and Boosting and can improve the classification accuracy, especially if the different tree models are diverse and not highly correlated. This can be achieved by keeping the subset ratio relatively low.

2.5.1.1 Accuracy measures and confusion matrices

One of the most common ways to summarize the model accuracy is through a Confusion (also known as misclassification or coincidence) matrix such as the one presented in Table 2.7 for a two-class target attribute. Positive refers to the target event instances, for example, churned customers or responders to a direct marketing campaign.

A Confusion matrix is a simple cross-tabulation of the predicted by the actual classes. It is the base for the calculation of various accuracy measures including the accuracy and the error or misclassification rate.

The accuracy denotes the percentage of instances correctly classified and is calculated as

In other words, it sums the table percentages of the Confusion matrix across the diagonal. In binary classification problems, a 50% estimated probability threshold is used to classify an instance as positive or negative. The error rate is the off-diagonal percentage—the proportion of records misclassified—and is calculated as

Since some mistakes are more costly than others, accuracy percentages are also estimated for each category of the target field. The percentage of actual positives (target instances) correctly captured by the model defines sensitivity:

The percentage of negative instances correctly classified is the specificity:

while

The false positive rate denotes the proportion of negative (nontarget) instances misclassified as positive (target). Remember the false positive rate when we discuss the ROC curve later in this section.

The aforementioned measures are very useful especially in the case of class-imbalanced problems where the target event is rare. In such cases, a model may present an acceptable accuracy rate while failing to identify the target instances. Imagine, for example, an organization with an overall churn rate of about 2%. A naïve model classifying all customers as nonchurners would yield an astonishingly high accuracy of 98%. Obviously, this model has no practical value as it fails to capture any churner as denoted by its 0 sensitivity.

Other accuracy measures include Precision, Recall, and the F-measure which stems from their combination.

The Precision measure represents the percentage of the predicted positives which were actual positives and can be thought of as a measure of exactness:

The Recall measure is the same as sensitivity.

The F-measure is the harmonic mean of Precision (a measure of exactness) and Recall (a measure of completeness). It ranges from 0 to 1 with larger values corresponding to better models. It is defined as

2.5.1.2 Gains, Response, and Lift charts

Many times, in the real world of classification modeling, we end up with models with moderate accuracy, especially in cases with a rare target event. Does this mean that the model is not adequate? Remember that in order to estimate the accuracy and the error rate of a model in a binary classification problem, we typically use a default propensity threshold of 50% for classifying an instance as positive. In problems with a rare target class, an increased FN rate and therefore an increased error rate may simply mean that the estimated propensities are often below 50% for actual positives. Does this mean that the model is not useful? To answer that, we must examine whether the positive instances yield higher estimated propensities compared to the negative ones. In other words, we must examine if the model propensities rank well and if they discriminate the actual positives from the actual negatives. The Gains, Response, and Lift/Index tables and charts are helpful evaluation tools that summarize the model’s performance and discrimination power. In this paragraph, we’ll present these charts and the measure of Lift which denotes the improvement in concentration of the target class due to the model. To illustrate the basic concepts and usage of these charts, we will present the results of a hypothetical churn model that was built on a binary target field which flagged churners.

The first step for the creation of such charts is to select the target class of interest, also referred to as the hit category. For Gains/Response/Lift charts as well for ROI and Profit charts which will presented immediately after, we assume that the classifier can estimate the probability of belonging at each class and hence the hit propensity which is the likelihood of the target class. Records/customers can then be sorted in descending order according to their hit propensities and binned into groups of equal size, typically of 1% each, named percentiles. The model accuracy is then evaluated within these percentiles.

In our hypothetical example, the target class is the category of churners, and the hit propensity is the churn propensity—in other words, the estimated likelihood by the churn model. Customers are split into 10 tiles of 10% each (deciles). The 10% of customers with the highest churn propensities comprise tile 1, and those with the lowest churn propensities, tile 10. In general, we expect high propensities to correspond to actual members of the target population. Therefore, we hope to find large concentrations of actual churners among the top model tiles.

The cumulative Table 2.8 evaluates our churn model in terms of the Gain, Response, and Lift measures.

But what exactly do these performance measures represent and how they are used for model evaluation? A brief explanation is as follows:

• Response %: “How likely is the target class within the examined quantiles?” It denotes the likelihood of the target outcome, the percentage of the actual churners (positives) within the tiles. It is a measure of exactness and is the same as the Precision measure discussed in Section 2.5.1.1.

  In our example, 10.7% of the customers of the top 10% model tile were actual churners, yielding a Response % of the same value. Since the overall churn rate was about 2.9%, we expect that a random list would also have an analogous churn rate. However, the estimated churn rate for the top model tile was 3.71 times (or 371.4%) higher. This is called the Lift. Analysts have achieved about four times better results than randomness in the examined model tile. As we move from the top to the bottom tiles, the model exactness decreases. The concentration of the actual churners is expected to decrease. Indeed, the first two tiles, which jointly account for the top 20% of the customers with the highest estimated churn scores, have a smaller percentage of actual churners (8.2%). This percentage is still 2.8 times higher than randomness.

• Gain %: “Which percentage of the target class falls in the tiles?” Gain % is defined as the percentage of the total target population that belongs in the tiles. It is the same as Sensitivity and Recall presented in Section 2.5.1.1; hence, it is a measure of completeness. It denotes the true positive rate, the percentage of true positives (actual churners) included in the tile.

  In our example, the top 10% model tile contains 37.1% of all actual churners, yielding a Gain % of the same value. A random list containing 10% of the customers would normally capture about 10% of all observed churners. However, the top model tile contains more than a third (37.1%) of all observed churners. Once again, we come upon the Lift concept. The top 10% model tile identifies about four times more target customers than a random list of the same size.

• Lift: “How much better is the classifier compared to randomness?” The Lift or Index assesses the factor of improvement in response rate due to the model. It is defined as the ratio of the model Response % (or equivalently Gain %) to the Response % of a random model. In other words, it compares the model quantiles with a random list of the same size. Therefore, it represents how much a trained classifier exceeds the baseline model of random selection.

The Gain, Response, and Lift evaluation measures can also be depicted in corresponding charts such as the ones shown in the following. The two added reference lines correspond to the top 5% and the top 10% tiles. The diagonal line in the Gains chart represents the baseline model of random guessing.

According to the cumulative Gains chart shown in Figure 2.36, when scoring an unseen customer list, data miners should expect to capture about 40% of all potential churners if they target the customers of the top 10% model tile. Narrowing the list to the top 5% percentile decreases the percentage of potential churners to be reached to approximately 25%. As we move to the right of the X-axis, the expected number of total churners (true positives) to be identified increases. But this comes at a cost of increased error rate and false positives. On the contrary, left parts of the X-axis lead to smaller but more targeted campaigns.

What we hope to see in a real model evaluation is a Gains curve steeply and smoothly rising above the diagonal along the top tiles before gradually easing off after a point.

Analysts can study Gains charts and compare the accuracy of models. A model closer to the diagonal line of random guessing is less accurate. A Gains chart typically also includes an Optimal or Best line which corresponds to the ideal model that classifies all records correctly.

Although we’d like a Gains curve to be close to the Optimal line, extreme proximity and absolute accuracy might indicate a problem with the model training such as using a predictor directly related with the target attribute.

By studying the Gains charts, analysts assess the model’s discriminating power. They also gain valuable insight about its future predictive accuracy on new records. These charts can be used for deciding the optimal size of the respective campaign by choosing the top propensity-based tiles to target. Hence, they may choose to conduct a small campaign, limited to the top tiles, in order to address only customers with very high propensities and minimize the false positive cases. Alternatively, especially if the cost of the campaign is small compared to the potential revenue, they may choose to expand their list by including more tiles and more customers with relatively lower propensities.

Figure 2.37 presents the cumulative Response chart for our hypothetical example.

It illustrates the estimated churn likelihood along the model tiles. As we move to the left of the X-axis and toward the top tiles, we have increased churn probabilities. These tiles would result more targeted lists and smaller error rates. Expanding the list to the right part of the X-axis, toward the bottom model tiles, would increase the expected false positive error rate by including in the targeting list more customers with low likelihood to churn.

The cumulative Lift or Index chart (Figure 2.38) directly compares the model predictive performance with the baseline model of random selection. The concentration of churners is estimated to be four times higher than randomness among the top 10% customers and about six times higher among the top 5% customers.

2.5.1.3 ROC curve

The ROC curve also visualizes the performance and the discriminating power of the classifier. It plots the model’s true positive rate, the sensitivity, in the vertical axis. Its horizontal axis corresponds to the false positive rate (1—specificity), the proportion of negative (nontarget) instances misclassified as positive. Therefore, the ROC curve depicts the trade-off between capturing more positives but with an increased cost of false positives.

Just as the Gains chart, the ROC curve is based on the rank ordering of the test instances in decreasing order according to their propensities. The Gains chart and the ROC curve also have the same vertical axis, the true positive percentage. Their difference is at the horizontal axis. In Gains charts, it plots the percentage of the total test population, while in ROC curves, the proportion of false positives. That’s why the ROC curve shape does not depend on the overall distribution of the target category, and hence, it is unaffected by oversampling. However, since in many real-world applications the density of the target category is small (for instance, below 1%), there is little difference between the proportion of the total test population and the proportion of total negatives, and hence, the ROC curve and the Gains chart have similar shapes.

If the model is adequate, its ROC curve will rise sharply near the vertical axis before easing off. In the case of a trivial model, the curve will approximate a diagonal line from the lower left to the upper right corner of the graph. A measure of the accuracy of the model is the area under the curve (AUC) which is equivalent to the c-statistic. It ranges between 0 and 1.0. The closer the AUC is to 1.0, the better the model. A model with AUC close to 0.5 is not better than random guessing. An ideal model will have a value of 1.0, while values above 0.7 can be considered adequate.

The AUC measure is also equivalent to the Gini index. The Gini index is calculated as the area between the ROC curve and the diagonal line of the random model divided by the area between the optimal curve and the diagonal. It ranges between 0 and 1.0 with values above 0.4 denoting acceptable efficiency. In fact:

2.5.1.4 Profit/ROI charts

Profit and ROI charts are extensions of the Gains/Response charts which incorporate cost and revenue information to help marketers decide their target lists based on estimated revenue.

The model estimated target probabilities are combined with expected cost and revenue information to calculate the probabilistic Profit and/or ROI for the model percentiles.

Marketers must specify:

• Cost: The estimated cost per offer (for each customer included in the campaign)
• Revenue: The anticipated revenue associated with each hit, that is, for each customer accepting the offer

Customers are sorted in descending order according to their hit propensities and then binned into percentiles as presented in Section 2.5.1.2. Then, the estimated (cumulative) profit per offer is calculated for each model percentile as follows:

Obviously, revenues concern only responders (hits), while costs apply to all records/offers.

By multiplying the profit per offer with the number of customers of the tile, we have the total (cumulative) profit for the tile. In case of an overhead cost, it should be subtracted from the total profit.

The estimated ROI per offer, expressed as the percentage return on cost, is calculated as

Hence, the ROI per offer is the ratio of the average profit to average cost for each record of the tile. Negative values indicate loss per offer and correspond to negative profit per offer.

2.6.1.1 Building propensity segments

Estimated propensities can be used for grouping customers in segments, in propensity pools, according to their target class likelihood. For instance, customers can be divided into groups of low, medium, and high churn likelihood as follows:

1. High Churn Risk

   Comprised of scored customers with churn scores above a selected propensity threshold with business meaning, for example, at least n times higher than the total population churn rate. This segment will be the basic pool for customers to be included in a retention campaign.

2. Medium Churn Risk

   Includes the customers with churn propensities lower than the cutoff value for the high-risk group but higher than the observed overall population churn rate.

3. Low Churn Risk

   Includes the customers with the lowest churn propensities, for example, below the observed overall population churn rate.

If the propensity segmentation is based on specific propensity cutoff values (“hard” thresholds) and it is monitored over time, then possible changes in the distribution of the pools should be tracked, investigated, and explained. If alternatively the grouping is based on frequency bins and propensity percentiles (“soft” thresholds, e.g., top 10%, medium 40%, low 50%), then the boundaries of the pools should be monitored over time to identify changes in the propensity distribution.

Propensity scores and respective segmentations can also be combined with other standard segmentation schemes such as value-based segments. For instance, when value segments are cross-examined with churn probability segments, we have the value-at-risk segmentation, a compound segmentation which can help in prioritizing the retention campaign according to each customer’s value and risk of defection.

2.8.1.1 Pilot campaign

Mining approach: This approach involves the training of a classification model on a random sample of prospects. We assume that a list of prospects is available with sufficient profiling information. A test campaign is run on a random sample of prospects; their responses are recorded and analyzed with classification modeling in order to identify the profiles associated with increased probability of offer acceptance. The trained models can then be used to score all prospects in terms of acquisition probability. The tricky part in this method is that it requires the rollout of a test campaign to record prospect responses in order to be able to train the respective models.

Modeling population: The modeling population of this approach is the random sample of prospects included in the pilot campaign.

Target population: The target population includes those who responded in the campaign.

Scoring population: The scoring population is consisted of all prospects who didn’t participate in the campaign.

Hints:

• All marketing parameters of the pilot campaign—such as product, message, and channel—must be the same with the ones of the actual designed campaign.
• A problem with pilot campaigns is that you may need large random samples and hence they are expensive. To build a good model, you need at least 100 respondents (positive responses). With an estimated response rate of around 1%, you need at least 10 000 prospects to achieve this.

The modeling phase of this approach is outlined in Figure 2.63.

2.8.1.2 Profiling of high-value customers

Mining approach: An alternative approach, often combined with the one described earlier, is to mine the list of prospects looking for potentially valuable customers. According to this approach, a classifier is trained on existing customers to identify the key characteristics of the high-value customers. The trained model is then deployed on the prospects to discern the ones with similar characteristics. Propensities now indicate similarity to high-value customers and not likelihood to uptake an acquisition offer.

Modeling population: The model training in this approach is based on existing customers.

Target population: The target population is comprised of high-value customers, for instance, customers belonging to the highest value segments.

Scoring population: The model rules are applied to the list of prospects.

Hints:

• The key to this process is to build a model on existing customers using only fields which are also available for prospects.
• For example, if only demographics are available for prospects, the respective model should be trained only with these data. Acquisition marketing activities could target new customers with the “valuable” profile, and new products related to these profiles could be developed, aiming to acquire new customers with profit possibilities.

The modeling phase of this approach is illustrated in Figure 2.64.

2.9.1.1 Pilot campaign

Mining approach: An outbound test campaign is rolled out on a random sample of existing customers who do not own the product. Responses are mined with classification modeling and the profiles of responders are analyzed. The generated model is then applied to existing customers who do not own the target product/service and hadn’t participated in the test campaign. Customers with increased probability to uptake the offer are identified and included in the large-scale campaign that follows.

Modeling population: The modeling population of this approach is the random sample of existing customers, not owners of the target product, included in the pilot campaign.

Target population: The target population includes those responded positively in the campaign offer and bought the product.

Scoring population: The scoring population includes current customers who do not own the product.

Hints:

• This approach can also be followed in the case of an inbound campaign. During a test period, a test inbound campaign is carried out on a sample of incoming contacts. Their responses are recorded and used for the development of a classifier which then targets incoming cross-selling campaigns.
• As mentioned previously, this approach is demanding in terms of time and resources. However, it is highly effective since it is a simulation of the actual planned campaign, provided of course all aspects—namely, product, message, channel, and direction (inbound/outbound)—are the same with the ones of the designed campaign.

The modeling phase of this approach is outlined in Figure 2.65.

2.9.1.2 Product uptake

Mining approach: Historical data are used and a classifier is trained on customers who did not own the target product at a specific time point in the recent past. The characteristics of the recent “buyers” are identified, and customers with the same outlook who do not currently own the product are selected for inclusion in the campaign.

The “product uptake” approach is effective since it tries to identify the behavioral patterns in the observation period which were followed by the occurrence of the target event, in this case product purchase. However, it requires the building of a customer view in more than one time points. In order to build the model, we must go back and use historical data which summarize the customer behavior at a past time period before the event occurrence. And then we have to move forward in time and use the current customer view in order to score customers with the generated model and estimate their likelihood to uptake the product.

Modeling population: All active customers not owning the target product at the end of the analyzed observation period.

Target population: Those customers who acquired the product in the recent past, within the examined event outcome period.

Scoring population: All active customers not currently owning the product.

Hints:

• This approach is effective; however, it is demanding in terms of data preparation since it involves different time frames. In model training, the customer “signature” is built in the observation period, and it is associated with subsequent product purchase. Then, current “signatures” are used for model deployment.
• The model propensities of this approach denote the likelihood of buying the product in the near future and can be used for targeting a planned cross-selling campaign. However, they are not estimates of offer acceptance probabilities as the ones calculated with the pilot campaign method.

The modeling phase of this approach is illustrated in Figure 2.66.

2.9.1.3 Profiling of owners

Mining approach: A model is trained on all active customers and identifies the data patterns and characteristics associated with ownership of the target product. The profile of owners (and preferably heavy users) of the target product is outlined. Customers with the same profile are identified among the population of nonowners, and they are included in the planned cross-selling campaign.

Modeling population: All currently active customers.

Target population: Customers owning (and preferably heavily using) the product to be promoted.

Scoring population: All nonowners of the product.

Hints:

• This approach is appealing due to its straightforwardness and simple data preparation. It does not require distinct observation and outcome windows and the examination of the customer view in different time periods. However, this comes at a cost. The predictive efficiency of this approach is limited since it takes into account the current profile of customers instead of the characteristics prior the purchase event who most likely leaded to the purchase decision.

The modeling phase of this approach is presented in Figure 2.67.

2.11.1.1 Pilot campaign

Mining approach: Customers owning but not heavily using the promoted product are identified, and a random sample of them is drawn to be included in the test campaign. Those selected receive an offer which promotes the usage increase, and their responses are collected and analyzed. Those who increased their usage comprise the target population. The trained model is then deployed on “infrequent” users who were left out of the pilot campaign, and those scored with propensities above the selected threshold are targeted.

Modeling population: The modeling population of this approach is the random sample of owners but “infrequent” users of the target product, included in the pilot campaign.

Target population: The target population includes those who increased their usage after receiving the respective offer.

Scoring population: “Infrequent” users who didn’t participate in the test campaign.

Hints:

• As in the case of cross-selling models, this approach can also be followed in the case of inbound campaigns.

The modeling phase of this approach is outlined in Figure 2.69.

2.11.1.2 Usage increase

Mining approach: A historical view of the customers is assembled, and customers owning but not heavily using the deep-sell product at the end of the observation period are analyzed. Those who substantially increased their usage in the event outcome period are flagged as “positive” cases and form the target class of the classification model. The model is then deployed on current owners/low users of the product. Those scored with high deep-selling propensities are included in the campaign that follows. The “usage increase” approach, as all approaches based on “historical views” of customers, is effective, yet it presents difficulties in its implementation compared to plain profiling approaches.

Modeling population: All owners/low users of the target product at the end of the observation period.

Target population: Those customers who increased their product usage in the outcome period that followed.

Scoring population: All active customers currently owning but infrequently using the deep-sell product.

Hints:

• An issue which deserves attention and the close collaboration of data miners with the marketers is, as always, the definition of the modeling population and of the target event and/class—in other words, the definition of what constitutes low and heavy usage and what signifies a substantial usage increase. The usage increase can be defined in terms of absolute or relative increase compared to the historical period analyzed.

The modeling phase of this approach is illustrated in Figure 2.70.

2.11.1.3 Profiling of customers with heavy product usage

Mining approach: A model is built on all owners of the deep-sell product which is trained to discern heavy from low users. The drivers of heavy product usage are discovered and clones of heavy users are identified among low users. These customers (low users predicted as heavy users) are targeted by the deep-selling campaign.

Modeling population: All active customers currently owning the target product.

Target population: Those heavily using the product/service to be promoted.

Scoring population: All low users of the product.

The modeling phase of this approach is illustrated in Figure 2.71.

2.12.1.1 Pilot campaign

Mining approach: A classifier is built on a sample of “basic” customers who were randomly chosen and received an up-selling offer for the target “premium” product. The model is trained on the recorded responses of the test-campaign list. Those who “switched” to the “premium” product are the positive instances and comprise the target population. The data patterns associated with offer acceptance are identified and captured by the generated model which is then deployed on the mass population of “basic” product owners. An “upgrade” propensity is estimated for all the scored customers, and a large-scale up-selling campaign is then conducted using these propensities.

Modeling population: The modeling population of this approach is the random sample of owners of the “basic” product.

Target population: The target population is comprised of those who accepted the offer and agreed to “upgrade” their product.

Scoring population: “Basic” product owners who didn’t participate in the test campaign and who obviously don’t have the “premium” product.

Hints:

• As in the case of cross-selling models, this approach can also be modified and applied in the case of inbound campaigns.
• As opposed to deep- and cross-selling, up-selling campaigns aim at widening the relationship with existing customers by upgrading their current products instead of selling more of the same product or additional products.

The modeling phase of this approach is outlined in Figure 2.72.

2.12.1.2 Product upgrade

Mining approach: This approach requires historical data and a historical view of the customers. A classifier is built on customers who owned the “basic” product/service at a specific time point in the recent past (observation period). The generated model is trained to identify the data patterns associated with upgrading to the premium product during the event outcome period. The scoring phase is based on the current view of the customers. The generated model is deployed on current owners of the entry-level product, and those with relatively high upgrade propensities are included in the campaign list. The “product upgrade” approach is analogous to the “product uptake” and the “usage increase” approaches presented for cross-/deep-selling models.

Modeling population: All owners of the “basic” product/service at the end of the observation period.

Target population: Those customers who upgraded to the “premium” target product within the event outcome period.

Scoring population: All customers who currently own the “basic” but not the “premium” product.

The modeling phase of this approach is illustrated in Figure 2.73.

2.12.1.3 Profiling of “premium” product owners

Mining approach: A classification model is built on the current view of both the “basic” and the “premium” product owners. The generated model discerns the data patterns associated with ownership of the target up-sell product. The scoring population is consisted of all owners of the “basic” product. The final campaign list includes all the “basic” product owners who are “clones” of the “premium” product owners. That is, although they do not own the target product, they present similar characteristics with its owners, and consequently, they are scored with increased “premium” product ownership propensities.

Modeling population: All current owners of the “basic” and the “premium” target product.

Target population: Owners of the “premium” target product.

Scoring population: All owners of the “basic” product.

The modeling phase of this approach is presented in Figure 2.74.