6.2.1 Designing the churn propensity model process

The decisions made in the design process concluded on the following.

6.2.1.4 Time periods and historical information required

For the needs of model training, the retrieved data covered three distinct time periods: the observation period, the latency period, and the event outcome period.

Twelve months of historical data was decided to be analyzed in order to identify data patterns relevant to attrition. Since the modeling procedure started at mid-2013 (start of July 2013), the observation window covered the whole 2012. A latency period of 2 months was reserved for preparing the relevant campaign. A period of 4 months was examined for churn. Thus, an active customer was considered as a churner if he/she had closed all his/she cards within the 4 months after the latency period, from March to July 2013. The approach is described in Figure 6.1 which is also a guide for the data setup for the model training.

6.4.1 From cards to customers: aggregating card-level data

Since the scope was to analyze and target customers, individual card usage had to be combined to convey the behavior of each customer. Therefore, the initial card usage data, listed in Table 6.1, had to be aggregated at the customer level. This step included the grouping of individual card records to summarized customer records with the use of standard summary statistics and the construction of customer attributes such as the total spending amount and the number of transactions of all cards, the last transaction date, etc. This step is illustrated in Figure 6.2.

Even after aggregation, the data preparation phase was far from over since summarized data had to be enriched with informative KPIs.

But before we get into that, let’s spend some more time examining the aggregation procedure. Before the actual grouping and the replacement of the card records with the customer records, cards already closed before the observation period (prior 2012-1-1) were filtered out. Obviously, it makes no sense to analyze cards that had already been terminated before the period under examination. At first, cards open at some point within the observation period were flagged as OPEN WITHIN. These cards comprise the population to be examined for the model.

Specifically, by examining the cards’ closing dates, a card was flagged as OPEN WITHIN if it hadn’t been closed (it had a null closing date) at all or it had been closed after the beginning of the observation period (it had a closing date after 2012-1-1). Additionally, only cards opened before the end of the observation period (the so-called REFERENCE DATE of 2012-12-31) were retained. Initially, cards open within the observation period were flagged using a Modeler Derive node as shown in Figure 6.3.

Then, only cards flagged as OPEN WITHIN were included in the modeling file. Cards involuntarily closed at any time were also discarded before the aggregation through a Modeler Select node (Figure 6.4).

Analysts also wanted to monitor the number of open cards per customer at different time points. They achieved this with a series of flags and more specifically:

• Open cards at the beginning of the observation period (2012-1-1, OPEN_AT_START field)
• Open cards at the end of the observation period (2012-12-31, OPEN_AT_END field)
• Open cards at the end of the latency period (2013-3-1, OPEN_AT_LATENCY)
• Open cards at the end of the event outcome period (2013-7-1, OPEN_AT_EVENT_OUTCOME_PERIOD field)

The flags were created with Modeler Derive nodes as sown in Figure 6.5.

These series of flags enabled the analysts not only to capture the trend of card ownership for each customer but also, as we’ll see in the next paragraphs, facilitated the identification of churners and the definition of the target field.

Furthermore, new fields were constructed which summated individual monthly fields for each card and summarized the total spending amount, the number of transactions, and the balances over the 12 months of the observation period. Figure 6.6 displays the Modeler Derive node for the calculation of the total number of transactions in the observation period.

These preparatory tasks were followed by the aggregation of the information per customer. Sums of spending amount, number of transactions, and balances were computed for each customer, and the card-level data were replaced with customer data form that point on. The most recent transaction date (maximum of LAST_TRX_DATE field) of all cards was calculated to enable the subsequent calculation of spending recency for each customer. Additionally, the first card opening (minimum of OPENING_DATE field) was derived to enable the calculation of the tenure for each customer. The Modeler Aggregate node was applied for aggregating the information at customer level as displayed in Figure 6.7.

6.4.2 Enriching customer data

The data preparation procedure was continued with the enrichment of customer data with usage KPIs.

The derived fields included:

• Customer tenure

  The tenure (months on books) of each customer (TENURE_CUSTOMER field). A Modeler Derive node and a date_months_difference function was used to calculate the months of relationship with each customer, based on the opening date of each customer’s first card (Figure 6.8).

• Monthly average spending amount

  The monthly average spending amount (AVG_SPENDING_AMOUNT field) of each customer over the 12 months of observation period. A Modeler Conditional Derive node was used to divide the total spending amount with the customer tenure, for new customers, or 12 months, for old customers as shown in Figure 6.9.

• Spending recency and frequency

  The spending recency and frequency were considered as churn predictors with potential predictive efficiency and were also derived. The spending recency (SPENDING_RECENCY field) was computed as the time from the last spending transaction (LAST_TRX_DATE_Max) to the last day of the observation period (REFERENCE DATE field) as shown in Figure 6.10.

  The spending frequency (SPENDING_FRECENCY field) was calculated as the percentage of the 12 months of the observation period with spending.

• Spending Deltas

  The average spending amount and spending frequency were also calculated for the last 4 months of the observation period, the 4 months nearest to potential churn. Deltas were then derived to denote changes during these 4 months. The idea was to capture a spending decline, something like a negative slope in spending, which might signify a churn prospective. The delta for spending amount (DELTA_SPENDING_AMOUNT field) was calculated as the signed ratio (±):

  

  It denotes the relative percentage increase or decrease of spending during the most recent months of the observation period. A Modeler Conditional Derive node was used for its computation as sown in Figure 6.11.

• Monthly average number of transactions and average transaction amount

  In a way analogous to the one for spending amount, the monthly average number of transactions (AVG_TRX field) and the respective delta (DELTA_TRX field), the change in the most recent observation period, were also derived. An additional KPI calculated denoted the average amount per spending transaction (AVG_TRX_AMOUNT field).

• Monthly average balances and balances frequency

  Concerning balances, monthly average balances (AVG_BALANCES field), and the ratio of months with balances (BALANCES_FREQUENCY field) were also derived.

• Limit ratios

  Also, simple ratios of (monthly average) spending to credit limit (SPENDING_LIMIT_RATIO field) and (monthly average) balances to limit (LIMIT_USAGE) were constructed. The last KPI denotes limit usage and is a good indicator for identifying customers that seem to have problems with their credit limit.

• Card ownership trends

  The ownership flags computed before the actual aggregation of data enabled the comparison of the number of open cards each customer had at different time points and the capturing of ownership trends. Specifically, the indicator END_START_CARDS_DELTAS denotes the signed (±) difference in the number of open cards a customer had at the end versus the beginning of the observation period. The second trend indicator END_WITHIN_CARDS_DELTAS captures the change in the number of open cards a customer had at the end of the observation period compared to all the cards he/she had during the whole observation period (Figure 6.12).

6.4.3 Defining the modeling population and the target field

After enriching predictors, it was time for selecting the modeling population and defining the target field. The approach followed was to include all customers with at least one open card at the end of the observation period (the so-called REFERENCE DATE, 2012-12-31). Remember that in deployment the REFERENCE DATE will correspond to the present. Therefore, in the scoring phase, the generated model will calculate propensities for all active customers at present. The modeling population was selected based on the number of open cards at the end of the observation period through a Modeler Select node (Figure 6.13).

Moreover, since latency period has been reserved, customers that have churned within 2 months after the observation period have been discarded. We want the model to be trained on churn cases for which there is a chance of retention. Immediate churners will probably leave during the campaign preparation. Even if the bank manages to contact them before their formal attrition, it’d be quite unlikely to change their decision. As shown in Figure 6.14, short-term churners have been discarded based on the number of open cards at the end of the latency period (OPEN_AT_LATENCY_Sum field).

Finally, the target field and population have been defined. Full churners, that is, customers without open cards at the end of the event outcome period (OPEN_AT_EVENT_OUTCOME_PERIOD_Sum field), have been flagged as churners (Figure 6.15, Modeler Flag Derive node).

6.6.1 Applying a Split (Holdout) validation: splitting the modeling dataset for evaluation purposes

A very common pitfall in data mining is to build a propensity model which memorizes the patterns of the specific training dataset and only performs well for the specific records. To avoid this pitfall, it’s highly advisable to train the model in one dataset and test it in a different one. In our example, a Split (Holdout) validation was applied through a Partition node (Figure 6.17) which split the modeling dataset into training and testing datasets through random sampling. The training partition size was set to 70%. Therefore, the 70% of the initial 22.693 records were used for the training of the model. The remaining 30% of the records were allocated at the testing partition and were used for assessing the model accuracy.

Additionally, a random seed was specified in order to “stabilize” the underlying random sampling procedure and ensure the same partitioning on every model run.

6.6.2 Balancing the distribution of the target field

Since only a small percentage of the modeling population (almost 7%) belonged to the target group (class of churners), the analysts balanced the distribution of the outcome field by applying a Modeler Balance node. The balance technique is often applied in propensity modeling in the case of rare target events. Many propensity models do not behave well in the case of imbalanced distribution of the target field. The weighting of outcomes corrects distribution imbalances and facilitates the identification of more refined patterns.

The Balance node (Figure 6.18) applies a stratified random sampling on the training dataset. A sample ratio of 1.0 was specified for the records belonging to the class of churners. Thus, all churned customers were retained in the training dataset. On the other hand, a sample ratio of 0.23 was specified for nonchurners; thus, about one every four of nonchurners was retained.

This disproportionate sampling, called undersampling, reduced the proportion of the more frequent class and boosted the percentage of churners in the training dataset from the initial 7% to approximately 25% after balancing. The balance effect and the final distribution of the CHURN field in the training partition is shown in Figure 6.19. As you can see, a more balanced distribution of 25–75% was achieved by reducing the number of nonchurners.

The testing partition was not balanced, allowing a simpler and more straightforward evaluation of the model results in the next steps.

A Cache was enabled at the Balance node, saving the data that pass through the node in a temporary folder on disk. This option ensured that the Balance sampling was stored and the training partition was not altered with each data pass.

6.6.3 Setting the role of the fields in the model

Each propensity model tries to learn the input data patterns associated with the occurrence of the target event. In Modeler, the role of each field, whether to be used as an input (predict) or as an output (be predicted), is set with a Type node (Figure 6.20).

In our case study, the CHURN field was the target field to be predicted and was set with an output role (Direction Out). The fields denoted as predictors in Table 6.2 were designated as inputs (Direction In). The rest of the fields were set with direction None and were omitted from model training.

6.6.4 Training the churn model

After selecting the predictors, it was time for model training. The Auto-Classifier node enabled the building and the comparison of multiple models in a single modeling run. As shown in Figure 6.21, the first criteria selected for ranking the models was the Lift at the top 5 percentile. Therefore, the Lift of each model, the increase in predictive performance compared to the baseline “model” of random selection, was calculated for the top 5% of customers with the highest churn scores and used for comparing and ranking.

Three Decision Tree models and an SVM model were selected for training (Figure 6.22). More specifically, the models built were CHAID, boosted C5.0, C&R Tree, and support vector machine (SVM).

The main model parameters specified for each model are displayed in Table 6.3.

6.9.1 Loading the data and setting the roles of the attributes

After loading the modeling dataset, analysts retained in the process only the target field (the label field, in RapidMiner’s terminology) and the predictors listed in Table 6.2. A Select Attribute operator was used to keep those fields and filter out the rest which did not contribute to the model.

In the next step of the process, a Set Role operator was used to assign a label (target) role to the CHURN field. The rest of the fields kept their default regular role and participated as predictors in the subsequent propensity model. The Set Role settings are presented in Figure 6.35.

6.9.2 Applying a Split (Holdout) validation and adjusting the imbalance of the target field’s distribution

Before training the classification model, a Split (Holdout) validation method has been applied through a Split Validation operator. This operator partitioned the modeling dataset into training and testing parts of 70 and 30%, respectively, as shown in Figure 6.36.

The partitioning was based on random sampling. A split ratio of 0.7 randomly assigned the 70% of the training dataset to the training sample and the remaining records to the testing sample. The Split Validation operator, is comprised of two subprocesses (Figure 6.37).

The left subprocess corresponds to the training sample and covers the model training phase. The right subprocess corresponds to the testing dataset and wraps the evaluation actions.

6.9.3 Developing a Naïve Bayes model for identifying potential churners

The balanced training instances were then fed into a Naïve Bayes model with Laplace correction and the model was trained. The model validation is presented in Section 6.9.4.

6.9.4 Evaluating the performance of the model and deploying it to calculate churn propensities

As mentioned previously, the model evaluation procedure is executed in the right subprocess of the Split Validation operator. The subprocess receives the testing sample which is then passed through a Performance (Binomial Classification) operator to calculate a set of performance measures. Evaluation measures require scored cases. Therefore, before the Performance operator, an Apply Model operator has been inserted to classify the unseen records of the testing sample and score (label) the cases according to the generated model.

The Confusion matrix of the model is presented in Figure 6.38. As we can see, 323 churners and 4422 nonchurners were classified correctly (TP, true positives, and TN, true negatives, respectively). The overall accuracy was 69.7%. That means that 69.7% of the testing cases were correctly classified, while the model failed to classify correctly the 30.3% of the cases (misclassification rate). Since the data were imbalanced, it is recommended to focus on more adequate measures such as Sensitivity, Specificity, F-measure, and Area under the ROC curve.

The ROC curve plots the model’s sensitivity in the y-axis against the (1-specificity) values in the x-axis, in simple words the trade-off between the true-positive rate (proportion of churners classified correctly) and the false-positive rate (proportion of nonchurners incorrectly labeled as churners) at different propensity cutoffs. The ROC curve of the Naïve Bayes model is depicted in Figure 6.39. It shows a steep increase at the left of the x-axis (higher cutoffs) where the curve substantially diverges from the diagonal line of random guessing, indicating acceptable model performance. The model presents a decent Area under the ROC curve measure of 0.75 and an F-measure of 23.85%.

The model’s Gains chart, created through a Create Lift Chart operator, is displayed in Figure 6.40. It is based on the 6808 validation cases (30% validation sample) and ranks customers into 20 equal-sized bins (tiles) according to their churn propensity (note that in order to apply the Lift chart on the testing cases, a Remember operator was used inside the testing subprocess to store the Lift chart, and then a Recall operator was used to restore it).

The model’s Lift was 3.5 at the top 5% percentile. Specifically, the top 5% percentile was comprised of the 341 customers with the highest estimated churn propensities (341/6808). Of those customers, 81 were actual churners. Thus, 17.4% of all churners (81/466) was included in the top 5% percentile, yielding a Lift of 3.5.

In the deployment phase, customers with open cards at the end of June 2013 were scored with an Apply Model operator. A sample of scored records is shown in Figure 6.41.

Three new estimated fields were derived by the model for each case. The model’s predicted class (prediction(CHURN) field) along with the prediction confidence for each of the two classes. The confidence(T) field is the estimated churn propensity and indicates the churn likelihood for each customer.

6.10.1 Building the model using the Classify Wizard

In the first step of the Classify Wizard, the source training data had been selected and loaded as shown in Figure 6.42. In this example, a specific data range of the active datasheet was specified as the modeling dataset. In general, training data can also be loaded using an external data source and/or an SQL query.

The second step of the wizard involved the selection of the predictors (“Inputs”) and the target (“Column to analyze”) (Figure 6.43). The complete catalogue of predictors is listed in Table 6.2.

6.10.2 Selecting the classification algorithm and its parameters

After experimentation with different classification algorithms, the modeling team decided to proceed with a Decision Tree algorithm with the default BDE (Bayesian Dirichlet Equivalent with Uniform prior) attribute selection method (“Score method”). The selected algorithm and its parameters were defined in the “Parameters…” menu as displayed in Figure 6.44.

The default “Split method” of “‘Both,” which optimally combines multiway and binary splits of the predictors, was applied. To obtain a more refined solution and a tree with more levels, the default “Complexity penalty” value was decreased to 0.1. Similarly, the “Minimum Support” value (minimum acceptable size of the leaf nodes) was set to a minimum of 10 instances.

6.10.3 Applying a Split (Holdout) validation

In order to avoid an optimistic validation of the model performance, it was decided to apply a Split (Holdout) validation. A random percentage of 30% of the training instances had been hold out to be used as the testing dataset as shown in Figure 6.45.

In the last step of the wizard, the created mining structure, including the model and the testing (holdout) dataset, was stored as shown in Figure 6.46. The stored testing dataset was subsequently used for model validation.

6.10.4 Browsing the Decision Tree model

The generated Decision Tree model is presented in Figure 6.47. By browsing the tree and by following the recursive partitioning and its branches, from the root down to the leaf nodes, we can gain insight on the data patterns associated with increased churn rate. In each node, a darker background color designates more dense concentrations of churners.

The spending frequency was selected for the first partition. Remember that the same predictor was selected for the first split in the Modeler CHAID tree. By studying the tree, we can infer that churn seems to be associated with low spending frequency and a decrease in transactions.

6.10.5 Validation of the model performance

The model validation was performed using the Classification Matrix Wizard and the Accuracy Chart Wizard of the Data Mining for Excel.

The Classification Matrix wizard evaluates the accuracy and the error rate of the classification model through a Confusion (misclassification) matrix. In the first step of the Classification wizard, the stored BDE Decision Tree model was selected for the validation as shown in Figure 6.48.

The target field as well as the desired format of the results (in counts and/or percentages) was specified in the second step of the wizard as shown in Figure 6.49.

Obviously, since a Holdout validation technique had been applied, the model was selected to be tested on the testing (holdout) dataset of the mining structure (Figure 6.50).

The accuracy and the error rate of the Decision Tree model are presented in Table 6.4. The model presented an overall accuracy of 93.11% and an error rate of 6.89%.

The detailed Confusion matrix of the model is presented in Table 6.5. The actual classes are listed in the columns of the matrix and are cross-examined with the predicted classes which are listed in the rows of the matrix.

The model classified all actual churners as nonchurners because the estimated churn propensities were lower than the threshold value of 50% for classifying an instance as churner. This was an anticipated result due to the sparse target class and the imbalanced distribution of the target, and it does not mean that the model was not useful. The discrimination effectiveness of the model was the main criterion for the evaluation of its usability, and this effectiveness was examined with a Gains chart, produced through the Accuracy Chart wizard (Figure 6.51).

The Gains chart of the model is displayed in Figure 6.52.

Table 6.6 lists the cumulative Gain %, that is, the cumulative percentage of actual churners, for the top 20 churn propensity percentiles.

A random 5% sample would have included 5% of the total actual churners. After ranking the customers according to their estimated churn propensities, the relevant proportion in the top 5% percentile has been raised to 20.68%, yielding a Lift of 4.14 (20.68/5).

Figure 6.53 presents the cumulative distribution of churners and nonchurners across the estimated propensity percentiles. The maximum separation (equal to the KS statistic) was 41.17%, and it was observed in the 32% percentile.

6.10.6 Model deployment

In the deployment phase, the “Query” wizard was used to deploy the stored and validated model on the scoring dataset which contained customers active at the time of scoring. The model estimates which were selected to be produced included the predicted class (based on the 50% propensity threshold), the prediction confidence (the estimated probability of the prediction), and the churn propensity (the estimated probability of churn) as shown in Figure 6.54.

A screenshot of a sample of scored customers and their model derived estimates is shown in Figure 6.55.