3.3.1 Definition of the business objective

Following the understanding of the business situation, the business objective should be clearly set as it will define all the next steps of the behavioral segmentation procedure. Then it should be translated to a data mining goal. The translation of a marketing objective to a data mining goal is not always simple and straightforward. However, it is one of the most critical issues, since a possible misinterpretation can result in the failure of the entire data mining project.

It is very important for the analysts involved to explain from the beginning to everyone involved in the data mining project the anticipated final deliverables and to make sure that the relevant outcomes cover the initially set business requirements.

3.3.2 Design of the modeling process

In this phase, crucial decisions should be made, concerning, among other issues, the segmentation perimeter and level as well as the appropriate segmentation dimensions by which customers will be grouped.

3.4.1 Investigation of data sources

The available data sources should be evaluated in terms of accessibility and validity. This phase also includes initial data collection and exploration in order to understand the available data.

3.4.2 Selecting the data to be used

The next step of the procedure involves the definition of the data to be used for the needs of the analysis.

The selected data should cover all the segmentation dimensions as all the additional customer information that will be used for profiling the revealed segments.

Cluster inputs can be numeric as well as categorical. The categorical inputs are handled by the cluster algorithms with appropriate distance measures or internal preprocessing encoding. However, in behavioral segmentation, a general recommendation is to avoid using inputs of mixed scale in the same model as categorical inputs can be overweighted in the formation of the clusters.

Likewise, demographical variables should be avoided in behavioral segmentation projects. Mixing behavioral and demographical information may result in unclear and ambiguous behavioral segments since two customers with identical demographical profile may have completely different behaviors.

For example, imagine the case of a father that has activated a mobile phone line for his teenage son. In a behavioral segmentation, based only on behavioral data, this line would most likely be assigned to the “young—SMS users” segment, along with other teenagers and young techno-fun users. Therefore, we might expect some ambiguities when trying to examine the demographical profile of the segments. In fact, this hypothetical example also outlines why the usage of demographical inputs should be avoided when the main objective is behavioral separation.

Only variables relevant with the specific business objective should be included in the model. Mixing all available attributes in an attempt to build a “silver-bullet” total segmentation that will cover all the aspects of a customer relationship with the organization (e.g., usage and payment behavior) usually leads to unclear segments with poor actionability. It is suggested to develop different segmentation schemes independently and then combine them to build differentiated marketing strategies.

3.4.3 Data integration and aggregation

The initial raw data should be consolidated for the creation of the final modeling dataset that will be used for the identification of the segments. This task typically includes the collection, filtering, merging, and aggregation of the raw data based on the modeling process designed.

For behavioral segmentation applications, a recent “view” of the customers’ behavior should be constructed and used. This “view” should summarize the behavior of each customer by using at least 6 months of observation data.

As outlined previously, the aggregation level of the modeling dataset should correspond to the wanted segmentation level. If the goal, for instance, is to segment bank customers, then the final dataset should be at a customer level. If the goal is to segment telephone lines (MSISDNs), the final dataset should be at a line level. To put it in a simple way, clustering techniques reveal natural groupings of records. So no matter where we start from, the goal is the construction of a final, one-dimensional, flat table, which summarizes behaviors at the selected granularity.

This phase is concluded with the retrieval and consolidation of data from multiple data sources (ideally from the organization’s mining datamart and/or Marketing Customer Information File) and the construction of the modeling dataset.

3.4.4 Data exploration, validation, and cleaning

A critical issue for the success of any data mining project is the validity of the used data. The data exploration and validation process includes the use of simple descriptive statistics and charts for the identification of inconsistencies, errors, missing values, and outlier (abnormal) cases. Outliers are cases that do not conform to the patterns of “normal” data. Various statistical techniques can be used in order to fill in (impute) missing or outlier values.

Outlier cases in particular require extra caution. Outliers are records with extreme values and unusual data patterns. They can be singled out by examining data through simple descriptive statistics or specialized algorithms. Outliers deserve special investigation as in many cases they are what we are looking for: exceptionally good customers or, at the other end, fraudulent cases. But they may have a negative impact on clustering models.

They can misguide the clustering algorithm and lead to poor and distorted results. In many cases, the differences between “outlier” and “normal” data patterns are so large that they may mask true differences among the majority of the “normal” cases. As a result, the algorithm can be misled to a poor solution that merely separates outliers. Consequently, the clustering model may come up with a poor solution consisting of one large cluster of “normal” behaviors and many very small clusters representing the unusual data patterns. This analysis may be useful, for instance, in fraud detection, but certainly, it is not appropriate in the case of general purpose segmentation. A recommended approach for an enriched general purpose solution would be to identify and exclude outliers in advance. Another approach could be to run an initial clustering solution, identify small outlier clusters, and then rerun and fine-tune the analysis after excluding the outlier clusters.

3.4.5 Data transformations and enrichment

This phase deals with the transformation and enrichment of the modeling dataset.

A significant data preparation step in segmentation models is the standardization (normalization) of the inputs so that their values are in similar scales. Quite often, the cluster inputs are measured in different scales. Since clustering models take into account the differences between records, the differences in the measurement scales can lead to biased clustering solutions simply because some fields might be measured in larger values. Fields measured in larger values have increased variability. If used in their original scale, they will dominate the cluster solution. Thus, a standardization process is necessary in order to bring fields to comparable scales and ensure that fields with larger values do not determine the solution.

The two most common standardization methods include the z-score and the 0–1 (or min–max or range) approaches. In the z-score approach, the standardized field is created as below:

The derived field has a mean of 0 and a standard deviation of 1.

The min–max (typically 0–1) approach rescales all values in the specified range as follows:

By the term data enrichment, we refer to the construction of informative indicators, key performance indicators (KPIs), based on the original inputs. The original inputs are combined, and functions such as ratios, percentages, and averages are applied to derive fields which better summarize the customer behavior and convey the differentiating characteristics of each customer.

This is a critical step that heavily depends on the expertise, experience, and business “imagination” of the project team since the construction of an informative list of inputs can lead to richer and more refined segmentations.

3.4.6 Input set reduction

The data preparation stage is typically concluded with the application of an unsupervised data reduction technique such as PCA/Factor Analysis. These techniques reduce the dimensionality of the inputs by effectively replacing a typically large number of original inputs with a relatively small number of compound scores, called factors or principal components. They identify the underlying data dimensions by which the customers will be segmented. The derived scores are then used as inputs in the clustering model that follows. The advantages of using a data reduction technique as a data preprocessing step include:

• Simplicity and conceptual clarity. The derived scores are relatively few, interpreted, and labeled. They can be used for cluster profiling to provide the first insight on the segments.
• Standardization of the clustering inputs, a feature important in order to yield an unbiased solution.
• Ensuring that all the data dimensions contribute equally to the formation of the segments.

3.6.1 “Technical” evaluation of the clustering solution

The internal cohesion and the separation of the clusters should be assessed with the use of descriptive statistics and specialized technical measures (such as standard deviations, interclass and intraclass distances, Silhouette coefficient, etc.). Additionally, data miners should also examine the distribution of the customers in the revealed clusters as well as the consistency of the results in different datasets. All these tests assess the segmentation solution in terms of “technical” adequacy. Additionally, the segments should also be assessed from a business perspective in terms of actionability and potential benefits. To facilitate this evaluation, a thorough profiling of the segments’ characteristics is needed.

Analysts should examine and evaluate the revealed cluster solution and assess, among others, the number and the relative size of the clusters and their cohesion and separation. A good clustering solution contains tightly cohesive and highly separated clusters. More specifically, the solution, through the use of descriptive statistics and specialized measures, should be examined in terms of:

1. The number of clusters and the size of each cluster

   The cluster examination normally begins by looking at the number of revealed clusters. In general purpose clustering, analysts typically expect a rich but manageable number of clusters. Analysts should look at the number of records assigned to each cluster. A large, dominating cluster which concentrates most of the records may indicate the need of further segmentation. On the other end, a small cluster with few records merits special attention. If considered as an outlier cluster, it could be set apart from the overall clustering solution and studied separately.

2. Cohesion of the clusters

   A good clustering solution is expected to be comprised of dense concentrations of records around their centroids. Large dispersion values indicate nonhomogeneous groupings and suggest further partitioning of the dataset. A number of useful statistics can be calculated to summarize the concentration and the level of internal cohesion of the revealed clusters such as:

   • Standard deviations and pooled standard deviations of the clustering fields. Data miners should start by examining the standard deviations of the clustering fields for each cluster, hoping for small values which indicate a small degree of dispersion. The pooled standard deviation of a clustering field is the weighted (according to each cluster’s size) average of the individual standard deviations for all clusters. Once again, we anticipate low variability and small values which denote increased cohesion.
   • Maximum (Euclidean) distance from the cluster center (centroid). Another statistic that summarizes the degree of concentration of each cluster is the maximum distance from the cluster center, the cluster radius. In a way, it represents the range of each cluster since it denotes how far apart lies the remotest member of the cluster.
   • Analysts should also evaluate the within-cluster distances (within-cluster variation): the distances between the members of a cluster and their cluster centroid. They could average these distances over all members of a cluster and look for clusters with disproportionately large average distances. These clusters are candidate for further segmentation. A technical cluster cohesion measure which is based on the (squared Euclidean) distances between the data points and their centroids is the sum of squares error (SSE). In order to compare between models, we can use the average SSE calculated as follows:
     
     where is the centroid of cluster i, a data point/record of cluster i, and N the total cases. A solution with smaller SSE is preferred.
3. Separation of the clusters

   Analysts also hope for well-separated (well-spaced) clusters.

   A good way to quantify the cluster separation is by constructing a proximity matrix with the distances between the cluster centroids. The minimum distance between clusters should be identified and assessed since this distance may indicate similar clusters that may be merged.

   Analysts may also examine a separation measure named sum of squares between (SSB) which is based on the (squared Euclidean) distances of each cluster’s centroid to the overall centroid of the whole population. In order to compare between models, we can use the average SSB calculated as follows:

   

   where is the centroid of cluster i, the coverall centroid, N the total cases, and N_i the number of cases in cluster i. The SSB measures the between-cluster distances. The higher the SSB, the more separated the derived clusters are.

A combined measure that assesses both the internal cohesion and the external separation of a clustering solution is the Silhouette coefficient which is calculated as follows:

1. For each record i in a cluster, we calculate a(i) as the average (Euclidean) distance to all other records in the same cluster. This value indicates how well a specific record fits a cluster. To simplify its computation, a(i) calculation may be modified to record the (Euclidean) distance of a record from its cluster centroid.
2. For each record i and for each cluster not containing i as a member, we calculate the average (Euclidean) distance of the record to all the members of the neighboring cluster. After doing this for all clusters that i is not a member of, we calculate b(i) as the minimum such distance in terms of all clusters. Once again to ease computations, the b(i) calculation can be modified to denote the minimum distance between a record and the centroid of every other cluster.
3. The Silhouette coefficient for the record i is defined as
   

   The Silhouette coefficient varies between −1 and 1. Analysts hope for positive coefficient values, ideally close to 1, as this would indicate a(i) values close to 0 and perfect internal homogeneity.

4. By averaging over the cases of a cluster, we can calculate its average Silhouette coefficient. The overall Silhouette coefficient is a measure of the goodness of the clustering solution, and it can be calculated by taking the average over all records/data points. An average Silhouette coefficient greater than 0.5 indicates adequate partitioning, whereas a coefficient less than 0.2 denotes a problematic solution.

Other popular cluster performance measures include the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). The BIC measure is also used by Modeler’s TwoStep algorithm for determining the optimal number of clusters. It assesses the goodness of fit of the solution, that is, how well the specific data are fit by the number of clusters.

Technical measures like the ones presented in this paragraph are useful, but analysts should also try to understand the characteristics of each cluster. The understanding of the clusters through profiling is required to take full advantage of them in subsequent marketing activities. Moreover, it is also a required evaluation step before accepting the solution. This process is presented in the next paragraph.

3.6.2 Profiling of the revealed segments

A profiling phase is typically needed in order to fully interpret the revealed segments and gain insight on their structure and defining characteristics. Profiling supports the business evaluation of the segments as well as the subsequent development of effective marketing strategies tailored for each segment.

Segments should be profiled by using all available fields as well as external information. Derived clusters should be interpreted and labeled according to their differentiating characteristics, and consequently, the profiling phase inevitably includes going back to the inputs and determining the uniqueness of each cluster in respect to the clustering fields.

The description of the extracted segments typically starts with the examination of the centroids’ table. A cluster centroid is defined by simply averaging all the input fields over all cluster members. The centroid can be thought of as the prototype or the most typical representative member of a cluster. Cluster centers for the simple case of two clustering fields and two derived clusters are depicted in Figure 3.17.

In the general case of M clustering fields and N-derived clusters, the table summarizing the cluster centers would have the form of Table 3.2.

Analysts should individually check each cluster and compare the means of the input attributes with the grand means, looking for significant deviations from the “overall”/the “typical” behavior. Therefore, they should search for clustering fields with relatively low or high mean values in a cluster.

The profiling phase should also examine the clusters in respect to “external” fields, not directly participated in the cluster formation, including KPIs and demographical information of interest. Normally, the cluster separation is not only limited to the clustering fields, but it is also reflected in other attributes. Therefore, data miners should also describe the clusters by using all important attributes, regardless of their involvement in the cluster building, to fully portray the structure of each cluster and identify the features that best characterize them.

This profiling typically involves the examination of the distribution of continuous profiling fields across clusters. Profiling attributes can be normalized (standardized) in order to diminish the measurement scale differences and more clearly portray the structure of the clusters. For categorical attributes, the procedure would be analogous, involving comparisons of counts and percentages. The scope again is to uncover the cluster differentiation in terms of the categorical fields.

Finally, classification models can augment the reporting and visualization tools in the profiling of the segments. The model should be built with the cluster membership field as the target and the profiling fields of interest as inputs. Decision Trees in particular due to the intuitive format of their results are typically used to describe the segment profiles.

3.6.3 Using marketing research information to evaluate the clusters and enrich their profiles

Marketing research surveys are typically used to investigate the needs, preferences, opinions, lifestyles, perceptions, and attitudes of the customers. They are also commonly used in order to collect valid and updated demographical information. It is strongly recommended to combine the data mining-driven behavioral segments with market research-driven demographical and needs/attitudinal segments. While each approach helps in the understanding of certain aspects of the customers, combining them provides deeper insight on different customer typologies.

For instance, provided a behavioral data mining segmentation has been implemented, random samples can be extracted from each segment, and through surveys and/or qualitative research and focus group sessions, valuable insight can be gained concerning each segment’s needs and preferences. Alternatively, the data mining and the market research approaches can be implemented independently and then cross-examined, not only as a mean for evaluating the solutions but also in order to construct a combined and integrated segmentation scheme which would provide a complete view of the customers.

In conclusion, combining data mining and market research techniques for customer segmentation can enable refined subsequent marketing strategies, based on thorough understanding of customer behavior and needs, as shown in the Figure 3.25.

Consider customers belonging to the same behavioral segment but having diverse needs and perceptions. This information can lead to tailored marketing strategies within each behavioral segment.

3.6.4 Selecting the optimal cluster solution and labeling the segments

Although all the aforementioned evaluation measures make mathematical sense and they can provide valuable help in the identification of the optimal solution, data miners should not solely base their decision on them. Many algorithms, such as Modeler’s TwoStep cluster, Excel’s clustering, and RapidMiner’s X-means, automatically propose an optimal number of clusters based on technical measures and heuristics. Once again, these proposed solutions should not be blindly accepted. A clustering solution is justified only if it makes sense from a business point of view. Actionability, potential business value, interpretability, and easiness of use are factors hard to quantify and, in a way, subjectively measured. However, they are the best benchmarks for determining the optimal clustering solution. The profiling of the clusters and the identification of their defining characteristics are an essential part of the clustering procedure. It should not be considered as a postanalysis task but rather as an essential step for assessing the effectiveness of the solution.

As with every data mining model, data miners should try many different clustering techniques and compare the similarity of the derived solutions before deciding on the one to choose. Different techniques that generate analogous results are a good sign for the identification of a general and valid solution. As in any other data mining model, clustering results should also be validated by applying the model in a disjoint dataset and by examining the consistency of the results.

The profiling and interpretation process ends with the labeling of the identified segments with names that appropriately designate their unique characteristics. Each segment is assigned an informative and revealing name, for instance, “Business Travelers,” instead of “Segment 1.” The naming of the segments should take into account all the profiling findings. These names will be used for communicating the segments to all business users and for loading them to the organization’s operational systems.

3.7.1 Building the customer scoring model for updating the segments

The deployment procedure should enable the customer scoring and update of the segments. It should be automated and scheduled to run on a frequent base to enable the monitoring of the customer base evolvement over time and the tracking of the segment migrations. Moreover, as nowadays markets change very rapidly, it is evident that a segmentation scheme can become outdated within a short time. Refreshment of such schemes should be made quite often. That’s why the developed procedure should also take into account the need of possible future revisions.

3.7.2 Distribution of the segmentation information

Finally, the deployment procedure should also enable the distribution of the segmentation information throughout the enterprise and its “operationalization.” Therefore, it should cover the upload of the segmentation information in the organization’s databases and CRM tools in order to enable customized strategies to be applied across all customers’ touch points.

3.7.3 Design and delivery of differentiated strategies

In order to make the most of the segmentation, specialization is needed. Thus, it is recommended to assign the management of each segment to a specific team. Each team should build and deliver specialized marketing strategies tailored to the segment’s needs, in order to improve the customer handling, develop the relationship with the customers, and grow the profitability of each segment. The responsibilities of the team should also include the tracking of each segment with appropriate KPIs and the monitoring of the competitors’ activities for each segment.