16.3.2 Data Extraction

The validity of the results obtained in software analytics projects is limited by the quality of the data extracted. For this reason, accurate data extraction is one of the most important phases in software analytics projects. In every research project, it is important to design the data extraction phase carefully.

The first step during data extraction is the choice of the right requirements for the data that will be extracted. If data requirements are changed after the data extraction phase, all the results of the study might be changed. Similarly, rescheduling extensions to surveys done with industry might be time-consuming.

Automation of the data extraction steps, when possible, reduces the extraction effort in the long term. On the other hand, tracking the history of the data extraction scripts helps in repeating older experiments when necessary. Over the years, we have employed several techniques for quantitative and qualitative data extraction for a given problem [56, 57]. In certain scenarios, we partly automated the qualitative techniques for efficiency. In this section, we describe these techniques and how we customized them for our problems.

16.4.1 Data Visualization

The easiest way to gain an understanding of the data collected from rich and complex software repositories is through visualization. Data visualization is a mature domain, with significant amounts of literature on charts, tables, and tools that can be used for visually displaying quantitative information [66]. In this section, we provide examples of the use of basic charts such as box plots, scatter diagrams, and line charts in our previous analytics projects to better understand the software data.

Box plots are helpful for visualization as they allow one to shape the distribution, central value (median), minimum and maximum values, and quartiles, and give clues about whether the data is skewed or contains noise—that is, outliers. Figure 16.2 illustrates a box plot used to visualize the distribution of a number of active issues owned by developers with different categories (defects found during functional testing, during system testing, and in the field) in a case study concerning a large software organization [61]. During this study, we used box plots to inform developers that the functional testing category dominates the issues owned by developers, and it has a slightly higher median than the system testing and field categories. Furthermore, outliers in each set (e.g., 48 functional testing issues are owned by a single developer) highlight potential noisy instances, or dominance of certain developers regarding issue ownership (Figure 16.3).

A scatter diagram is another visualization technique used to explore the relationship between two variables—that is, two metrics that you want to monitor. During an exploratory study on Eclipse releases [67], we used scatter diagrams to observe the trend between the number of edits of source files (first variable) and the number of days between edits (second variable) for three different file sets—that is, files with beta-release bugs, files with other types of bugs, and files with no bugs. Figure 16.4 shows the scatter diagrams for three file sets of an Eclipse release [67]. It is seen that that there is not a clear trend such as a positive monotonic relationship between the two variables. However, it is clear that files with beta-release bugs are concentrated in smaller regions, with very few edits done frequently (in small time periods). On the basis of these visualizations, we suggested that the developers should concentrate on those files that are not edited too frequently in order to catch beta-release bugs [67].

Other types of charts are also useful for the purpose of visualizing software development, such as effort distribution of developers [53], issue ownership, or developer collaboration [61]. For example, we identified the collaboration network of developers and what factors affected the stability of the team working on a piece of large, globally developed and commercial software [68]. We used line charts to visualize the developer collaboration on the basis of the code that developers worked on together. In another study, we also used line charts to depict the number of issues owned and being fixed by developers on a monthly basis (see Figure 16.3). These charts are very easy to plot, and yet they are informative if they are monitored periodically—that is, every week/month or every sprint in agile practices, or per development team.

16.4.2 Reporting via Statistics

Descriptive statistics such as minimum and maximum data points and mean, median, or variance of software metrics are computed prior to building analytics, since these statistics are informative about the distributional characteristics of metrics, central values, and variability. During an industry collaboration, it is a common practice to compute these descriptive statistics (e.g., [56, 61, 69]) or use visualizations at the beginning of a software analytics project, whereas other techniques such as statistical tests are more useful in the next steps to reveal existing relationships between software metrics or to identify differences in terms of distributions of two or more software metrics.

Below, we summarize a set of statistical analysis techniques that were reported in our previous projects conducted with our industry partners. Detailed descriptions of these techniques and their computations can be found in fundamental books on statistics (e.g., [70, 71]); here we explain the benefits of using these methods on software metrics to identify unique patterns.

Correlation analysis. This type of analysis identifies whether there are statistical relationships between variables, and in turn, it helps to decide on a set of independent variables that would be the input of a predictive analytics. Correlations can be computed among software metrics or between metrics and a dependent variable, such as the number of production defects, the defect category, the number of person-months (to indicate development effort), and issue reopening likelihood. Correlations between two variables can be measured using two popular approaches: Pearson’s product-moment correlation coefficient and Spearman’s rank correlation coefficient.

Pearson’s correlation coefficient is a measure of the linear relationship between two variables, having a value between 1 (strong positive linear relation) and − 1 (strong negative linear relation) [70]. The calculation of this coefficient is based on covariances of two variables, and there are different guidelines for interpreting the size of the coefficient depending on the problem being studied. For example, in empirical software engineering, a correlation coefficient above 75% may be interpreted as a strong relation between the variables, whereas in empirical studies in psychology, a quite small correlation coefficient of 37% may indicate a strong relation [72] .

In a case study concerning a large software organization operating in the telecommunications industry, we studied the linear relationship between confirmation bias metrics representing developers’ thought processes and defect rates using Pearson’s correlation coefficient and found that there is a medium (21%) to strong (53%) relationship between the variables [43]. On the basis of correlation analysis, we subsequently filtered the metrics that were significantly correlated and used the resulting metric set to build a predictive analytics that estimates the number of defects of a software system.

Spearman’s rank correlation coefficient is a nonparametric measure of the statistical relationship between two variables [70]. Like Pearson’s correlation coefficient, it takes values between 1 and − 1, and is appropriate for both continuous and categorical variables. Differently, calculation of Spearman’s correlation coefficient is based on the ranks that are computed from the raw values of the variables, and is independent of the distributional characteristics of the two variables. Thus, the correlation can indicate any monotonic relationship, in contrast to the linear relationship in Pearson correlation.

In a study that investigated the experience of software developers and testers, company size, and reasoning skills of software developers and testers in four to medium-sized to large software organizations [41], we computed Spearman’s rank correlation coefficient to observe the relationship between the number of bugs reported by testers and production defects, and the relationship between developers’ reasoning skills and defect-proneness of the code. Analyses showed that there is a significant (rank coefficient 0.82) positive relation between prerelease and production defects. We concluded that testers may report more bugs than the number that developers fix before each release, and hence, as more bugs are reported, the number of production defects increases.

In another case study concerning a large software development organization, we used Spearman correlation to analyze the relationship between metrics that characterize reopened issues (issues closed and opened again during an issue life cycle) [60]. We found a strong statistical relationship between the lines of code changed to fix a reopened issue and the dependencies of a reopened issue on other issues—that is, the higher the proximity of an issue to other issues, the more lines of code are affected during its fix.

In summary, although these results may sometimes look trivial, they are very influential for software teams, and development leads to observation of the hidden facts about the development process in an organization, and/or confirmation of assumptions that developers usually make in their daily decision making process. On the other hand, these types of correlation analyses show only whether there is a statistical relationship between two variables and the scale of that relationship. To further analyze what type of relationship (e.g., linear or nonlinear) exists between these two variables, scatter diagrams can be used or other statistical tests (e.g., hypotheses testing) may be applied. Below, we provide examples of some of these tests that were used in our previous work.

Goodness-of-fit tests. These tests can be used to compare a sample (distribution) against another distribution or another sample that is (assumed to be) generated from the same distribution. Similarly to correlation computation, there are several tests measuring the goodness of fit, and the appropriate one should be chosen depending on the number of instances in a sample, the number of datasets that will be compared, and the distributional characteristics of the sample. For example, in a comparative study between test-driven development and test-last programming, we used the Kolmogorov-Smirnov test to compare the code complexity and design metrics of a software system that is developed using test-driven development against a normal distribution [73]. The Kolmogorov-Smirnov test is a nonparametric test for checking the equality of two continuous probability distributions or for comparing two samples [71]. Our results [73] show that none of the metrics are normally distributed, and in turn, we applied analytics that are appropriate for samples generated from any type of distribution. In other studies, we applied the same goodness-of-fit test to compare development activities between experienced and novice team members of a software organization [40] or between developers and project managers [39].

In some of our studies, software metrics took categorical values, and hence other goodness-of-fit tests such as the chi-square test were more appropriate to compare the categorical metric values from two or more groups of samples. For example, during a case study concerning a software organization [42], the chi-square test was used to compare the distributions of confirmation bias metrics, which characterize developers’ thought processes and reasoning skills, among developers, testers, and project managers. On the basis of the test results, we stated that reasoning skills are significantly different among the three development roles. Hence, we suggested using these metrics in deciding the assignment of roles and in forming a more balanced structure in the software organization.

Differences between populations. Though goodness-of-fit tests can be used to compare the differences between two or more populations—that is, samples—a better approach is to form a null hypothesis (H₀) and use statistical hypothesis tests to check if the null hypothesis is supported. For example, if we aim to check whether developers write more tests using test-driven development compared with test-last programming, we need to collect two samples that include the number of tests collected from a development activity using test-driven development and test-last programming. It is important that the samples are independently drawn from different populations. Later, we can form a null hypothesis as follows: The difference between the number of tests written by developers using test-driven development and the test-last approach has a mean value of 0.

A test (e.g., t test and the Mann-Whitney U test, also called the Wilcoxon rank-sum test) can reject the null hypothesis, meaning that the mean values are significantly different between two development practices, or it cannot reject the null hypothesis, meaning that the difference of means is not significant enough to make a statistical claim [71]. Hypothesis tests cannot actually prove or disprove anything, and hence they also calculate the power of the significance—for example, a p value of 0.05 tells us that there is a 95% chance of the means being significantly different. As this p value increases, it is more likely that the difference happens by chance.

We have often used hypothesis tests to compare two or more samples representing different software systems, development methods (e.g., Mann-Whitney U test [73]) and/or software teams (e.g., Mann-Whitney U test [41], and Kruskal-Wallis analysis of variance with more than two teams [43]). Alternatively, these tests can be used to compare the performance of two predictive analytics in order to decide which one to choose—for example, t test [26, 49].

In summary, hypothesis testing techniques could be carefully selected by considering the information about sample distributions (e.g., t test for normally distributed samples, and nonparametric tests such as the Mann-Whitney U test for the others) during a descriptive analytics process. We also suggest that descriptive statistics, reporting via visualization or through statistical tests, could guide researchers or data analysts in software organizations in cases where there is rich, multivariate, and often noisy data.

In this section, we present our experiences from multiple industry collaborative projects in terms of predictive analytics. Throughout this section we will review multiple algorithms, from simpler alternatives to more complex ones. However, our main intention is not to repeat the content of commonly used machine learning algorithms, since there are more than enough machine learning books [74–76]. We rather intend to share the lessons learned in the course of applying predictive analytics for more than a decade to industry-academia collaborative projects.

Often, we will see that knowing the learner (aka prediction algorithm) is only the part of the story, and the practitioner needs to be flexible and creative in terms of how he/she uses and alters the algorithm depending on the problem at hand. So, before going any further, just to set the right expectations, we briefly quote a conversation with an industry practitioner. Following the presentation of a predictive analytics model, a practitioner from a large software company asked the following question: “Would your predictive model work under all conditions?” (“Conditions” meaning different datasets and problem types.) The short answer to that questions is: “No.” There is no predictive model that would yield high-performance measures under all the different conditions. Therefore, if you are looking for a silver-bullet predictive model, this section will disappoint you. On the other hand, we can talk about a certain approach to predictive models as well as some likely steps to be followed, which have been applied in real life to a number of industry projects by the authors of this chapter.

Terminology. A predictive model is a specific use of a learner that is often aided by preprocessing and postprocessing steps to improve predictive performance [30]. We will talk about improving the performance in Section 16.5.3. For the now, let us focus our attention on the learners, which are machine learning algorithms that learn the known instances and provide a prediction for an unknown instance. The known instances refer to instances for which we know the dependent variable information—for example, the defect information of software components. These instances are also referred as the “training set.” The unknown instances are the ones for which we lack the dependent variable information—for example, the software components that have just been released, and hence that do not have defect information. These instances are referred to as the “test set.” In the example of defective and nondefective software components, a predictive model is supposed to use the training set and learn the relationship between the metrics defining software components and the defects. Using the relationship learned, the predictive model is expected to provide accurate estimates for the test set—that is, the newly released components.

Go from simple to complex. One misleading approach to predictive modeling is to bluntly use whichever learner we come across. It may be possible to get a decent estimation accuracy with a randomly selected learner, but it is unlikely for such a random learner to be used by practitioners. For example, in a software effort estimation project for an international bank, every month we presented the results of the experiments to the management as well as the software developers [9, 52]. The focus of these presentations was never merely the performance, but how and why the algorithm presented could achieve the presented performance. In other words, merely using a complex algorithm, without a high-level explanation of how and why it applies to the prediction problem at hand, is unlikely to lead to adoption by product groups. Therefore, it is a good idea to start the investigation with an initial set of algorithms that are easy to apply and explain, such as linear regression [38], logistic regression [60, 77], and k-nearest neighbors (k-NN) [62, 78]. The application of such algorithms will also serve the purpose of providing a performance baseline, so that we can see how much added value the more complex additions will bring. These algorithms are relatively easier to understand, and in most of the industry projects in which the authors participated, they proved to have quite good performance [38, 60, 62, 77, 78]. On the other hand, simplicity just for the sake of choosing algorithms that nontechnical audiences can easily understand is also misleading. The simplicity of the algorithm for that purpose should have a minimal effect on the decision making process. If the best performing algorithm turns out to be some complex ensemble of relatively more difficult to understand learners, then so be it. But to make the case for deciding on a complex alternative, we should start simple and make sure that value is added with the complexity.

Linear regression is a predictive model that assumes a linear relationship between the dependent and independent variables [74]:

$\begin{array}{l}\hfill y=X\beta +\epsilon .\end{array}$

The independent variables (defined by X) are multiplied by coefficients (β), and we want to set up the coefficients such that the error (ε) is minimized.

We have used linear regression in various industry settings for regression problems (the problems in which the dependent variable is a continuous value). The focus of a project in which we collaborated with a large telecommunications company was the effects of confirmation bias on the defect density (defect density measured in terms of the defect count, which is a continuous value, and hence a regression problem). We opted to use linear regression in this project as the confirmation bias metrics were observed to be linearly related to defect counts [38]. The observation of a linear relationship can be checked with the R² value (aka the coefficient of determination) as well as by plotting the metric value against the defect count. The R² value measures how much of the response variable variation (around its mean) is explained by a linear model. It can take values from 0 to 1, where 0 means none of the response variable variation is explained by the linear model, whereas 1 means all of the response variable variation is explained by the linear model. Therefore, values close to 1 mean a better model fit. In the confirmation bias project we defined the defect density for each developer group as the ratio of the total number of defective files created/updated by that group to the total number of files that group created/updated. So as to visualize the effect of confirmation bias on software defect density, we constructed a predictive model based on linear regression, with confirmation bias metrics as the predictor (independent) variables and defect density as the response variable. Our results showed that 42.4% of the variability in defect density could be explained by our linear regression model (R² = 0.4243)

Note that in the telecommunications company project, we used a linear regression model so as to predict a continuous value (defect count) [38]. However, not all the predictive problems that we face in real-life predictive analytics involve continuous variables. In the case of discrete dependent variables, we can make use of logistic regression [60, 77]. For example, assume that we know only whether a software module is defective or not (but not the exact defect count), then we can define the discrete classes of “defective” and “nondefective.” Such a grouping would give us a two-class (aka binary) problem (instead of a continuous-variable prediction problem). In the case of two-class classification problems (y_i = [0,1]), logistic regression is a frequently used prediction algorithm. The authors have employed logistic regression in various defect prediction projects [60]. The general logistic regression formula is as follows:

$\begin{array}{l}\hfill \mathrm{Pr}(y_i=1)={\mathrm{logit}}^{-1}(X_i\beta ),\end{array}$

where Pr(y_i = 1) denotes the probability of y_i belonging to class 1, X is the vector of independent variable values, and the β is the coefficient vector. One benefit of this predictive method is that—given the logistic regression provides high accuracy—one can use the corresponding coefficient values in order to see the importance of different input variables. An example application of this approach can be found in one of our projects [60], where we used logistic regression to analyze the possible factors behind issues being reopened in software development. For this purpose, we fit a logistic regression model to the collected issue data, but our main aim was to understand which issue factors are most important (through the use of coefficient values). Therefore, for the analysis of factors leading to reopened issues, logistic regression was an appropriate choice.

Another simple, yet quite successful learner for classification problems is naïve Bayes [33, 62, 78]. Particularly for software defect prediction studies, the authors have observed that naïve Bayes proved to be better than some more complicated rule-based learners (such as decision trees) [62]. As the name of the learner suggests, the naïve Bayes classifier is based on the Bayes theorem, which states that our next observation depends on how new evidence will affect old beliefs:

$\begin{array}{l}\hfill P(H|E)=\frac{P(H)}{P(E)}\prod _i^{}P(E_i|H).\end{array}$

In Equation 16.3, given that we have old evidence E_i and a prior probability for a class H, it is possible to calculate its next (posterior) probability. For example, for a classifier trying to detect defective modules in a piece of software, class H would represent the class of defective modules. Then the posterior probability of an instance being defective (P(H|E)) is the product of the fraction of defective instances P(H)/P(E) and the probability of each observation P(E_i|H).

The success of this learner for defect prediction tasks also paved the way for us to use it in conjunction with other learners. For example, we have employed the k-NN learner as a cross-company filter for naïve Bayes [62]. Before going into how we made use of the k-NN learner as a filter, we briefly explain how it works: k-NN finds the labeled k instances most similar to the test instance. The distance is calculated via a distance function (e.g., Euclidean distance or Hamming distance).

k-NN can also be used for classification [78] and regression [79] problems. For a classification problem, usually the majority vote (i.e., the majority class of k-NN ) is given as the predicted value. For a regression problem, the mean or median of the dependent variable value k-NN is given. Our use of k-NN as a filter in the defect prediction domain—that is, a classification problem [62]—questions whether organizations without data of their own (so-called within-company data) can use the data from other organizations (so-called cross-company data). In our initial experiments we use naïve Bayes as the predictive method and compare the performance when an organization uses within-company data versus cross-company data. The performance results show that the use of cross-company data yields a poor performance. This is understandable as the context of another organization may differ considerably. Then we use k-NN to filter instances from the cross-company data—for example, instead of using all the instances of cross-company data in a naïve Bayes classifier, we first find the closest cross-instances to the test instance (using k-NN). Filtering only the closest instances improves the performance of cross-company data such that it is very close to that of within-company data. The ability to use cross-company data has been quite important in a number of our projects, because our observation is that initially the within-company data will be quite limited at best. In other words, “a practitioner will have to start the analysis with the data at hand.” In such cases, the ability to use cross-company data has helped us provide initial predictions for the within-company test data [52].

Other relatively more complex learners are also frequently employed in software engineering research as predictive methods. Neural networks [49] and decision trees [50, 80] are examples of such learners that were used by the authors. We will not go deeply into the mechanics of these learners; however, we will provide the general idea and possible inherent biases because having an idea of how these algorithms work and their biases aids a practitioner in choosing the right learner for datasets of different characteristics.

Neural networks are known to be universal approximators [81]—that is, they can learn almost any function. Neural networks are defined to be a group of connected nodes, where the training instances are fed into the nodes that form the input layer (see Figure 16.5) and the information is transmitted to hidden layer nodes. During transmission, each connecting edge applies a weight to the number it receives from the previous node. Although there is one hidden layer in Figure 16.5, there may be multiple hidden layers to model more complex functions. Finally, the values are fed into the output layer, where a final estimate for each dependent variable is obtained. Note that instances can be fed into a neural network one by one—that is, the model can update itself one instance at a time as the training instances become available. However, the problem with neural networks is that they are sensitive to small changes in the dataset [82]. Also, overtraining a neural network will have a negative impact for the future test instances that we want to predict. We addressed such issues related to neural networks by arranging the neural networks in the form of an ensemble [49]. Application of our neural network ensemble on software effort estimation data revealed that unlike the problems inherent in a single neural network, having an ensemble of neural networks provides stabler and improved accuracy values [49]. We will discuss ensembles of other learners further in the next section as a way to improve the predictive power of learners.

A capable and commonly used learner type is decision trees [50, 80]. Unlike neural networks that take instances one by one, decision trees require all the data to be available before they can start learning. Decision trees work by recursively splitting data into smaller and smaller subsets on the basis of the values of independent features [83]. At each split the instances that are more coherent with respect to the selected features are grouped together into a node. Figure 16.6 presents a hypothetical decision tree that uses two features, F1 and F2. The way this decision tree would work for a test instance is as follows: If F1 of this feature is smaller than or equal to x, then the decision tree will return the prediction of A. Otherwise, we will go down the node of F1 > x and look at F2. If F2 > y is true for our test instance, then we would predict B, otherwise we would predict C.

Particularly for datasets where instances form interrelated clusters, it is a good idea to try decision tree learners as their assumptions overlap with the structure of the data. Software effort estimation data is a good example of this case, where instances (projects) form local groups that are composed of similar projects. Therefore, we made use of decision trees in different industry projects focusing on estimating the effort of software projects [50, 52, 53]. One of our uses of decision tree learners on software effort data [50] includes the idea of grouping similar instances so as to convert the software effort estimation problem into a classification problem (recall that originally software effort estimation is a regression problem). By using decision trees, we form software effort intervals, where an interval is identified by the training projects falling in the final node of a decision tree. Then the test instance is fed into the decision tree and it finds its final node, where the estimate is provided as an interval instead of a single value.

Things to keep in mind:

1. The learner is only part of the picture. Until now we have reviewed some of the learners that we successfully applied on different real-life software engineering data. We have also seen how these learners can be altered for different tasks—for example, our use of k-NN as an instance-filtering method [62] or the use of decision trees to convert a regression problem into a classification problem [50]. However, the application of the learner on the data is only one part of the picture. There are possible pitfalls we identified over the course of different projects that a practitioner should be aware of when building a predictive model.

2. Building a predictive model is iterative.As you are applying different learners on the data, you should consider including the domain experts (or customers) of the data in the loop. In other words, we recommend discussing the initial results and your concerns with the domain experts. The initial benefit of having the customer in the loop is that you can get early feedback about the quality of the data. For certain instances that your learner yields a poor performance, you can get a clear explanation as to what the cause is. Another benefit will be to identify potential data issues early on. If there is a feature that looks suspicious, the domain experts may give you a clear answer as to whether the data is wrong. For example, in one of the industry projects, for some projects the bug count was surprisingly low, whereas the test cases for these projects were quite high. A domain expert who worked on this project may tell you whether this is indeed the case, or whether in this particular project bugs were not tracked consistently. Then you may decide not to include this unhealthy information in your model, as we did in that particular project.

3. Automation is the key. During any part of the project, you may need to rerun all your experiments. In an industry project, we learned halfway into the project that one of the assumptions we were told for a group of projects would be invalid for another group of projects developed within the preceding year. So we had to repeat all the analyses we had done up to this point for the projects developed within the preceding year. If we had not coded our experiments, it would have taken much time to repeat all the experiments. Therefore, it is beneficial to use machine learning toolkits such as WEKA [76], but it is also critically important to have your experiments coded for reruns of the experiments. Another reason why it is important to code your predictive models is customization. You may want to combine or alter different learners in a specific way, which may be unavailable through the user interface of a toolkit. Coding your experiments may provide you with the flexibility of customization.

16.5.2 Performance Evaluation

Depending on the problem being studied, performance evaluation of predictive analytics can be computed differently. Naturally, the output of a predictive analytics—that is, a prediction model—is compared against the data points in the test set, or against the actual data points after they have been retrieved and stored in data repositories of software organizations. The type of output variables can be categorical or continuous, which determines the set of performance evaluation measures. For instance, in a defect prediction problem, the output of a predictive analytics can take a categorical value (e.g., defect-proneness of a software code module, i.e., package, file, class, method, being defect-free (0), or being defect-prone/defective (1)) or it can take a continuous value ranging from zero to infinity (e.g., number of defect-prone code modules in a software product). For a model with a categorical output, a typical confusion matrix can be used for computing a set of performance measures. Table 16.5 presents a confusion matrix in which the rows represent the actual information about whether the software module is fixed or not because of a defect, whereas the columns represent the output of a predictive analytics.

All performance measures can be calculated on the basis of the four base measures presented in Table 16.5—namely, true positives (TPs), false negatives (FNs), false positives (FPs), and true negatives (TNs). For example, in the context of defect prediction, TPs are the actual number of defective modules that are correctly classified as defective by the model, FNs are the number of defective modules that are incorrectly marked as defect free, FPs are the number of defect-free modules that are incorrectly classified as defective, and TNs are the number of defect-free modules that are correctly marked as defect free by the model. A predictive analytics ideally aims to classify all defective and defect-free modules accurately—that is no FNs and no FPs. However, achieving the ideal case is very challenging in practice.

In our empirical studies, we have used six popular performance measures to assess the performance of a classifier: accuracy, recall (also called TP rate or hit rate or specifically in defect prediction, probability of detection rate), FP rate (also called false alarms in defect prediction), precision, F measure, and balance. The calculations of these six measures are shown below [30, 51, 77]:

$\begin{array}{ll}\hfill \mathrm{ACC}& =(\mathrm{TP}+\mathrm{TN})/(\mathrm{TP}+\mathrm{TN}+\mathrm{FN}+\mathrm{FP}),\hfill \end{array}$

$\begin{array}{ll}\hfill \mathrm{REC}& =\mathrm{TP}/(\mathrm{TP}+\mathrm{FN}),\hfill \end{array}$

$\begin{array}{ll}\hfill \mathrm{PREC}& =\mathrm{TP}/(\mathrm{TP}+\mathrm{FP}),\hfill \end{array}$

$\begin{array}{ll}\hfill \mathrm{FPR}& =\mathrm{FP}/(\mathrm{FP}+\mathrm{TP}),\hfill \end{array}$

$\begin{array}{ll}\hfill F\mathrm{measure}& =2(\mathrm{PREC}\times \mathrm{REC})/(\mathrm{PREC}+\mathrm{REC}),\hfill \end{array}$

$\begin{array}{ll}\hfill \mathrm{BAL}& =1-\sqrt{{(1-\mathrm{REC})}^2+{(0-\mathrm{FPR})}^2},\hfill \end{array}$

where ACC is accuracy, REC is recall, PREC is precision, FPR is the FP rate, and BAL is balance. ACC, REC, PREC, F measure, and BAL should be as close to 1 as possible, while FPR should be close to 0. However, there is an inverse relationship between REC and PREC—that is, it is only possible to increase one (REC) at the cost of the other (low PREC). If these two are prioritized over all performance measures, it is useful to compute F measure, which depicts the harmonic mean between REC and PREC.

Furthermore, there is a positive relationship between REC and FPR—for example, an algorithm rends to increase REC at the cost of a high false alarm rate. BAL is a measure that incorporates the relationship between REC and FPR; it calculates the distance between an ideal case (REC = 1, FPR = 0) and the performance of a predictive analytics.

Achieving high REC and PREC while lowering FPR, and in turn, achieving high BAL and F measure, has been a real challenge for us when building predictive analytics in software organizations. A good balance between these measures needed to be targeted according to the business strategies in these organizations. For example, in software organizations operating in mission-critical domains (e.g., embedded systems), software teams aim to catch and fix as many defects as possible before deploying the production code. Thus, they prioritize achieving a high REC over a low FPR from a predictive analytics. Conversely, in software organizations operating in competitive domains (e.g., telecommunications), reducing costs is the primary concern. Accordingly, they would like to reduce the additional costs caused by high FPR and achieve a balance between REC, PREC, and FPR.

We have preferred to use REC, FPR, and BAL in most of our work since reducing the cost of predictive analytics by lowering FPR has always been the focus of our industry partners. For example, in a case study concerning a software organization operating in the telecommunications domain, we deployed a software analytics for predicting code defects [9]. On the basis of user feedback, we calibrated the analytics to reduce FPR while getting as high REC as possible. The model that was later deployed in the company produced 87% recall and 26% false alarms [84]. In a study concerning a company operating in embedded systems, we built a predictive analytics for catching the majority of code defects at the cost of high false alarm rates [85]. We reported that an ensemble of classifiers was able to catch 82% recall and 35% false alarms. In other case studies, we have studied different techniques for lowering false alarm rates in defect prediction research (e.g., algorithm threshold optimization [29], increased information content of training data [24, 26, 30], and missing data imputation [15]).

For localizing the source of software faults, we built a predictive analytics that extracts unique patterns stored in previous fault messages of a software application and assigns a newly arrived fault message to where it belongs [51]. In the context of fault localization, both REC and PREC are equally important since misclassification of faults has equal costs for both classes. In other words, localizing a fault to component A even though it belongs to component B can be as costly as localizing a fault to component B though it belongs to component A. Therefore, the proposed model in Ref. [51] managed to achieve 99% REC and 98% PREC in the first application, whereas it achieved 76% REC and 71% PREC rates in the second application.

Another predictive analytics was built for a software organization to estimate software classes that need to be refactored on the basis of code complexity and dependency metrics [86]. The proposed model was able to predict 82% of classes that need refactoring (TP rate) using 13% inspection effort by developers.

If predictive analytics produces a continuous output (e.g., total number of defects, project effort in terms of person-months), evaluating its performance requires either a transformation of the output to categorical values or use of other performance measures that are more appropriate for a continuous model output. For example, we built a software effort estimation model that would dynamically predict the interval of a project’s effort—that is, a categorical value that indicates which effort interval is the most suitable for a new project—rather than the number of person-months [80]. Since the output was categorical, performance measures that were used in defect prediction studies—that is, REC, PREC, FPR, and ACC—were used to assess the performance of the proposed effort estimation model.

A more convenient approach is to use other performance measures that are more suitable for evaluating the performance of a model that produces continuous output. Some of these measures that we have extensively used are the magnitude of relative error (MRE), the mean MRE (MMRE), and predictions at level k (PRED(k)). Calculation of each of these measures is shown below:

$\begin{array}{l}\hfill {\mathrm{MRE}}_i=\frac{\mid x_i-\widehat{x_i}\mid }{x_i},\end{array}$

$\begin{array}{l}\hfill \mathrm{MMRE}=\mathrm{mean}({allMRE}_i),\end{array}$

$\begin{array}{l}\hfill \mathrm{PRED}(k)=\frac{100}{N}\sum _{i=1}^N\left\{\begin{array}{l}\text{1 if}{\mathrm{MRE}}_i\le \frac{k}{100},\hfill \\ \text{0 otherwise.}\hfill \end{array}\right.\end{array}$

Ideally, MRE and MMRE should be as low as 0, whereas PRED(k) should be close to 1. Instead of MMRE, the median magnitude of relative error (MdMRE) can abe used, since the mean of a sample is always sensitive to outliers in that sample compared with the median. In the context of software effort estimation, reducing MRE indicates that effort values are estimated with low error rates. As the error rate decreases, PRED increases. The value of k in the PRED calculation is usually selected as 25 or 30, meaning that the percentage of estimations whose error rates are lower than 25% or 30%. PRED might be a better assessment criterion for software practitioners, since it shows the variation of the prediction error—that is, what percentage of predictions achieve a level of error that was set by practitioners. Hence, it implicitly presents both the error rate (k) and the variation of this error among all predictions made by the analytics.

We have used MMRE and PRED(25) measures during our empirical studies in the context of software effort estimation, and we observed that there is not an optimal value for these measures that fits best for all software projects and teams. For example, in one study, one of our colleagues from our research laboratory built different predictive analytics to estimate the project effort in person-months [87]. Two datasets—that is, one public dataset and another dataset collected from different software organizations—were used to train the predictive analytics. The results show that the best model applied to the public dataset produced 29% MMRE, 19% MdMRE, and 73% PRED(30), whereas it produced 49% MMRE, 28% MdMRE, and 51% PRED(30) when applied to the commercial dataset.

In other studies, we have proposed relatively more complex analytics to estimate software projects’ effort in the case of a limited amount of training data. For example, the intelligence in our predictive analytics proposed in Ref. [49] works as follows: it has an associative memory that estimates and corrects the error of a prediction on the basis of the algorithm’s performances over past projects. The results obtained using this associative memory show that we could achieve 40% MMRE, 29% MdMRE, and 55% PRED(25), compared with a simple classifier that predicts 434% MMRE, 205% MdMRE, and 10% PRED(35) [49].

On the basis of the increase/decrease in performance measures, the best performing algorithm can be selected when building predictive analytics. A raw comparison between the values of a performance measure may sometimes be problematic—that is, the change in a measure might happen because of the sample used for training the algorithm and the improvements may not be statistically significant. So, even though an algorithm produces higher values in terms of a performance measure—say, 30% over 27%—statistical tests should be used to confirm that the change does not happen by chance.

Statistical tests, as explained in Section 16.4.2, can be used to identify patterns of software data as well as to compare the performance of predictive analytics that are built with different algorithms (e.g., the Mann-Whitney U test [73], and Nemenyi’s multiple comparison test and the Friedman test [52]). Box plots are also useful visualization techniques to depict the differences between the performance measures of two or more predictive analytics [30, 62]. Other performance measures we used for predictive analytics that produce continuous output can be found in Ref. [38].

Finally, we calculated context-specific performance evaluations, such as cost-benefit analysis (in terms of the amount of decrease (in lines of code) in inspection effort for findings software defects [84, 85]) in empirical studies for software defect prediction. These types of analysis are also easy to understand and interpret for software practitioners as they represent the tangible benefits—that is, effort reduction—of using predictive analytics.

16.5.3 Prescriptive Analytics

Once you decide on an algorithm, it is possible to improve the performance via some simple methods such as the following:

• forming ensembles [49, 85, 88],

• applying normalization [50, 80],

• feature selection through methods such as information gain [62],

• instance selection via k-NN-based sampling.

Ensembles are one powerful way to combine multiple weak learners into a stronger learner [88]. The idea behind ensembles is that different learners have different biases (recall how neural networks and decision trees are different from one another); hence, they learn different parts of the data. When their predictions are combined, they may complement one another and provide a better prediction. For example, in one of our studies, we employed a high number of prediction methods and combined them in simple ways such as taking the mean and median of the predictions coming from single learners [88]. However, before combining single learners, we paid attention to choosing the successful ones, where we ranked the learners and combined only the ones that had a high performance. Such a selection of only the successful learners provided us with ensembles that are consistently more successful than all the single learners. In other words, in the formation of an ensemble, it is better not to include the single learners whose performance is already low when they are run alone.

Normalizing the values of features is “a must” to improve learner performance if some of the features have very high values compared with others. For example, a feature keeping the number of classes defined in a project will have much smaller values compared with another feature that keeps the number of lines of code in this project. If you are using a k-NN learner, the impact of lines of code will dominate over the impact of the number of classes in a Euclidean distance calculation. A very common method we use for a number of different datasets is min-max normalization [50, 80]:

$\begin{array}{l}\hfill \frac{x_i-min(X)}{max(X)-min(X)},\end{array}$

where X represents the feature vector and x_i represents a single value in this vector. Although a very high number of features may be available to practitioners to be included in a data set, it is usually the case that some of these features are less important than the others. We have observed in different scenarios that it is in fact possible to improve the performance of a learner by selecting a subset of all the features [62, 88]. There are multiple ways to select features, such as information gain [62], stepwise regression [88], and linear discriminant analysis to name just a few.

Similarly to the idea of “not all features being helpful” for a prediction model, not all the instances are beneficial for a prediction problem too. There may be various reasons why certain instances should be filtered out of the data. For example, the instance may contain erroneous data, or the data stored concerning the instance may be correct, but it may be so different from all the rest of the instances that it turns out to be an outlier. In either case, we may want to filter out such instances from the dataset. We have discussed our use of k-NN-based instance filters prior to us of a naïve Bayes classifier [62], which is a good example of instance-based filtering. k-NN-based instance filters select only the training instances closest to the test instance, so the learner uses only filtered-out instances. Another algorithm that was inspired by the k-NN-based filters is filtering by variance [79], where we selected only the instances that form groups of low variance (of the dependent variable value).

The proposed methods of ensembles, feature and instance selection, and feature normalization are a subset of possible ways to improve the performance of a learner. However, their application should not be interpreted as a must. Depending on the problem and dataset at hand, a practitioner must experiment with his or her application separately or as combinations (e.g., feature and instance selection applied together). The combination that yields the best performance should be preferred.

In summary, predictive analytics requires a clear definition of business challenges from a statistical point of view: from understanding data through visualizations and statistical tests to defining the input-output of a predictive analytics; from the selection of an algorithm to performance assessment criteria, and finally, to improving the performance of the algorithm. On the basis our previous experience, we provide Table 16.6 to guide software data analysts toward building a software analytics framework. Note that Table 16.6 is by no means an exhaustive list of all the possible algorithms for predictive purposes. However, it covers the selected algorithms that were employed in the multiple industrial case studies that we have discussed so far.