Causing the Future With Bayesian Networks

Bayesian Networks (BN) offer a unique solution to the next generation of AI usage on real data since they have the ability to encode causal relationships (for predicting the future) and to go one step further with its property of “information propagation through the network.” This property makes it possible to understand the entire system, such as how a change in one variable can affect the others, and how important certain variables in a network are, and start gaming over the network to see which scenarios would give the best outcome (what is called “causing the future“). Thus, BNs provide real-time solutions that are beyond the capabilities of human reasoning and present them in simple, easy to interpret graphical models [3].

Causal relationship is formed based on two processes in which one of them (“cause“) is partly responsible for the occurrence or change of the other (“effect“). The direction of a causal relationship is not two-sided as in correlations. For example, smoking increases the risk of cancer, but patients who are diagnosed with cancer do not have to be smokers. There could be other factors such as genetics that cause cancer. If there are several factors involved in a causal relationship, BNs provide an efficient and easier way to define this relationship, observe how a change in each one of the factors causes the others, and infer the future (the outcome) based on the historical knowledge (on the inputs).

The Need for a Hybrid Approach in Software Analytics

BNs have been rigorously used in many disciplines such as healthcare and medicine that are more mature disciplines in terms of research design and guidelines. Similar to healthcare, evidence-based software engineering must put this research approach into practice in order to support and improve the decisions made by software practitioners on the usage and adaptation of a specific technology [1]. Earlier work on BNs in software engineering included validating dependability of software-based systems, estimating system reliability, making resource decisions, modeling project trade-offs, and estimation of software effort [4–9].

We need to recognize the fact that software intelligence provided by academia in the form of prediction models, software analytics, etc. should aim to help practitioners explain large volumes of data, understand what works and what does not, in practice, that would also let them make better decisions. These models should use the power of AI techniques, produce fact-based, faster, error-free solutions, and hence, help practitioners by reducing their workload during decision making and analysis. However, none of these models replace practitioners; instead they evolve via iterative feedback by practitioners. So a hybrid approach considering both the output of AI-based models and user knowledge is essential.

In our previous work we used the power of BNs to solve a specific problem of predicting software reliability, to combine the power of Bayesian statistics and expert knowledge, and to build hybrid models that can easily adapt to the environment, depending on the amount of local data in software organizations [1]. We observe that existing models in the literature were often built based on expert knowledge. Metrics were defined through surveys with experts, and metrics’ distributions and their causal relationships were also determined by senior developers/researchers. But this approach causes building models that are highly dependent on human judgment. We overcame this limitation by introducing a hybrid data collection and model construction process, and utilizing a suitable inference methodology. We collected both qualitative (expert knowledge) and quantitative (local data repositories) data in forming the nodes of BN. The proposed network models the processes of the software development life cycle (SDLC) and their relationships with software reliability. Using this causal network, we estimated the reliability of consecutive releases of software projects before a release decision, in terms of their residual (post-release) defects.

Collecting artifacts and defining metrics from all software processes may not be feasible. On the other hand, quantifying processes with survey data also has a bias due to subjective judgments of the software team, as their experience and qualifications are diverse. Therefore, we need a hybrid data collection approach by using quantitative data extracted from the organization’s software repositories and qualitative data collected through surveys.

A hybrid BN is necessary to handle a mixture of continuous and categorical variables in a single model, such that categorical variables can also be parents of continuous variables. A hybrid BN also handles continuous variables without transforming them (ie, discretization). This also helps to avoid potential biases due to discretization in small-state spaces, such as software engineering datasets.

In this work, our industry wanted to decide “when to stop testing and release the product.” Therefore, we aimed at predicting software reliability by learning the probability of residual defects given observations from software metrics. This probability is computed, ie, “inferred,” from the BN via several techniques. We used Gibbs sampling, a well-known Monte Carlo technique on approximation of the joint probability distribution over conditional distributions. Gibbs sampling is well-suited for inferring the posterior distribution in a BN formed with a mixture of continuous and categorical variables, some of which also have missing data. Missing data in surveys is a common problem, and we had missing data in our surveys as well.

Our findings in this study showed that the effects of software processes and their causal relationships on software reliability are worth the investigation. Construction of a BN gave us some insights into understanding process factors in detail and their effect on software reliability. We used different factors/metrics to cover the 3Ps (people, process, product) of an SDLC. We have seen that causal models are better at predicting software reliability than generalized linear models, which treat each factor independently. The BN model in our industry partner predicts 84% of post-release defects with less than a 25% Mean Relative Error.

Use the Methodology, Not the Model

Software organizations need predictive models to make release decisions. Using estimated post-release defects and the company’s pre-determined threshold for software reliability, it is possible to estimate a release readiness level for each release. This threshold helps managers to make a decision either to release the software, or to delay the release, or to cancel the release. Measuring/quantifying software processes helps software organizations build a measurement repository and monitor trends of development practices in a systematic way. Our experience in building BNs in different organizations showed that there are not a specific set of metrics that should be used in any software development organization to predict reliability in terms of post-release defects. Causal relationships also change from one model to another based on local data collected from the organization. Hence, we suggest that practitioners follow a methodology rather than using same metric sets with the same causal relationships among each other.

Our experience in industry as well as the findings of our systematic review in [3] showed that there is a need for a systematic approach in building BNs. The dataset characteristics, source of data, and types of input and output variables would be important in:

1. identification of structure learning techniques to model cause-effect relationships between variables, and the rationale of choosing the selected technique, and in

2. identification of parameter learning, with respect to

○ how to set the prior distributions, and

○ how to estimate the final parameters of the model (inference), and the rationale of choosing the selected technique.

As always, the tool support and tool selection is also an important consideration for adoption and sustainability of a given technique/model in the industry.

Similar to computational biology and healthcare, we need to make decisions under uncertainty using multiple data sources. As we understand the dynamics of BNs and the techniques used for model learning, and inference, these models would enable us to uncover hidden relationships between variables, which cannot be easily identified by experts [1]. BNs will help us determine how scientific belief is modified by data. In the software engineering/software analytics community (both academia and practice), we should embrace BNs as a method for evidence-based statistics to be used in inference and decision making.