9

 

SOLUTION EVALUATION

Solution Evaluation includes the processes to validate a full solution or a segment of a solution that is about to be or has already been implemented. Evaluation determines how well a solution meets the business needs expressed by stakeholders, including delivering value to the customer.

The Solution Evaluation processes are as follows:

9.1 Evaluate Solution Performance—The process of evaluating a solution to determine whether the implemented solution or solution component is delivering the business value as intended.

9.2 Determine Solution Evaluation Approach—The process of determining which aspects of the organization and/or solution will be evaluated, how performance will be measured, when performance will be measured, and by whom.

9.3 Evaluate Acceptance Results and Address Defects—The process of deciding what to do with the results from a comparison of the defined acceptance criteria against the solution.

9.4 Obtain Solution Acceptance for Release—The process of facilitating a decision on whether to release a partial or full solution into production and eventually to an operational team, as well as transitioning knowledge and existing information about the product, its risks, known issues, and any workarounds that may have arisen in response to those issues.

Figure 9-1 provides an overview of the Solution Evaluation processes. The business analysis processes are presented as discrete processes with defined interfaces, although, in practice, they overlap and interact in ways that cannot be completely detailed in this guide.

KEY CONCEPTS FOR SOLUTION EVALUATION

Solution Evaluation activities are performed to assess whether or not a solution has achieved the desired business results. Solution evaluation practices apply to anything that needs to be evaluated, from a discrete usage scenario to a broad business outcome. Solution Evaluation consists of the work done to analyze measurements obtained for the solution by comparing the actual results of acceptance testing to the expected or desired values, as defined by the acceptance criteria. Over the long term, these activities evaluate whether the expected business value of a solution has been achieved.

Analyzing the results from surveys, focus groups, or the results of exploratory testing of functionality are examples of qualitative or coarsely quantitative evaluation activities. Other evaluation activities involve obtaining more precise quantitative measurements, such as directly looking at data from a solution. Nonfunctional characteristics of a solution are often evaluated with measurements as well. For example, performance standards in service-level agreements can be measured for actual compliance. Comparing estimated and actual costs and benefits may also be part of Solution Evaluation. For solutions involving manufacturing, evaluations may include comparisons between actual and expected production outputs or conformance to tolerances for a product. For solutions involving software, analyzing comparisons between expected and actual values of data manipulated by the high-level functionality of the solution may be part of evaluation.

Evaluation activities may occur:

• At any point when a go/no-go or release decision needs to be made for a solution or a substantive segment of it;
• During a short-term period after a solution or segment is put into operation, such as after a warranty period; or
• Well after a solution is put into operation, to obtain a long-term perspective about whether the business goals and objectives for the solution were met and whether the value expected continues to be delivered.

Solution Evaluation often requires early preparation, so that what is needed to perform this work is in place later when evaluation is conducted. Preparation for evaluating a solution includes defining and confirming the expected business value, identifying and defining what kind of performance data will be used to evaluate whether value has been achieved, confirming that performance data will actually be available, and obtaining baseline or control data when necessary. Definition of specific evaluation criteria, such as the expected or desired range of values for the selected metrics, supports analysis activities beyond Solution Evaluation; therefore, determining specific evaluation criteria is considered part of Section 7.4 on Define Acceptance Criteria.

For portfolios, programs, and projects, the following can be stated with regard to Solution Evaluation:

• Evaluation of an implemented solution may be used to identify new or changed requirements, which may lead to solution refinement or new solutions.
• Solution Evaluation can provide input into go/no-go business and technical decisions when releasing an entire solution or a segment of it.
• Evaluation may identify a point of diminishing returns, such as the point where additional value that could be obtained from a solution does not justify the additional effort needed to achieve that value. In this case, Solution Evaluation gives teams the ability to “end early” even if there is still additional functionality that could be built, allowing funds to be reallocated to work on higher-priority projects that can bring additional business value.
• Assessed limitations of the solution might be the basis for recommendations for a wide variety of follow-up activities, ranging from actions to improve the performance of the solution to recommendations for replacing it or phasing it out.
• Solution evaluations provide a basis for portfolio and program management to make decisions about new products and product enhancements.

Complicating factors for evaluating a solution are as follows:

• Some of the benefits and value of the solution may seem to be intangible, and therefore, not possible to measure. For intangible benefits, it may be necessary to define measurements that provide indirect evidence that the benefits have been achieved.
• Some of the information needed to evaluate the solution may not be needed for the solution itself. Obtaining and using such data may add to the costs of developing the solution.
• Some aspects of a solution that reflect the benefits and value may not be measurable until well after a solution is released. In these situations, the operational business area responsible for the product or perhaps an enterprise organizational area may take responsibility for identifying and measuring leading indicators.

These factors are yet other reasons for thinking about Solution Evaluation as part of initial product development efforts. For more information on Solution Evaluation, see Section 6.3 of Business Analysis for Practitioners: A Practice Guide.

9.1 EVALUATE SOLUTION PERFORMANCE

Evaluate Solution Performance is the process of evaluating a solution to determine whether the implemented solution or solution component is delivering the business value as intended. The key benefit of this process is that the analysis provides tangible data to determine whether the solution that the business has invested in is achieving the expected business results and serves as an input to decisions about future initiatives. The inputs, tools and techniques, and outputs for this process are shown in Figure 9-2. Figure 9-3 depicts the data flow diagram for the process.

Evaluating solution performance involves determining whether a solution or solution component that has been put into operation is delivering the desired business value. The actual business value of a solution is measured in terms of the business goals or objectives. Business analysis also assesses the underlying reasons behind the results obtained.

Evaluation of solution performance typically occurs after a solution has been released. Therefore, it is more likely that the evaluation of solution performance will occur during portfolio or program activities rather than project activities. The extent to which business value has been achieved and the reasons for the achievement are significant factors to consider when making product decisions within a portfolio or program. However, Evaluate Solution Performance not only supports product decisions within a program or portfolio, but also may be used during a project at any time when solution evaluation can provide insights.

Data to evaluate the business value are often measured by and obtained from the business area that takes ownership of the solution or by instrumentation that has been built into the product. Business analysis techniques are used to analyze variations between desired and actual results as part of assessing the business value of the solution. Because many measurements of business value need to take place after a solution is released and often need to be measured over the long term to detect trends, there needs to be organizational commitment to invest in making the measurements and to invest in building or purchasing the capabilities to measure business value when those capabilities would not otherwise be available. When such investments are not possible, organizations need to consider less costly next-best-alternative ways to measure business value. Proxies for business value might be used in these situations.

The relevant business goals and objectives, evaluated acceptance results from a previous release, performance data, and baseline data, when available, are analyzed to determine whether value has been delivered, as well as the reasons for better-than-expected results or the root causes of any problems. Among the typical reasons for missed business value are:

• Technical causes,
• Business practices or constraints,
• Resistance to the product or the way it is intended to be used, and
• Opportunistic workarounds devised by those who use the product to get around real or perceived solution limitations.

The assessment of the performance of a product becomes input into recommendations for improving the long-term performance of the solutions and for portfolio and program management decisions about further enhancements to the product, decisions about new products, and decisions to replace or discontinue products.

9.1.1 EVALUATE SOLUTION PERFORMANCE: INPUTS

9.1.1.1 BUSINESS CASE

Described in Section 4.6.3.1. The business case describes pertinent information to determine whether the initiative is worth the required investment. It is an authoritative source where expected benefits have been stated.

9.1.1.2 BUSINESS GOALS AND OBJECTIVES

Described in Section 4.3.3.1. Business goals and objectives specify stated targets that the business is seeking to achieve. They provide the context for evaluating solution performance, because they are a measurable description of the expected business value. Examples of business objectives include increased throughput in the manufacture of a product by x%, reduction in costs by y%, and increased sales volume by $z.

Many products support business goals and objectives that can be associated with one or more key performance indicators (KPIs), which are target performance levels that are usually defined by an organization's executives. For organizations that already define and measure KPIs, it may sometimes be possible to use one or more of these KPIs to evaluate the solution, taking advantage of existing measurement capabilities.

9.1.1.3 EVALUATED ACCEPTANCE RESULTS

Described in Section 9.3.3.1. Evaluated acceptance results are comparisons between the acceptance criteria and the actual results, and the root cause for variances between them. The evaluated acceptance results from when the solution was released describe the state of the product, and specifically any proficiencies or deficiencies it may have, which might have altered the expected business value. A product that exceeds expectations can provide a basis for future opportunities. A product that does not meet expectations may have defects, which will necessitate analysis of the cost to address the defects and the business impact of addressing them or accepting them.

9.1.1.4 PERFORMANCE DATA

Performance data are a quantified output of a product. Examples of outputs that might be measured include throughput in the manufacture of a product, volume of some output generated by the product, reduction in costs, productivity achieved in using a service, number of users adopting a solution, amount of revenue generated, sales volume, number of individuals reached in a marketing campaign, and levels of customer satisfaction.

Performance data are used to determine the actual business value of a product by assessing the performance data before and after a release. Any performance data from a prior version of a product or manual process represent a baseline. For example, if business value is measured in terms of increased sales volume, then the quantified business value is the difference in sales volume before (the baseline) and after release of the solution.

If there is no baseline of performance data, then either the performance data after a release can represent the business value or estimates of the original baseline can be made. Ideally, performance data are measured for the stated business goals and objectives. However, when that is not possible, proxies for those objectives might be used to determine business value. For example, proxy performance data could include measurements that describe the effectiveness or quality of the product, such as the average duration of a task while using the product, response time for solutions involving software, or counts of errors made while performing a task.

Performance data are generally measured by and obtained from the business area that takes ownership of the solution or by instrumentation built into the solution. Performance data also can be measured using capabilities externally available to the solution, such as the results from surveys or focus groups conducted after a solution has been released.

9.1.1.5 SOLUTION EVALUATION APPROACH

Described in Section 9.2.3.1. The solution evaluation approach provides the agreements made about how solution evaluation activities will be performed. The solution evaluation approach identifies which types of metrics will be used to evaluate whether the expected business value has been achieved. It explains how solution performance will be evaluated. Because solutions can be evaluated long after their release, the original solution evaluation approach may need to be updated over time based on the current state.

9.1.2 EVALUATE SOLUTION PERFORMANCE: TOOLS AND TECHNIQUES

9.1.2.1 COST-BENEFIT ANALYSIS

Cost-benefit analysis is a financial analysis tool used to determine the benefits provided by a project or solution against its costs. Considering the results of an actual versus expected cost-benefit analysis is one way to assess the business value of the solution. Conducting a cost-benefit analysis can help determine whether recommendations made as part of the evaluation of solution performance are likely to be cost-effective. For more information on cost-benefit analysis, see Section 4.4.2.2.

9.1.2.2 ELICITATION TECHNIQUES

Elicitation techniques are used to draw out information from sources. Facilitated workshops, focus groups, interviews, and observation are among the elicitation techniques often used to uncover root causes for identifying the differences between the expected and actual business value of a solution and making recommendations to address those variances:

• Facilitated workshops. Use a structured meeting led by a skilled, neutral facilitator and a carefully selected group of stakeholders to collaborate and work toward a stated objective. Facilitated workshops may be conducted with decision makers to collaboratively work together to identify the root cause and any variances. For more information on facilitated workshops, see Section 6.3.2.4.
• Focus groups. Provide an opportunity to obtain feedback directly from customers and/or end users. Focus groups can be used to learn about which deficiencies in business value are important or of concern to stakeholders. For more information on focus groups, see Section 6.3.2.5.
• Interviews. May be conducted with individual stakeholders and users to gain insights about root causes in situations where expected and actual business value differs widely. The privacy and confidentiality of an individual interview may reveal considerations that might not otherwise be expressed in a facilitated workshop or focus group. For more information on interviews, see Section 6.3.2.6.
• Observation. An elicitation technique that provides a direct way of eliciting information about how a process is performed or a product is used by viewing individuals in their own environment performing their jobs or tasks. For some products, direct observation of users actually performing their jobs or tasks may uncover workarounds that they use to compensate for product gaps. These workarounds may have had unintended consequences leading to missed business value. Such workarounds could be missed when using interviews and other verbal communication techniques, because it may not occur to users who are familiar with using the product for a long time to mention how they compensate for a lack—or perceived lack—of functionality. For more in formation on observation, see Section 6.3.2.7.

For more information on elicitation techniques, see Section 6.3.2.

9.1.2.3 PRODUCT PORTFOLIO MATRIX

A product portfolio matrix, also known as a growth-share matrix, is a market analysis quadrant diagram used by some organizations to qualitatively analyze their products or product lines. One axis reflects market growth (or demand for a product) from low to high, while the other reflects the market share of the organization from low to high. The matrix provides a quick visual way to evaluate which products are meeting or exceeding performance expectations in the marketplace, and as a result, can contribute to evaluating the performance of a product.

Figure 9-4 is an example of a product portfolio matrix. In this figure, assuming that the costs to produce, market, and distribute the product are not a significant factor, the products that provide the most significant benefits to the organization would be found in the upper left quadrant, because these are the products where the organization has a high market share in a market with a high growth rate. Those in the upper right quadrant are regarded as having good potential because, although they have a low market share, they are in a market that is continuing to grow. Those in the lower left quadrant, with a high market share in a low growth market, are considered a dependable income stream.

9.1.2.4 PRIORITIZATION SCHEMES

Prioritization schemes are different methods used to prioritize requirements, features, or other product information. For the purposes of evaluating solution performance, prioritization schemes can be used to rank the value of the benefits obtained from the solution as well as the severity of any challenges it creates. Doing so can provide a way to consider whether the benefits outweigh the challenges and vice versa. For more information on prioritization schemes, see Section 7.7.2.5.

9.1.2.5 ROOT CAUSE AND OPPORTUNITY ANALYSIS

Root cause analysis is used to determine the basic underlying reason that causes a variance, defect, or risk. Opportunity analysis is used to study the major facets of a potential opportunity to determine the possible changes in products offered to enable its achievement. When used to evaluate solution performance, root cause and opportunity analysis can help uncover the reasons why the actual business value either did not meet or possibly exceeded expectations and where there may be further product improvement opportunities. The results of root cause and opportunity analysis can be used when making decisions about new products and whether to enhance, retire, or discontinue existing products. For more information on root cause and opportunity analysis, see Section 4.2.2.9.

9.1.3 EVALUATE SOLUTION PERFORMANCE: OUTPUTS

9.1.3.1 ASSESSMENT OF BUSINESS VALUE

The assessment of business value is the result from comparing expected business value from a solution against the actual value that has been realized. If the desired business value was not achieved, the assessment includes the reasons why.

The assessment is used to make decisions about whether to develop a new product or to enhance, retire, or discontinue an existing product. The work done to implement any recommendations will need to be prioritized as part of the ongoing planning activities the organization performs for portfolio or program management. For more information on assessing business value, see Section 6.10.3 of Business Analysis for Practitioners: A Practice Guide.

9.1.4 EVALUATE SOLUTION PERFORMANCE: TAILORING CONSIDERATIONS

Adaptive and predictive tailoring considerations for Evaluate Solution Performance are described in Table 9-1.

Table 9-1. Adaptive and Predictive Tailoring for Evaluate Solution Performance

Aspects to Be Tailored	Typical Adaptive Considerations	Typical Predictive Considerations
Name	Evaluate Solution Performance
Approach	After a solution is launched, performance data are used to compare business value to what was expected. Subjective and objective evidence is collected. Root causes are identified for deviations from what was expected. Recommendations may be made to address reducing the deviations. Root causes become input into decisions to enhance, maintain, replace, or discontinue the product.
Deliverables	Assessment results can be formally documented or not.

9.1.5 EVALUATE SOLUTION PERFORMANCE: COLLABORATION POINT

Several roles may contribute to evaluating solution performance. Functional managers may provide insights on how the product is performing that might not be obvious solely by looking at metrics, and DevOps (which some organizations use to coordinate activities and improve collaboration between development and operational areas) may provide insights on how product performance is trending during the warranty period. Subject matter experts (SMEs) may be responsible for conducting the evaluation or may help surface the reasons behind the results obtained. Once the performance data are acquired, portfolio, program, and project managers use the results when making strategic and tactical decisions.

9.2 DETERMINE SOLUTION EVALUATION APPROACH

Determine Solution Evaluation Approach is the process of determining which aspects of the organization and/or solution will be evaluated, how performance will be measured, when performance will be measured, and by whom. The key benefit of this process is that performance indicators and metrics are selected or defined so they can be collected, reported on, and evaluated to support the continual improvement of the organization and/or product. The inputs, tools and techniques, and outputs for this process are shown in Figure 9-5. Figure 9-6 depicts the data flow diagram for the process.

Determining an approach for Solution Evaluation involves conducting research, discussions, and analysis to identify how and when to evaluate a product. Determining the solution evaluation approach should include:

• Planning when and how often solution evaluation activities should be performed. Solution evaluation may occur during solution development for some components of a solution, just before a release, soon after a release, or long after a release;
• Planning which evaluation techniques will be used. Not all techniques need to be decided before analysis begins, but by thinking ahead, it is more likely that business analysts will be prepared to use a variety of techniques;
• Planning how the evaluation results will be analyzed and reported;
• Planning how the progress of solution evaluation and its outputs will be communicated to stakeholders and other interested parties, including what level of formality is appropriate; and
• Planning which metrics will be used to evaluate performance and how they tie to the business goals and objectives.

A metric is a set of quantifiable measures used to evaluate a solution or business. When performing solution evaluation, a metric defines how solution performance can be quantified. Many metrics can be used to compare the tangible properties of the solution, such as throughput, productivity, or efficiency. There are two common types of metrics defined by the solution evaluation approach:

• Metrics that will be used to evaluate the solution or its components for acceptance during or shortly after development. Acceptance criteria are defined to set acceptable ranges on these metrics. For information about acceptance criteria, see Section 7.4.3.1; and
• Metrics likely to be used later to determine if the business value was delivered. Because solution performance is not evaluated until after a solution is released, the choice of metrics or how they are measured might change over time.

When determining the solution evaluation approach, the team needs to consider the following about metrics:

• What type of performance data will be collected for the metrics, when, and how frequently?
• Who is responsible for collecting and reporting the performance data?
• Are there built-in collection and reporting mechanisms already available for these metrics? If not, what additional capabilities are needed? Who is going to cover the costs of developing them?

Information about the problem or opportunity that the solution will address and the product scope provide a basis for deciding which types of metrics could help the organization assess the product's performance. There may be industry or organizational standards that have defined benchmarks that can be used to compare a solution with general expectations for it.

The solution evaluation approach should be defined as early as possible in the product development life cycle because some of the metrics identified in it may require capturing additional information, above and beyond what the product itself requires. Without early definition, there is a risk that necessary information will be expensive or impossible to obtain. When considering new metrics, the cost of capturing actual measurements and reporting on them is an important factor. When the proposed metrics appear too costly or time-consuming to justify their use, less costly, next-best alternatives may be considered.

9.2.1 DETERMINE SOLUTION EVALUATION APPROACH: INPUTS

9.2.1.1 METRICS AND KPIs

A metric is a set of quantifiable measures used to evaluate a solution or business. In solution evaluation, a metric defines how solution performance can be quantified. Many types of metrics could be used to evaluate a solution. Business objectives are a type of metric that describe the desired business value of a product. Key performance indicators (KPIs) are a related type of metric, usually defined by an organization's executives, used to evaluate an organization's progress toward meeting its objectives or goals. Other, more granular metrics that also trace back to business objectives can be used to evaluate the interim success of a solution during or after development. Defining a solution evaluation approach involves choosing which metrics to use to evaluate a solution under development as part of its acceptance criteria or a previously released solution as part of its performance. When choosing, consideration can be given to metrics that are already available within the organization.

9.2.1.2 PRODUCT SCOPE

Described in Section 4.6.3.2. Product scope is defined as the features and functions that characterize a solution. The product scope and the business goals and objectives on which a solution is based provide suggestions as to what types of information should be collected and measured for evaluation purposes.

9.2.1.3 SITUATION STATEMENT

Described in Section 4.1.3.2. A situation statement describes the problem or opportunity and its effect on the organization. The wording of the situation statement may suggest ways to measure whether or not the problem or opportunity has been addressed. The situation statement might also be used to determine how, when, and how frequently a solution should be evaluated.

9.2.2 DETERMINE SOLUTION EVALUATION APPROACH: TOOLS AND TECHNIQUES

9.2.2.1 ELICITATION TECHNIQUES

Elicitation techniques are used to draw information from sources. Elicitation techniques can be used to identify existing metrics or potential new metrics for determining solution acceptance or evaluating whether business value has been delivered. Elicitation techniques are also helpful to identify ideas about how a solution can be evaluated.

A few common elicitation techniques that can support determining the solution evaluation approach are document analysis, facilitated workshops, and interviews:

• Document analysis. An elicitation technique used to analyze existing documentation to identify relevant product information. A team may be able to gain ideas for likely metrics from documentation that inventories the kind of information that is currently collected or that will be available once the solution is in place. The team may also be able to reuse all or part of a documented solution evaluation approach. For more information on document analysis, see Section 6.3.2.3.
• Facilitated workshop. A structured meeting led by a skilled, neutral facilitator and a carefully selected group of stakeholders to collaborate and work toward a stated objective. The structure of a facilitated workshop promotes an efficient and focused meeting for proposing metrics. For more information on facilitated workshops, see Section 6.3.2.4.
• Interviews. Used to elicit information about the solution evaluation approach. They can be scheduled with various stakeholders who possess key information for determining the approach. When necessary, individual interviews can provide each stakeholder with an opportunity to speak candidly about the ability of the organization to capture proposed metrics. For more information on interviews, see Section 6.3.2.6.

For more information on elicitation techniques, see Section 6.3.2.

9.2.2.2 GROUP DECISION-MAKING TECHNIQUES

Group decision-making techniques are methods that can be used in a group setting to bring participants to a final decision on an issue or topic under discussion. Decision-making techniques are used to reach consensus about the solution evaluation approach. For example, teams need to reach decisions about which metrics to collect, the value of information against costs to obtain it, which of the proposed metrics to use, and the frequency of collecting metrics.

For more information on group decision-making techniques, see Section 8.3.2.7.

9.2.2.3 PRIORITIZATION SCHEMES

Prioritization schemes are different methods used to prioritize requirements, features, or any other product information. When developing the solution evaluation approach, the interest in obtaining information needs to be balanced against the cost of obtaining and communicating that information. The balance is achieved through decision making and may require prioritizing among the proposed metrics.

Prioritization schemes such as MoSCoW can help decide which metrics must, should, could, and won't be collected, as can weighted rankings. For more information on prioritization schemes, see Section 7.7.2.5.

9.2.2.4 RETROSPECTIVES AND LESSONS LEARNED

Retrospectives and lessons learned leverage past experiences to plan for the future. As part of devising a solution evaluation approach, retrospectives and lessons learned can be used to learn about the ease of collecting and analyzing different types of metrics and the effectiveness of different evaluation techniques. For more information on retrospectives and lessons learned, see Section 5.7.2.4.

9.2.3 DETERMINE SOLUTION EVALUATION APPROACH: OUTPUTS

9.2.3.1 SOLUTION EVALUATION APPROACH

The solution evaluation approach describes when and how a solution will be evaluated, the types of metrics that will support evaluation, the feasibility of collecting and communicating the actual performance data for these metrics, and who is responsible for conducting the evaluation and communicating results. For information about what to include in a solution evaluation approach, see Section 3.4.15 of Business Analysis for Practitioners: A Practice Guide.

9.2.4 DETERMINE SOLUTION EVALUATION APPROACH: TAILORING CONSIDERATIONS

Adaptive and predictive tailoring considerations for Determine Solution Evaluation Approach are described in Table 9-2.

Table 9-2. Adaptive and Predictive Tailoring for Solution Evaluation Approach

Aspects to Be Tailored	Typical Adaptive Considerations	Typical Predictive Considerations
Name	Not a formally named process	Determine Solution Evaluation Approach
Approach	Considered part of initial planning and taken into account when defining acceptance criteria, creating a definition of done, and during coordination with quality assurance. The decisions may be documented informally or discussed among the team.	The approach for Solution Evaluation is defined as part of planning and prior to defining acceptance criteria.
Deliverables	Not a separate deliverable.	The solution evaluation approach becomes a component of the business analysis plan.

9.2.5 DETERMINE SOLUTION EVALUATION APPROACH: COLLABORATION POINT

Many roles can participate in determining the solution evaluation approach. Any manager whose area could be responsible for collecting, monitoring, evaluating, or funding the costs to obtain the performance data could participate in determining and approving the approach. Risk compliance and legal areas may provide requirements as to what needs to be measured, for how long, and for what length of time performance data need to be retained. Project sponsors also need to be included because they hold different insights as funders and may have different priorities in terms of what they are willing to spend for Solution Evaluation. Operational leads may provide ideas for metrics and assess the ability of their operational areas to measure them and provide insights into the costs associated with collecting them.

9.3 EVALUATE ACCEPTANCE RESULTS AND ADDRESS DEFECTS

Evaluate Acceptance Results and Address Defects is the process of deciding what to do with the results from a comparison of the defined acceptance criteria against the solution. The key benefit of this process is that it allows for informed decision making about whether to release all or part of a solution and whether to undertake changes, fixes, or enhancements to the product. The inputs, tools and techniques, and outputs for this process are shown in Figure 9-7. Figure 9-8 depicts the data flow diagram for the process.

This process compares the acceptance criteria and the actual results of acceptance testing to provide recommendations on how to deal with situations where aspects of a solution do not meet the acceptance criteria specified for it. It covers acceptance testing at any level of granularity, from something as big as an entire solution release to something as small as one business scenario (composed of one or more user stories). It focuses on the actual results from comparing acceptance testing to their acceptance criteria, rather than on the tests themselves. This distinction supports common practice in the industry, where organizations distinguish between roles that conduct business analysis and roles that perform testing. The pass/fail aspect of comparing actual results from acceptance testing to testing criteria is typically a task within the testing discipline. Business analysis is then needed to consider the magnitude and severity of the defects, to determine their root cause, to identify risks associated with them, and to identify and recommend ways to address them. As part of those recommendations, business analysis can consider the business impacts and costs incurred by repairing or working around the defects, along with potential business impacts and costs of deploying the solution without addressing the defects.

The test results that are part of evaluating acceptance criteria and addressing defects may come from:

• Exploratory tests and user acceptance tests,
• Day-in-the-life (DITL) tests,
• Preproduction or simulated production testing,
• Tests of functionality within a scenario, and
• Tests of nonfunctional requirements.

For more information about testing approaches, see Section 6.6 of Business Analysis for Practitioners: A Practice Guide.

In organizations that conduct some form of ongoing monitoring and acceptance testing for quality, such as in manufacturing or construction, differences between actual results and acceptance criteria can be evaluated to look for trends and patterns in terms of increases or decreases in unacceptable results. For solutions that involve software, where an organization has adopted automated regression testing, it may be possible to spot trends and patterns in out-of-tolerance results.

The evaluation process includes determining the root cause of any variance or defect. It may include an analysis of the cost to address the defect and the business impact of addressing it or accepting it. Recommendations for how to address the defect can also be made and may include:

• Potential workarounds to business practices or product usage that will not interfere with other product functionality or cause the product to behave in unintended ways;
• Possible modifications to the product, which could require a change request;
• Potential adjustments to how or which results are measured;
• Identifying a need to investigate the technical causes of the defect; and
• Communications to customers and users to clarify how the product is expected to be used.

Activities to evaluate acceptance results and address defects generally occur at any point at which a go/no-go or release decision needs to be made for a solution or a component of it. However, they may also occur while working with product defects discovered after the solution is put into operation.

9.3.1 EVALUATE ACCEPTANCE RESULTS AND ADDRESS DEFECTS: INPUTS

9.3.1.1 ACCEPTANCE CRITERIA

Described in Section 7.4.3.1. Acceptance criteria are concrete and demonstrable conditions that have to be met for the business stakeholders or customers to accept the item. They may take the form of lists of acceptance criteria for each user story in an adaptive approach or a list of higher-level acceptance criteria for a release or solution in a predictive approach. Acceptance criteria are a key input because they indicate the conditions against which the solution is being tested.

9.3.1.2 ACTUAL ACCEPTANCE RESULTS

Actual acceptance results contain the pass/fail results from comparing test results against the acceptance criteria, often provided by a quality control team. In business analysis, the actual acceptance results are then analyzed to determine the reasons for any differences between the test results and acceptance criteria and to recommend how to address the defects. For organizations that conduct regression testing, regression test results may be valuable to analyze as well, as they may reveal situations where new or enhanced functionality impacts existing functionality.

9.3.2 EVALUATE ACCEPTANCE RESULTS AND ADDRESS DEFECTS: TOOLS AND TECHNIQUES

9.3.2.1 PRIORITIZATION SCHEMES

Prioritization schemes are different methods used to prioritize requirements, features, or any other product information. As part of addressing defects, prioritization schemes can be used to rank each defect based on its severity and how likely a user is to encounter it, as well as on the impact of not fixing the defect. For more information on prioritization schemes, see Section 7.7.2.5.

9.3.2.2 ROOT CAUSE ANALYSIS

Root cause analysis techniques are used to determine the reason for a variance or a defect. Root cause analysis techniques can be used to discover the reasons why actual results did not meet the acceptance criteria. These root cause reasons will also figure into decisions about whether or not to move forward to obtain solution acceptance for a release. For more information on root cause analysis techniques, see Section 4.2.2.9. For an example of how root cause analysis can be applied to evaluating acceptance criteria and addressing defects, see Section 6.10.3 of Business Analysis for Practitioners: A Practice Guide.

9.3.2.3 TRACEABILITY MATRIX

A traceability matrix is a grid that links product requirements from their origin to the deliverables that satisfy them. A traceability matrix can be used to establish relationships among product information, deliverables, and project work to ensure that each relates back to business objectives. As part of evaluating acceptance results, a traceability matrix can be used as a tool to assess the business impact of not addressing variations from acceptance criteria and defects; for example, there could be a significant business impact from not addressing defects associated with features that trace to a high-priority objective. For more information on traceability matrices, see Section 8.2.2.5.

9.3.2.4 VARIANCE ANALYSIS

Variance is a quantifiable deviation, departure, or divergence from a known baseline or expected value. Variance analysis is a technique for determining the cause and degree of difference between the baseline and actual performance. Irrespective of product life cycle, when there are significant differences between the acceptance criteria and actual results from testing, variance analysis may be applied to consider causes of the differences. For more information on variance analysis, see Section 5.7.2.6.

9.3.3 EVALUATE ACCEPTANCE RESULTS AND ADDRESS DEFECTS: OUTPUTS

9.3.3.1 EVALUATED ACCEPTANCE RESULTS

Evaluated acceptance results provide a summarized comparison between the acceptance criteria and the actual acceptance results, along with the root cause for variances or defects, the analysis of the cost to address the defect, and the business impact of addressing it or accepting it. Some organizations will track the evaluated acceptance results and recommendations in logs. When evaluated acceptance results are communicated, recommendations about ways to address the defect may be included. For organizations that make it a practice to maintain documentation about their products and projects, another type of evaluated acceptance result can be obtained by comparing approved requirements to as-built documentation.

9.3.4 EVALUATE ACCEPTANCE RESULTS AND ADDRESS DEFECTS: TAILORING CONSIDERATIONS

Adaptive and predictive tailoring considerations for Evaluate Acceptance Results and Address Defects are described in Table 9-3.

Table 9-3. Adaptive and Predictive Tailoring for Evaluate Acceptance Results and Address Defects

Aspects to Be Tailored	Typical Adaptive Considerations	Typical Predictive Considerations
Name	Not a formally named process; performed as part of the work to demonstrate and obtain feedback on the solution	Evaluate Acceptance Results and Address Defects
Approach	Typically covers a slice of product capabilities at a time, along with the acceptance criteria associated with those capabilities. Demonstrate what was developed during the most recent iteration, obtain feedback from decision makers, and add defects that need to be addressed to the backlog for prioritization. When conducted for a specific user story, if the demonstration does not prove conformance to the acceptance criteria, the entire story can be rejected and returned to the backlog to be reprioritized.	Performed on a segment of the product that is to be delivered as part of a release or the entire product. Use evaluated acceptance results and the acceptance criteria as a starting point to determine root cause for variances or defects. Consider the costs to address the defects and the business impacts of addressing or accepting them. Provide recommendations for how to address the defects.
Deliverables	Additions of defects to the backlog.	Documentation of evaluated acceptance results, along with any associated change requests.

9.3.5 EVALUATE ACCEPTANCE RESULTS AND ADDRESS DEFECTS: COLLABORATION POINT

Many roles collaborate with the business analyst when evaluating acceptance results and determining how to address identified defects. Developers, designers, quality control analysts, and subject matter experts (SMEs) from the business or operational areas may have insights on reasons for a variance between actual results and the expected results. Any role involved in identification, prioritization, or adjudication of change requests, such as an individual on a change control team, might take leadership for addressing defects in consultation with those responsible for business analysis.

9.4 OBTAIN SOLUTION ACCEPTANCE FOR RELEASE

Obtain Solution Acceptance for Release is the process of facilitating a decision on whether to release a partial or full solution into production and eventually to an operational team, as well as transitioning knowledge and existing information about the product, its risks, known issues, and any workarounds that may have arisen in response to those issues. The key benefit of this process is the creation of an agreed-upon break between building a solution and releasing a solution for acceptance by the stakeholders. The inputs, tools and techniques, and outputs for this process are shown in Figure 9-9. Figure 9-10 depicts the data flow diagram for the process.

Obtaining solution acceptance for a release provides the stakeholders who are accountable for the product with an opportunity to decide whether it should be released in whole, in part, or not at all.

For solution acceptance, the term release may refer to releasing a solution or a segment of a solution into a production environment while its development team is still responsible for it. It can also refer to releasing a solution or a segment of it to the operational area that will take responsibility for it.

Release decisions are typically based upon:

• Acceptability of the solution, as evidenced by the evaluated acceptance results;
• Confirmation that the organization is ready for the release;
• Confirmation that the transition activities to prepare for the release have been completed to the degree necessary, including the coordination of this solution release with any other releases that are happening concurrently; and
• Acceptance of any remaining product risks and workarounds.

A decision to release a solution to its operational area requires confirmation of the completion of any warranty period that may have been established. A warranty period is an interval of time during which the product development team is responsible for addressing any defects found after the solution is released for production.

Depending on organizational norms, obtaining a release decision may include obtaining sign-off. For an iterative or adaptive project life cycle, informal sign-off generally occurs at the end of each sprint or iteration, with formal sign-off, when required, occurring prior to each release of the solution to the production environment. For a predictive project life cycle, sign-off usually occurs at the end of the project life cycle, either immediately prior to release to production or post release after the warranty period is complete.

9.4.1 OBTAIN SOLUTION ACCEPTANCE FOR RELEASE: INPUTS

9.4.1.1 APPROVED REQUIREMENTS

Described in Section 8.3.3.1. Approved requirements are requirements that are verified, validated, and deemed an accurate representation of what the product development team should build. Approved requirements provide the baseline for comparisons made in release decisions between what was approved for the solution and what was actually implemented in the product.

9.4.1.2 EVALUATED ACCEPTANCE RESULTS

Described in Section 9.3.3.1. Evaluated acceptance results provide a summarized comparison between the acceptance criteria and the actual results, along with the root cause for variances or defects, the analysis of the cost to address the defect, and the business impact of addressing it or accepting it. For solutions developed using an adaptive life cycle, the definition of done is also part of these results.

Evaluated acceptance results provide concrete evidence of whether or not the solution has met or exceeded expectations. Any recommendations within evaluated acceptance results for how to deal with the discrepancies may impact when and whether a solution is accepted for release.

9.4.1.3 PRODUCT RISK ANALYSIS

Described in Section 7.8.3.1. The product risk analysis includes the consolidated results from identifying and analyzing product risks. The product risk analysis consists of the recommended responses to manage and control potential threats and strategies to take advantage of potential opportunities related to the product. This context is required when seeking solution acceptance for a release to determine the degree to which the product risks have been addressed. Product risks not addressed within the duration of the project may be transferred to operational teams to manage going forward.

9.4.1.4 READINESS ASSESSMENT

Described in Section 5.5.3.1. A readiness assessment is an evaluation of how well an organization is prepared for a change. It provides an evaluation of the ability of an organization to transition to the future state enabled by the solution. It also identifies risks to achieving readiness for the transition, and may also propose responses for how to address those risks. Consideration of whether any unaddressed readiness risks remain and whether the organization is truly ready for the release at the proposed point in time will figure into release decisions.

9.4.1.5 STAKEHOLDER ENGAGEMENT AND COMMUNICATION APPROACH

Described in Section 5.3.3.1. The stakeholder engagement and communication approach summarizes all the agreements for governing how stakeholders will be engaged and communicated with across the portfolio, program, or project. The stakeholder engagement and communication approach includes information about which roles are accountable for the release decision and how the release decision will be made.

9.4.1.6 TRANSITION PLAN

Described in Section 5.5.3.2. A transition plan is based on the readiness assessment as well as the transition strategy. It covers development of all the communication, rollout, training and user documentation procedure updates, business recovery updates, and other collateral and final production tasks needed to successfully cut over and adapt to the future state. It provides the information needed to coordinate and ensure that the release of the solution will occur at a time when the business can accept the changes and that any interruptions caused by the transition itself are not in conflict with other in-process programs and project work.

At a minimum, a transition plan would include a checklist of transition activities with “no later than” completion dates. When formalized, a transition plan would have a schedule developed in collaboration with and managed by those responsible for project management and operations. On projects using an adaptive delivery approach, rather than developing a formal transition plan, transition planning may manifest itself in setting up a reserved block of time or specific iteration to work through transition details, coordinating this time with the operational area that will own the product. For solutions involving software, this effort often includes time to clean up “technical debt,” the tactical workarounds within the solution that could result in a product that would be difficult to maintain or enhance over time. A very important part of obtaining solution acceptance involves confirming that the transition activities have been completed.

9.4.2 OBTAIN SOLUTION ACCEPTANCE FOR RELEASE: TOOLS AND TECHNIQUES

9.4.2.1 FACILITATED WORKSHOPS

Facilitated workshops use a structured meeting led by a skilled, neutral facilitator and a carefully selected group of stakeholders to collaborate and work toward a stated objective. Whenever possible, release decisions should be made during a facilitated workshop to allow all stakeholders to hear the rationale for the decisions made. The individuals who evaluated the acceptance criteria should attend and contribute to decision making. It can be helpful to provide summarized information in tabular or visual form (i.e., charts/graphs/pictorial) whenever possible, to help decision makers render their decisions. Summarizing the evaluated acceptance results, along with the remaining product and readiness risks, incomplete transition activities, and any gaps between approved requirements and what the solution will deliver, provides decision makers with the necessary information to render a decision more quickly. For more information on facilitated workshops, see Section 6.3.2.4.

9.4.2.2 GROUP DECISION-MAKING TECHNIQUES

Group decision-making techniques are used in a group setting to bring participants to a final decision on an issue or topic under discussion. To make the release decision, group decision making is conducted using an agreed-upon model for reaching a decision. The decision model should be identified as part of the stakeholder engagement and communication approach. For more information on group decision-making techniques, see Section 8.3.2.7.

9.4.3 OBTAIN SOLUTION ACCEPTANCE FOR RELEASE: OUTPUTS

9.4.3.1 RELEASE DECISION

A release decision may permit the release or partial release of the solution, delay it, or disapprove and prevent it. A release decision often includes a sign-off. The formality of the sign-off depends upon the type of project, the type of product, the project life cycle, the scale of the release, and corporate and regulatory constraints. Organizations with informal sign-off practices need to obtain sign-off in the manner that is acceptable to the organization. For more information on the format of a sign-off and typical situations where a formal sign-off is required, see Section 6.9 in Business Analysis for Practitioners: A Practice Guide.

9.4.4 OBTAIN SOLUTION ACCEPTANCE FOR RELEASE: TAILORING CONSIDERATIONS

Adaptive and predictive tailoring considerations for Obtain Solution Acceptance for Release are described in Table 9-4.

Table 9-4. Adaptive and Predictive Tailoring for Obtain Solution Acceptance for Release

Aspects to Be Tailored	Typical Adaptive Considerations	Typical Predictive Considerations
Name	Not a formally named process; performed as part of Demonstration and Feedback. For large-scale implementations, there may be a formal process to obtain solution acceptance for release	Obtain Solution Acceptance for Release
Approach	Often, verbal approval alone is obtained with the product owner, as part of demonstration and feedback, that a solution or solution segment meets the definition of done and can be released to production. For a full solution release using an adaptive life cycle, the approach may match what is done in a predictive life cycle. Some large organizations are moving to a DevOps approach, to coordinate activities and improve collaboration between development and operational areas. DevOps may also use automated practices to support release management, such as continuous integration and automated regression testing. These approaches can streamline the way organizations push releases to production and can make a release almost seamless with its approval.	A formal meeting to review inputs and render a decision on an entire solution or a substantial segment of a solution, as part of a phased release. This formal meeting to make the release decision may be part of an overall release management approach, which coordinates the release of all products.
Deliverables	Release decision with whatever level of formality of documentation is required.

9.4.5 OBTAIN SOLUTION ACCEPTANCE FOR RELEASE: COLLABORATION POINT

Business analysts collaborate with several roles when obtaining solution acceptance for a release. For formal solution acceptance approaches, a release management organization might be responsible for coordinating all release-related activities, including obtaining solution acceptance. In many organizations, someone in a project management role is typically accountable to obtain the sign-off. Operational managers may confirm that all business transition considerations have been sufficiently addressed and that they are comfortable with the solution and any workarounds associated with it. Quality control may be consulted to confirm that all product testing concerns have been sufficiently addressed, and architects and designers may be consulted to confirm that technical product concerns have been sufficiently addressed. For informal solution acceptance, project teams may make the release decision part of a demo and feedback process in collaboration with other product team members in participation.