A timeline is a graphic representation of the sequence in which the steps are performed and the time needed to execute those steps. The timeline not only shows the time required to execute a process but also the time required for each step. It can also show the time for substeps. Timelines enable someone who is responsible for building an agile testing process to evaluate that process.

A process is the means by which a task is performed. Whereas in manufacturing most processes are automated, professional processes rely much more on the competence of the individual performing the process. A professional process consists of two components: the people tasked with completing the process, and the process itself. The process steps normally assume a level of competence for the individual performing the process, and therefore much of the process need not be documented. For example, a programmer coding in a specific language follows a process, which assumes that the programmer following the process is competent in that specific language. Therefore, the process does not attempt to explain the programming language.

A poorly defined process relies much more on the competency of the individual performing that process than does a well-defined process. For example, a poorly defined requirement-gathering process may require only that the defined requirements be easy to understand. A well-defined process may utilize an inspection team of requirement-gathering analysts to determine whether the requirements are easily understandable. In addition, a well-defined process may have a measurement process to measure easy-to-understand requirements.

Normally, as processes mature, more of the process is incorporated into the steps and less depends on the competency of the individual. Therefore, the reliance on people performing the process tends to go down as the process matures.

As the timeline is defined, the variability inherent in the process is also defined. Variability can include the quality of the products produced by the process, as well as the time required to produce those products. A process with extensive variability is considered to be “out of control,” whereas a process with acceptable variability is considered to be “in control.”

Figure 24-1 illustrates variability in the software testing process. Chart A shows a bell-shaped curve showing extensive variability. For example, to perform a specific task in the software testing process may take an average of 100 hours but have a variability of between 24 and 300 hours to perform that task. Chart B shows that same task in the software testing process with minimal variability. For example, the average time to perform that task, as illustrated in Chart B, may be 100 hours, with a range of between 90 and 110 hours to perform that task. Thus, Chart B shows a process under control, which is more desirable than the out-of-control process illustrated in Chart A.

Figure 24-1. Variability in the software testing process.

A process step has the following attributes:

Time-Compression Workbenches

The workbench concept needs to be expanded and used to identify causes of variability. Figure 24-2 shows a time-compression workbench. Added to this workbench are the activities that provide the greatest opportunity to reduce test-time variability. These additional activities are the ones that most commonly cause variability of a software test step.

Figure 24-2. A time-compression workbench.

The following four factors cause the most testing variability:

• Rework associated with not meeting entrance criteria. The entrance criteria define the criteria the input products must meet to be utilized by the workbench. Failure to meet entrance criteria means that the previous workbench has not met the exit criteria for that workbench.

  A more common scenario is that the entrance criteria for the input are not checked. In other words, the worker for the workbench accepts defective input products. This means that the defects are incorporated into the workbench activities. The net result is that the longer a defect “lives,” the more costly it is to correct it. An obvious time-compression strategy is to check entrance criteria.

• Worker competency. If the worker is not competent (skill sets or in using the work process), the probability of making defects increases significantly. Note that using the workbench effectively also assumes the worker is competent in using the tools in the toolbox. Therefore, an obvious time-compression strategy is to ensure the competency of the individuals using the workbench step.

• Internal rework. Internal rework generally indicates either the worker is incompetent or the processes provided to the worker are defective. In our testing workbench example, if the tester did not prepare tests for all the code specifications and those defects were uncovered by a test inspection process, internal rework would occur. An obvious time-compression strategy is to reduce internal rework.

• External rework. External rework refers to a situation in which the worker for a specific workbench cannot complete the workbench without additional input from previous workbench step(s).

Developing Timelines

A testing timeline shows the number of workdays needed to complete the testing. The three tasks that need to be followed to develop a software testing timeline are as follows:

1. Identify the workbenches.

2. Measure the time for each workbench via many testing projects.

3. Define the source of major variability in selected workbenches.

Identifying the Workbenches

The workbenches that create the software testing timeline must be defined. These are not all the workbenches in the software test process, just those that might be defined as “the critical path.” These workbenches, if lined end to end, determine the time span to complete the software testing.

There may be many other workbenches in software testing that need not be considered in this timeline definition. The types of workbenches that need not be considered include the following:

• Workbenches done in parallel with another workbench, but by themselves do not influence the timeline. For example, there may be a testing estimation workbench that is performed in parallel with defining the testing plan workbench; but it is defining the testing plan workbench that is the one constraining test time, rather than the estimating workbench.

• Report generation workbenches, such as time reporting, status reporting, and so forth.

• Workbenches/tasks normally performed during wait time, such as software testing documentation. Note that as the timeline is “shrunk,” some of these workbenches may, at a later time, affect the timeline.

• Estimating changes to the software testing plan.

• Staff training/staff meetings, unless they delay the completion of a critical workbench.

It is important to note that the process for “compressing” software testing time need not be a high-precision exercise. For example, if a critical workbench is left out, or a noncritical one added, it will not significantly affect the process for compressing testing time. Those corrections can be made as the process is repeated over time.

The workbenches defined for the timeline may or may not be the same workbenches/tasks/steps defined in the organization’s software testing process. The timeline workbenches may divide a workbench in the organization software testing process, or it may combine one or more steps/tasks in the testing process into a single workbench. What is important is that the workbench be a critical component of software testing (in the opinion of the team responsible for compressing testing).

The workbenches must be identified and defined in a sequence from the beginning to the end of the testing process. For each workbench, the following information is needed:

• Workbench name. The name may be one assigned by the software testing agile implementation team or the name given in the organization’s software testing process.

• Input(s). The entrance criteria that will initiate action for the workbench. The names of the input(s) should be those used in your organization’s software testing process.

• Workbench objective(s). The specific purpose for performing the workbench. This should be as specific as possible and, ideally, measurable. Note that in compressing the workbench, it is extremely important that those involved clearly understand the objective(s) for performing that workbench.

• Output(s). The exit criteria for products produced by the workbench. The output(s) should be identified by the name used in the organization’s software testing process.

• Approximate estimated timeline. This should be the number of workdays required to execute the workbench. The agile implementation team can determine the unit of measure they desire for the timeline. It can be days, half-days, or hours.

Once defined, this information should be transcribed to Work Paper 24-1. Note that this work paper is representative of what is needed; the actual work paper should be much larger.

Table 24-1. Define the Timeline Software Testing Workbenches

	WORKBENCH 1	WORKBENCH 2	WORKBENCH 3	WORKBENCH 4	WORKBENCH 5
Input(s)
Workbench Name
Workbench Objective(s)
Output(s)
Approximate Estimated Workdays Timeline

The timeline is a completion timeline and not a person-hour timeline. In other words, a three-day timeline is three workdays. It may, in fact, take several individuals all working the three available workdays to complete the task in three days.

Measuring the Time for Each Workbench via Many Testing Projects

The objective of this task is to measure an approximate workbench timeline in workdays for each workbench included in Work Paper 24-1. Note that this would normally be done only for those workbenches that have an extended (that is, large) calendar-day timeline. For each workbench selected to calculate the completion timeline, Work Paper 24-2 should be completed. The objective of this work paper is to measure the completion timeline for a specific workbench for many different projects. If historical data is available, measuring the timeline for 10 to 20 different projects is ideal. However, in practice, most organizations calculate the timeline for fewer projects. If historical data is not available, it should be collected from projects currently being tested.

Table 24-2. Workbench Completion Calendar Day Timeline

Workbench Name:
Project Timelines:
Project(s)	Start Date	Date “Do” Procedures Completed First Time	Date Workbench Completed	Minimal Timeline Calendar Days	Actual Timeline Calendar Days
Average No Rework Calendar Days Timeline:				No Rework Calendar Days Variability:
Average Actual Calendar Days Timeline:				Actual Calendar Days Variability:

For each project for which the workbench completion timeline is being calculated, the following information should be collected and recorded on Work Paper 24-2:

• Workbench name. The name assigned to the specific workbench on Work Paper 24-1.

• Project. The name of the software testing project for which the completion timeline is being documented.

• Start date. The calendar date that identifies when the workbench activity commenced.

• Date “do” procedures completed first time. This is the date on which the workbench was completed if no rework was required. In other words, if everything was done perfectly, this is the completion date. However, if rework was required, that rework extends the date of completion.

• Date workbench completed. This is the date that the work assigned this specific workbench was completed, including any rework.

• No-rework delivery days. This is the number of days between the start date for this project and the date the do procedures could have been completed the first time if there was not rework.

• Actual timeline days. This is the number of workdays between the start date of this workbench to the date the workbench was completed.

After the workbench completion timeline data has been collected for a reasonable number of projects, the following should be calculated:

• Average no-rework timeline. The average days as calculated in the “minimum timeline days” column. The no-rework completion timeline days are totaled for all the projects and divided by the number of projects to get this particular calculation.

• Variability of no-rework days. The variability for the average no-rework day timeline is the number of days that the workbench was completed earlier than the average no-rework days and the number of days it was completed later than the average no-rework days. For example, if the average number of days is five and one project is completed in three days and another in eight days, the variability is between plus three days and minus two days.

• Average actual days timeline. This is calculated by totaling the days in the “total timeline days” column and dividing by the number of projects.

• Variability of actual days. This is calculated by determining which project was completed earliest and which was latest. The number of days early from the average actual days and the number of days late produce the plus and minus variability.

Figure 24-3 shows an example of calculating the delivery timeline for three workbenches that performed testing. Note that to complete a timeline like this, you should use projects of equal size and complexity (perhaps by classifying projects tested as small, medium, or large).

Table 24-3. Workbench completion workday timeline.

Workbench Name:
Project Timelines:
Project(s)	Start Date	Date “Do” Procedures Completed First Time	Date Workbench Completed	Minimum Timeline Workdays	Actual Timeline Workdays
A	June 1	June 18	June 26	14	20
B	July 30	August 30	September 8	26	33
C	November 3	November 18	December 1
			÷ 3 =	18	25
Average No Rework Workdays Timeline:18			No Rework Workdays Variability:-4 to +8
Average Actual Workdays Timeline:25			Actual Workdays Variability:-5 to +8

As you can see in Figure 24-3, test planning was performed for three projects. For each project, the Minimum Timeline Workdays is the number of workdays between the start date and the date on which the first draft of the test plan was completed. The actual timeline workdays for each project is the number of workdays from the start of test planning until the test plan is complete. This example shows that the minimum average number of workdays for test planning is 18, with a variability of minus 4 and plus 8. The minus 4 means that some test planning projects were completed 4 days earlier than the average, and one was completed 8 days longer than the average. The same basic calculation in the example is performed for the actual timeline workdays.

Defining the Source of Major Variability in Selected Workbenches

For each workbench selected for calculating the workbench completion timeline using Work Paper 24-2, a variability completion timeline analysis should be performed. This analysis should be performed for both the best projects (i.e., completed in less than the average number of workdays) and the least-efficient projects (i.e., required more than the average number of workdays to be completed).

For projects with both the below- and above-average variability, an analysis should be done by the agile implementation team to determine where they believe the major source of variability occurred. For example, if a particular workbench took an average of five workdays and one project completed it in three workdays, the agile implementation team would need to determine the source of the two-day variability. For example, the project team that completed it in three days may have used a tool that none of the other projects used. In this case, the tool would be the source of variability. On the other hand, if a workbench is completed in an average of 5 workdays and one team took 10 workdays to complete it, the source of that variability may be lack of training or competency on the part of the individual performing the workbench. That lack of competency would be the source of the process variability.

The workbench components on Work Paper 24-3 are those previously described in this chapter. For the workbench being analyzed, the agile implementation team would identify one or more of what they believe are the source(s) of variability for that workbench. When they identify the source, they should then attempt to determine the probable cause. To complete Work Paper 24-3, the following needs to be determined:

• Variability analyzed. This analysis should be performed both for those projects below the average timeline workdays and again for those projects above the average timeline workdays. (Note: The team may decide only to evaluate the single best and worst performances of the workbench, or it may analyze several of the better and several of the worst performances of the workbench.)

• Source of variability. For each of the ten workbench components, the team should determine whether they believe that component is or is not the source of variability.

• Root cause. For each workbench component the team believes is the source of variability, they should try to determine the root cause of that variability. Again, it might be the use of a tool by one project, or the lack of training or competence on the part of the worker in another project.

Table 24-3. Completion Timeline Variability Analysis

Workbench Name:
Variability Analyzed:	Below Average		Above Average
Workbench Component	Source of Variability		Root Cause
	Yes	No
Input Criteria
Checking Input Criteria
Do Procedures
Check Procedures
Toolbox
Worker Competency
Internal Rework
External Rework
Exit Criteria
Other (specify)

The objective of performing Step 1 of the seven-step process to compress testing time is to identify some specific components of a testing process in which you can reduce the variability. When Work Papers 24-1 through 24-3 are complete, the agile implementation team should begin to develop a “shopping list” of potential process completion timeline improvements. Work Paper 24-4 is provided for that purpose. Note that this work paper is also used in Steps 2 and 3.

To complete Work Paper 24-4, the agile implementation team needs to provide the following information:

Work Paper 24-5 is a quality control checklist for Step 1 that the agile implementation team can use to minimize the probability of performing this step incorrectly.

If the investigation indicates that a particular aspect of this step was incorrectly performed, it should be repeated. (Note: Some teams prefer to review the quality control checklist before they begin the step to give them a fuller understanding of the intention of this step.)

This step has explained work processes and how an analysis of those work processes can lead to ideas for completion timeline compression. The step has provided a tool, the workbench concept, for analyzing each critical task in the software testing process. The step also provided an expanded workbench for timeline compression. This expanded workbench provides a process to identify ideas to compress the testing timeline. These ideas are used in Step 6 to identify the specific ideas that will be turned into an improvement process. The next step, Step 2, is designed to help the team identify potential causes of weakness in the testing process.

Agile systems maximize flexibility and minimize variability. The primary purpose of this step has been to identify the variability in the software testing process so that the variability in those steps/tasks that would be included in an agile testing process is minimized.

The traditional role of software testers is to validate that the requirements documented by the development team have, in fact, been implemented. This role makes two assumptions. First, that the requirements are, in fact, what the customer/user truly needs. Second, that part of the role of the tester is to identify, for the development team, those requirements that have been incorrectly implemented.

Problems are associated with the testers validating the defined requirements, as follows:

Quality factors are those attributes of software that relate to how the functional requirements are implemented. For example, ease of use is a quality factor. Requirements can be implemented in a manner that is not easy to use. However, customers/users rarely specify an ease-of-use requirement. The quality factors may be the deciding factors as to whether sales will be made.

Testers must change their role from validating developer-defined requirements to representing customers/users. As this discussion continues, keep in mind these two distinct focuses and how testers (representing users/customers) can eliminate many of the tester capability barriers.

Software Testing Relationships

The interaction of four relationships in software testing helps to determine the agility and the performance of testing. Each party has a perspective concerning the software testing, and that perspective affects test team performance capabilities:

1. The customer/user. The perspective of the customer/user focuses on what they need to accomplish, their business objectives (a subjective determination). They may or may not be able to express these business objectives in the detail software testers need. This perspective is often called “quality in perspective.”

2. The software development team. The development team focuses on how to implement user requirements. They seek to define the requirements to a level that allows the software to be developed (an objective determination). (If they can build that software and meet the requirements, however, they may or may not be concerned as to whether it is the right system for the user/customer.) This is often called the “quality in fact” perspective.

3. IT management. The environment established by IT management determines the testing methodology, methodology tools, estimating testing tools, and so forth. The environment determines what testers can do. For example, if they do not have an automated tool to generate test data, testers may be restricted as to the number of test transactions they can deal with.

4. The testers. The perspective of the testers focuses on building and executing a test plan to ensure the software meets the true needs of the user.

Operational Software

The testing capability barrier is two dimensional: One dimension is efficiency; the other dimension is effectiveness. However, these two dimensions are affected by software “quality factors.” Figure 24-4 shows that these quality factors affect how each party views its role compared to another stakeholder’s perspective.

Figure 24-4. Relationships affecting software testing.

Software Quality Factors

Software quality is judged based on a number of factors, as outlined in Figure 24-5. These factors are frequently referred to as “success factors,” in that if you satisfy user desire for each factor, you generally have a successful software system. These should be as much a part of the application specifications as are functional requirements.

Table 24-5. Software attributes (quality factors).

Factor	Definition
Correctness	Extent to which a program satisfies its specifications and fulfills the user’s mission objectives.
Reliability	Extent to which a program can be expected to perform its intended function with required precision.
Efficiency	The amount of computing resources and code required by a program to perform a function.
Integrity	Extent to which access to software or data by unauthorized persons can be controlled.
Usability	Effort required to learn, operate, prepare input, and interpret output of a program.
Maintainability	Effort required locating and fixing an error in an operational program.
Testability	Effort required testing a program to ensure that it performs its intended function.
Flexibility	Effort required modifying an operational program.
Portability	Effort required to transfer a program from one hardware configuration and/or software system environment to anther.
Reusability	Extent to which a program can be used in other applications related to the packaging and scope of the functions that program performs.
Interoperability	Effort required to couple one system with another.

Tradeoffs

It is incorrect to assume that with enough time and resources all quality factors can be maximized. For example, to optimize both integrity and usability is an unrealistic expectation. As the integrity of the system is improved (e.g., through more elaborate security procedures), using the system becomes more difficult (because the user must satisfy the security requirements). Thus, an inherent conflict is built in to these quality factors.

Figure 24-6 shows the relationship between the 11 software quality factors. Note in this figure that a relationship exist between efficiency and most other factors.

Figure 24-6. Relationships between software quality factors.

Figure 24-7 shows the impact of not specifying or incorporating all the quality factors in software testing. Let’s consider one quality factor: maintainability. Figure 24-7 shows that maintainability must be addressed during software test design, code, and testing. However, no significant impact occurs on the system test if maintainability has not been addressed in software testing. Likewise, no impact occurs on operation, initially, if software maintainability has not been addressed in software testing. What is crucial is that there is a high impact, because the software needs to be revised and transitioned into operation. Thus, a high cost is associated with software that is difficult to maintain.

Figure 24-7. The impact of not specifying software quality factors.

The objective of discussing quality factors, software testing relationships, and the relationships between quality factors and the tradeoffs of implementing such is to help you understand some of the reasons for performance barriers. Your software testing staffs can only develop software with a predefined effectiveness and a predefined efficiency. If you intend to compress software testing time, you must understand these barriers and incorporate them into the software testing time-compression activities.

Capability Chart

The easiest way to understand the capability barrier is to illustrate that barrier on a capability chart, shown in Figure 24-8. This figure shows the two dimensions of the chart: efficiency and effectiveness. Efficiency is a measure of the productivity of software testing, and effectiveness is a measurement of whether the test objectives are being met.

Figure 24-8. Software testing capability chart.

Figure 24-8 shows ten different projects. At the conclusion of each project, the project is measured for efficiency and effectiveness. The location on the software testing capability chart is determined by that efficiency/effectiveness measurement.

Let’s look at two examples. Project A rates very high on efficiency but low on effectiveness. In other words, the test team optimized the resources available to test a project in a very efficient manner. However, the customer of Project A is not satisfied with the test results. Project J is just the opposite. The customer of Project J is extremely pleased with the results, but the test team was very inefficient in testing that project.

In an effort to compress testing time, an agile implementation team should expect that tested projects on this chart will provide the solutions to compress testing time. For example, the practices used in testing Project A, if they were transferable to the other projects, might result in high test efficiency. Likewise, if the practices used in Project J could be transferred to all tested projects, there might be high customer satisfaction with all projects. Identifying these practices is the key component of this time-compression step.

The capability barrier line illustrated in Figure 24-8 represents the best an IT test team can do given current practices, staff competency, and management support. Agile testers must use new and innovative practices to enable an organization to break through their capability barrier.

A question that people ask about this capability chart is why the capability barrier line does not represent the results of the most efficient and most effective project. Note that if Project A were as effective as Project J, it would be outside the capability barrier line. The reason for this is the relationship between the quality factors. As an organization project becomes more efficient, other quality factors suffer. Likewise, if they become more effective, the efficiency factors deteriorate. Therefore, the best compromise between effectiveness and efficiency will be less than the most effective project or the most efficient project. (Note: This is only true using current best practices. New test practices may enable an organization to break through their testing capability barrier.)

Measuring Effectiveness and Efficiency

Measuring software testing efficiency and effectiveness involves two activities: defining the measurement criteria, and developing a metric for efficiency and a metric for effectiveness. Completed testing projects can then be measured by these two metrics. The result of measuring a specific software testing project is then posted to the software testing capability chart (see Figure 24-8).

There is no single correct way to measure software testing effectiveness and efficiency. The IT industry has not agreed on such a metric. On the other hand, many organizations perform these measurements regularly. The measurement is one of the activities that should be conducted at the conclusion of a software testing project.

The correct metric for efficiency and effectiveness is a metric that is agreed on by the involved parties. In a similar manner, there is no perfect way to measure the movement of the stock market. However, a metric called the Dow Jones Average has been developed and agreed on by the involved parties. Thereby, it becomes an acceptable metric for measuring stock market movement. Does everyone agree that the Dow Jones Average is perfect? No. On the other hand, there is general consensus that it is a good measurement of movement of the stock market.

General rules must be followed to create an acceptable metric, as follows:

• Easy to understand. The metric cannot be complex. The recommendation is that the metric range be between 0 and 100, because many measurements are on a scale of 0 to 100.

• Limited criteria. If there are too many criteria involved in the metric, it becomes overly complex. The recommendation is that no more than five criteria be included in the metric.

• Standardized and objective criteria. There needs to be a general definition of the criteria so that it can be used by different people and, as much as possible, the criteria should be objective (objective meaning that it can be counted as opposed to making a judgment).

• Weighted criteria. All criteria are not equal. Therefore, the metric should weight criteria. The most important criteria should get the most weight.

Defining Measurement Criteria

The most common way to measure software testing effectiveness is to determine whether testers can validate the presence or absence of the development team’s defined requirements. In this case, four different metrics enable you to measure software testing effectiveness, as follows:

• Requirements tested and found correct.

• Requirement does not execute as specified.

• Requirement is missing.

• Requirement found in tested software, but not specified by development team.

Measuring Quality Factors

The quality factor of “correctness” refers to testers validating whether the requirements specified by the development team work. However, the quality factor of correctness does not include all the other quality factors that the test team should be addressing. For example, it does not address whether the software is maintainable. Note that the software can be implemented with all the functional requirements in place and working, but cannot be efficiently or effectively maintained because the software was not built with maintenance in mind. For example, the logic may be so complex that it is difficult for a maintainer to implement a change in the software.

If the tester represents the customer/user, the tester’s responsibility may include testing some or all of the quality factors. This is generally determined in the relationship of the tester to the customer/user. If the tester is to evaluate more than the correctness quality factor, additional effectiveness criteria must be determined and used.

The generally accepted criteria used to define software quality are as follow:

• Traceability. Those attributes of the software that provide a thread from the requirements to the implementation with respect to the specific testing and operational environment.

• Completeness. Those attributes of the software that provide full implementation of the functions required.

• Consistency. Those attributes of the software that provide uniform design and implementation techniques and notation.

• Accuracy. Those attributes of the software that provide the required precision in calculations and outputs.

• Error tolerance. Those attributes of the software that provide continuity of operation under non-nominal conditions.

• Simplicity. Those attributes of the software that provide implementation of functions in the most understandable manner (usually avoidance of practices that increase complexity).

• Modularity. Those attributes of the software that provide a structure of highly independent modules.

• Generality. Those attributes of the software that provide breadth to the functions performed.

• Expandability. Those attributes of the software that provide for expansion of data storage requirements or computational functions.

• Instrumentation. Those attributes of the software that provide for the measurement of usage or identification of errors.

• Self-descriptiveness. Those attributes of the software that provide explanation of the implementation of a function.

• Execution efficiency. Those attributes of the software that provide for minimum processing time.

• Storage efficiency. Those attributes of the software that provide for minimum storage requirements during operation.

• Access control. Those attributes of the software that provide for control of the access of software and data.

• Access audit. Those attributes of the software that provide for an audit of the access of software and data.

• Operability. Those attributes of the software that determine operation and procedures concerned with the operation of the software.

• Training. Those attributes of the software that provide transition from current operation or initial familiarization.

• Communicativeness. Those attributes of the software that provide useful inputs and outputs that can be assimilated.

• Software system independence. Those attributes of the software that determine its dependency on the software environment (operating systems, utilities, input/output routines, etc.).

• Machine independent. Those attributes of the software that determine its dependency on the hardware system.

• Communications commonality. Those attributes of the software that provide the use of standard protocols and interface routines.

• Data commonality. Those attributes of the software that provide the use of standard data representations.

• Conciseness. Those attributes of the software that provide for implementation of a function with a minimum amount of code.

The software criteria listed here relate to the 11 software quality factors previously described. Figure 24-9 shows this relationship. For each of the factors, this figure shows which software criterion should be measured to determine whether the software factor requirements have been accomplished.

Table 24-9. Software criteria and related quality factors.

	Factor	Software Criteria
•	Correctness	Traceability Consistency Completeness
•	Reliability	Error Tolerance
		Consistency Accuracy Simplicity
•	Efficiency	Storage Efficiency Execution Efficiency
•	Integrity	Access Control Access Audit
•	Usability	Operability Training Communicativeness
•	Maintainability	Consistency
•	Flexibility	Modularity Generality Expandability
•	Testability	Simplicity Modularity Instrumentation Self-descriptiveness
•	Portability	Modularity Self-descriptiveness Machine Independence
•	Reusability	Software System Independence
•	Interoperability	Generality Modularity Software System Independence
		Modularity Communication Commonality Data Commonality

Note that some of the software criteria relates to more than one quality factor. For example, the modularity criterion is included in six of the factors, and consistency in three. As a general rule, you could assume that those criteria that appear most frequently have a higher relationship to the overall quality of the application system than do those criteria that appear only once. On the other hand, if the user rated a specific quality factor very high in importance, and that factor had a criterion that appeared only once, that criterion would be important to the success of the application as viewed from a user perspective.

The quality factors and the software criteria relate to the specific application system being developed. The desire for quality is heavily affected by the environment created by management to encourage the creation of quality products. An environment favorable to quality must incorporate those principles that encourage quality.

Defining Efficiency and Effectiveness Criteria

Efficiency has sometimes been defined as “doing the job right”; and effectiveness has sometimes been defined as “doing the right job.” Because of the lack of standards, IT metrics, and common software testing processes, it is difficult to reach agreement on efficient versus inefficient testing. For example, one would like an efficiency measure, such as X hours to test a requirement. However, because requirements are not equal, a requirement weighting factor would have to be developed and agreed on before this efficiency criteria could be used.

The lack of industry-accepted efficiency metrics should not eliminate the objective to measure testing efficiency.

There are many ways to measure software testing efficiency. One is the efficiency within the testing process. For example, do testers have the appropriate skills to use the testing tools? Does management support early involvement of testers in the development project? Are the testing processes stable and do they produce consistent results? Another way to measure software testing efficiency is the efficiency of the process itself.

Measuring Effectiveness

Measuring effectiveness of software testing normally focuses on the results achieved from testing. The generally acceptable criteria used to define/measure the effectiveness of testing are as follows:

• Customer satisfaction. How satisfied are the software customers with the results of software testing.

• Success criteria. Customers predefine quantitative success criteria. (If testers meet those criteria, it will be considered a successful project.)

• Measurable test objectives. Objectives stated in a manner that can be measured objectively so that it can be determined specifically whether the objectives are achieved.

• Service-level agreement. The contract between the customer and the testers as to the expected results of the testing efforts, how it will be measured, responsibilities in completing the testing, and other criteria considered important to the success of the test effort.

• Inconsistency between the test and customer culture. If customer and tester cultures are of different types, this can impede the success of the project. For example, if the customer believes in using and complying with a process, but testers do not, conflicts can occur. Step 5 addresses these culture differences.

Measuring Efficiency

Generally acceptable criteria used to define/measure software testing efficiency are as follows:

• Staff competency. Train staff members in skills necessary to perform those tasks within the software testing process. (For example, if staff members use a specific automated test tool, the staff members should be trained in the usage of the tool.)

• Maturity of software testing process. Usually a commonly accepted maturity level, such as the levels in SEI’s CMMI.

• Toolbox. The software testing effort contains the tools needed to do the testing efficiently.

• Management support. Management support of the use of testing processes and tools, meaning that management requires compliance to process and compliance to the use of specified tools, and rewards based on whether processes are followed and tools are used efficiently.

• Meets schedule. The software testing is completed in accordance with the defined schedule for software testing.

• Meets budget. The software testing is completed within budget.

• Software test defects. The number of defects that software testers make when testing software.

• Software testing rework. The percent of the total test effort expended in rework because of defects made by solution testers.

• Defect-removal efficiency. The percent of developer defects that were identified in a specific test phase as compared to the total number of defects in that phase. For example, if the developers made 100 requirement defects during the requirement phase, and the testers found 60 of those defects, the defect-removal efficiency for the requirements phase would be .6, or 60 percent.

• Percent of deliverables inspected. The criteria transferred from validation to verification, removing defects at a point where they are cheaper to remove.

• Software testing tool rework. The amount of resources consumed in using testing tools because of defects in those tools (or defects in how those tools were used).

Building Effectiveness and Efficiency Metrics

The agile implementation team should select the criteria used to measure the effectiveness and efficiency of the testing process. They should select what they believe are reasonable measurement criteria for both efficiency and effectiveness. These can be taken from the criteria examples described in this step or from criteria agreed on by the agile implementation team. They should document the criteria selected.

The measurement criteria should be recorded on Work Paper 24-6 and should include the following:

• Criteria name

• Description

• Efficiency

• Effectiveness

• Rank

Table 24-6. Criteria Recommended to Measure Software Testing Effectiveness and Efficiency

		Measuring
Criteria	Description	Efficiency	Effectiveness	Rank

After the agile implementation team has agreed on the criteria used to measure test efficiency and effectiveness, they need to rank those criteria. The ranking should be a two-part process:

1. Rank the criteria high, medium, or low. (Those ranked high are the best criteria for measurement.)

2. Starting with the high criteria, select no more than five criteria for efficiency and five for effectiveness. Note: These do not have to be those ranked high, but those ranked high should be considered before those ranked medium.

Both the efficiency and effectiveness metrics are developed the same way. It is recommended that three to five criteria be used. Then the criteria must be measurable. Use the following method to do that:

1. A method for calculating a criteria score must be determined. It is recommended that the calculated score for individual criteria be a range of 0 to 100. This is not necessary, but it simplifies the measurement process because most individuals are used to scoring a variable using a range of 0 to 100.

2. The criteria used for efficiency or effectiveness must then be weighted. The range should total 100 percent. For example, if there are five criteria and they all weighted equally, they each will be given a 20 percent rating or 20 percent of the total effectiveness or efficiency score.

3. To calculate a specific efficiency or effectiveness score for a project, the criteria score is multiplied by the weighting percentage to produce an efficiency or effectiveness score. The individual scores for efficiency and effectiveness are added to produce a total score.

Work Paper 24-7 is designed to record the criteria score and weighting. You can also use this work paper to calculate a total effectiveness and efficiency score for the selected software testing projects.

Table 24-7. Measuring Software Testing Effectiveness/Efficiency

Software Project Name:
Efficiency Criteria	Method to Calculate Criteria Score	Weight	Efficiency Score
		Total 100%
	Total Efficiency Score
Effectiveness Criteria	Method to Calculate Criteria Score	Weight	Effectiveness Score
		Total 100%
	Total Efficiency Score

Figure 24-10 shows an example of an efficiency and effectiveness metric. For both efficiency and effectiveness, three criteria are defined. For each criterion, a method to calculate the criteria score has been determined. The criteria score is then multiplied by the weighting percentage, and Figure 24-10 shows a project example efficiency score of 76 percent and an effectiveness score of 80 percent.

Figure 24-10. Examples of an effectiveness and efficiency metric.

After Work Paper 24-7 has been developed, the efficiency and effectiveness score for a reasonable number of software testing projects should be calculated. These can be historical projects or projects that will be completed in the near future. Those scores are then posted to Work Paper 24-8. Each project is to be posted to the intersection of the efficiency score and effectiveness score. Figure 24-10 shows an example of this posting. (Note: The circled criteria scores are the ones assigned for this “Project Example.”)

Figure 24-8. Recording Efficiency and Effectiveness Scores

Identifying Best Practices from Best Projects

The projects that score best in efficiency and the projects that score best in effectiveness should be selected for analysis. The agile implementation team should analyze those projects to determine the practices that cause some to be most efficient and some to be most effective. If the agile implementation team is knowledgeable in the software testing methodology, and has in-depth discussions with the test teams whose projects are either efficient or effective, the best practices can usually be identified.

The best practices identified should be posted to Work Paper 24-9. The information posted should include the following:

• Best practice. The name of the best practice, which may be named by the project team or it may be a specific commonly used best practice.

• Description. The description should clearly state the objective of the best practice.

• Project used in. The name of the software testing project that used this practice so that the project team can be consulted if more information is needed.

• Application efficiency/effectiveness. Indicate whether this is a best practice that improves efficiency or effectiveness. Check the appropriate column.

Table 24-9. Potential Best Practices for Compressing Software Testing Completion Time

Best Practice	Description	Project Used In	Application Efficiency	Application Effectiveness

By using these practices the software testing process can become more agile.

The end objective of performing Steps 1 through 3 of the seven-step process is to identify the specific components of a software testing process where the timeline can be reduced.

Work Paper 24-9 has identified the potential best practices for compressing software testing time. The practicality of using these best practices should be investigated. Those that the agile advancement team believes would enhance the agility of the software testing process should be identified as software testing agile best practices and posted to Work Paper 24-10.

To complete Work Paper 24-10, the agile implementation team needs to provide the following information:

Step 2 is an important aspect of compressing software testing time. A misunderstanding of the process or making an error in the performance of Step 2 can lead to missed opportunities to compress software testing time. A quality control checklist is provided for the agile implementation team to use to minimize the probability of performing this step incorrectly.

Work Paper 24-11 is a quality control checklist for Step 2. The investigation should focus on determining whether a specific aspect of the step was performed correctly or incorrectly.

Individuals should first review and answer the questions individually. Then the agile implementation team should review these questions as a team. A consensus Yes or No response should be determined. “No” responses must be explained and investigated. If the investigation indicates that the particular aspect of the step was incorrectly performed, it should be repeated. (Note: Some teams prefer to review the quality control checklist before they begin the step to give them a fuller understanding of the intention of this step.)

The objective of this step is to identify those best practices that, when implemented in new projects, will “compress” software testing time. The method to identify these best practices is to develop a metric that scores the effectiveness and efficiency of completed software testing projects. Those projects that scored high in efficiency are candidates to contain practices that will improve efficiency in future testing efforts. Those projects that scored high in effectiveness are candidates to contain best practices that will improve the effectiveness of future testing projects. By focusing on improving effectiveness and efficiency of the software testing effort, the total completion time should compress. The product of this step is a list of best practices identified as ideas or candidates to use to compress software testing time. The next step focuses on assessing the strengths and weaknesses of the existing software testing process.

Software development processes were developed years before much effort was expended on building software testing processes. Even today, many organizations have loosely defined testing processes. In addition, many computer science curricula do not include software testing as a topic. This step defines four criteria for an effective software testing process, and provides you with an assessment to evaluate your software testing process against those criteria.

It is common today to relate effectiveness of a process to the maturity of the process, maturity meaning primarily that variability is removed from the process. This means that each time the process is performed, it produces similar results. For example, if an IT organization desires a defect-removal efficiency of 95 percent in the requirements phase, a process that meets that objective time after time is considered a mature or effective process.

Software testing process characteristics make some more effective than others. In simplistic terms, the answers to these questions will help you determine the effectiveness of a process:

The five primary components of a software testing process are as follows:

Efficient test processes perform those five components with minimal resources. They do the work “right the first time.” This means that there are minimal defects and minimal rework performed by the software testers.

When software testing is treated as an art form, and depends on the experience and judgment of the testers, it may or may not be effective and efficient. If it is effective and efficient, it is primarily because of the competency of the individual testers. However, when a test process is people dependent, it is not necessarily repeatable or transferable between projects.

Developing and Interpreting the Testing Footprint

At the conclusion of assessing the five criteria in Work Paper 24-12, total the number of Yes responses in each criterion. Then post the number of Yes responses to Work Paper 24-13. This work paper is a software testing process assessment footprint chart. To complete this chart, put a dot in the criteria line that represents the number of Yes responses for that category. For example, if in category 1 (i.e., management commitment to quality software testing), you have two Yes responses, put a dot on the line on the number 2 circle. After posting the number of Yes responses for all five categories, draw a line between the five dots. The connection of the five dots results in a “footprint” that illustrates the assessment results of your software testing process. Explain your No responses in the Comments column.

Figure 24-13. Software Testing Process Assessment Footprint Chart

Three footprint interpretations can be easily made, as follows:

• Software testing process weaknesses. The criterion or criteria that score low in the number of Yes responses are areas of weakness in your software testing process. The items checked No indicate how you could strengthen a particular category.

• Software testing process strengths. Those criteria that have a high number of Yes responses indicate the strengths of your software testing process. Where your software testing process is strong, you want to ensure that all of your software testing projects benefit from those strengths.

• Minimal variability of strengths and weaknesses. Building an effective process involves moving the footprint envelope equally in all five criteria toward the five Yes response levels. The management commitment criterion would push the other four criteria in the right direction. However, there should not be significant variability between the five criteria. Ideally, they will all move outward toward the five Yes response levels one level at a time.

Examples of the type of analysis that you might make looking at this footprint are as follows:

1. If you scored a 5 for management commitment to the process, but the other categories are low in Yes responses, this indicates not “walking the walk” in utilization of an effective software testing process.

2. When there is a wide discrepancy between the numbers of Yes responses in different categories, it indicates a waste of resources. For example, investing a lot in the Do procedures for the process, but not building an effective environment, does not encourage consistency in use.

It is generally desirable to have the time-compression team evaluate their software testing footprint. They need to look at the footprint, draw some conclusions, discuss those conclusions, and then attempt to reach a consensus about them.

If a worker is given a poor process, that process reduces the probability of the individual being successful at performing the work task. Rather than the work task supporting success, the worker must modify, circumvent, or create new steps to accomplish the work task. In other words, the worker is diverted from doing an effective software testing job because of the time-consuming activity of surviving the use of a poor work process.

The objective of this self-assessment is to emphasize that work processes are much more than just the steps the software tester follows in performing the work task. For a work task to be effective, management must be committed to making that work process successful; management must provide the resources and training necessary to ensure worker competency and motivation in following the process; and management must provide the resources and skills sets needed to keep the software testing process current with the organization’s business and technical needs.

If that work process is enhanced using the principles of agile work processes, the workers will have the competency, motivation, empowerment, and flexibility needed to meet today’s software testing needs.

At the conclusion of the self-assessment process, the time-compression team should identify ideas for workday timeline improvements to your software testing process. Consider all category items assessed with a No response as a potential improvement idea. Record these ideas on Work Paper 24-14.

Work Paper 24-15 is a quality control checklist for Step 3. The quality control investigation should focus on determining whether a specific aspect of the step was performed correctly or incorrectly.

The agile implementation team should review these questions as a team. A consensus Yes or No response should be determined. “No” responses should be explained and investigated. If the investigation indicates that the particular aspect of the step was incorrectly performed, it should be repeated. (Note: Some teams prefer to review the quality control checklist before they begin the step to give them a fuller understanding of its intention.)

This step has proposed a process to self-assess the software testing process to identify ideas to compress testing delivery time. The step provided the criteria associated with effective software testing processes. These effectiveness criteria were converted to a software testing self-assessment work paper. The result of conducting the self-assessment was to develop a software testing process footprint. This footprint, coupled with the results of the self-assessment, should enable the agile implementation team to develop many ideas for compressing software testing delivery time.

The activity of compressing software testing time involves changing the way people do work. Thus, these individuals have a “stake” in the change happening or not happening. Individuals react differently as they consider what is their stake in the time-compression effort.

An important component in the people barrier concept is the “WIIFM” concept. WIIFM stands for “What’s in it for me?” If an individual cannot identify WIIFM in a proposed change, he or she will either not help the change occur, or will openly resist the change. Individuals have four different reactions to change:

The agile implementation team must identify the key individuals in the activities for compressing software testing time and determine what stake they have in the change. These individuals usually include the following:

It may be desirable to identify these individuals by name. For example, list the individual names of IT management. This should be done whenever the individual is considered personally involved in the change or can influence a change. In other instances, the stakeholders can be grouped (for example, testers). If the agile implementation team determines that the testers all have approximately the same stake in the change, they do not need to name them individually.

In viewing which stake an individual has, the agile implementation team must carefully evaluate the individual’s motives. For example, some individuals may outwardly support compressing software testing time because it is politically correct to do so. However, behind the scenes, they may believe that the wrong group is doing it or it is being done at the wrong time, and thus will openly support it, but are in fact, in the “Stop It From Happening” stakeholder quadrant.

This people barrier is highly individual. A red flag or hot button is something that causes a negative reaction on the part of a single individual. It is normally associated with someone in a management position, but can be associated with anyone who can exercise influence over the successful completion of a compression project. An individual, for a variety of reasons, may be strongly against a specific proposal. For example, if an improvement approach is assigned to “reduce defects,” the word defect may raise a red flag with a specific individual. They do not like the word. They prefer words such as problem or incident. Thus, by changing a word, a red flag can be avoided.

Other examples of red flags/hot buttons include the following:

Document these red flag/hot button barriers on Work Paper 24-17.

Changes to the software testing process that can compress software testing time may be desirable, but the existing skills must be available if the implementation of the change is to be successful. In some instances, this is judgment; for example, the manual work process needs to be changed, and the agile implementation team believes for that specific process no one in the IT organization has the needed skills. In other instances, missing skills are obvious. For example, a new tool is recommended, but no one in that organization knows how to use that tool.

Document these competency barriers on Work Paper 24-17.

In many instances, the administrative procedures are organizational barriers that inhibit change. Many believe that these administrative and organizational barriers are designed to slow change. Some people believe that rapid change is not good, and by imposing barriers they will cause more analysis before a change is made. Also, delaying implementation will enable all individuals to make known any personal concerns about the change.

A partial list of the administrative and organizational barriers that can inhibit compressing software testing time follows:

The organizational and administrative barriers should also be documented on Work Paper 24-17. At this point, it is unimportant whether a specific barrier/obstacle will stop a time-compression project. It is important to list as many barriers and obstacles as the agile implementation team feels might affect any time-compression project. The relationship between a specific barrier/obstacle and a specific time-compression project will be addressed when a plan is developed to implement a specific time-compression idea.

The barrier/obstacle cannot be adequately addressed until the root cause is determined. Likewise, the barrier/obstacle will continue to exist until the root cause of the barrier/obstacle has been addressed. The process of determining the root cause is an analytical process.

Consider an example. Assume the agile implementation team believes that a new software testing estimating tool will create a better estimate for software testing, in that it will allocate the appropriate time needed for defining software testing objectives. The agile implementation team may believe that if the testing objectives are better defined, they will “compress” the remainder of the software testing time. When the idea is presented for approval, the involved manager indicates that no funds are available to acquire the estimation tool. One might assume that the lack of funds is the root cause. However, when an analysis is done, it becomes obvious that the approving manager does not want that specific estimating tool, but prefers the current estimation method. If the agile implementation team focuses on obtaining funding, the tool will never be acquired. If team members identify the root cause as the manager’s resistance to the tool, their efforts will focus on convincing the manager that the tool will be beneficial. If they address that root cause, the funds might become available.

Work Paper 24-18 is the recommended analysis method the agile implementation team should undertake to identify the root cause for a specific barrier or obstacle. It is also called the “Why-Why” analysis. One of these work papers should be used for each barrier/obstacle listed on Work Paper 24-17 that the team believes should be analyzed to identify the root cause.

Figure 24-18. Barrier/Obstacle (“Why-Why”) Analysis

The agile implementation team then begins the “Why-Why” analysis. Let’s revisit the previous example. The barrier/obstacle is the inability to obtain management approval because of lack of funding. The agile implementation team then begins to ask the question “Why?” In this example, it would say “Why can’t we get management funding for the estimation tool?” The first answer is that no funds are available. They then begin to ask the question “Why are no funds available?” and they arrive at the conclusion that no funds are available because the root manager does not like the estimation tool. If the agile implementation team believes that is the root cause of the barrier/obstacle, they post that root cause to Work Paper 24-17.

This analysis may be simple to conclude, or it may be complex. A complex analysis may involve many primary and secondary contributors. The “Why-Why” analysis should continue until the agile implementation team believes they have found the root cause. If the barrier identified affects a specific compression-improvement plan, the plan should include the approach recommended to overcome the barrier. If that plan does not work, the “Why-Why” work paper should be revisited to look for another potential root cause.

The success of using this analysis depends on the agile implementation team having an in-depth understanding of how the organization works. They also need to know the traits and characteristics of the individual in authority to overcome the barrier. The analysis will not always lead to the root cause, but in most instances it does.

After the potential root cause of a specific barrier/obstacle has been identified, the agile implementation team needs to identify how they will address that specific root cause. The “how to address component” can be done in this step, or the agile implementation team can wait until they are developing a specific time-compression plan and then determine how to address the root cause.

There are no easy answers to how to address the root cause. Again, the more the agile implementation team knows about the organization and the characteristics of the key individuals in the organization, the easier the solutions become. Some of the solutions as to how to address the root cause of the barrier/obstacle include the following:

The determination about how to address a specific root cause of barrier/obstacle should be posted to Work Paper 24-17. Note the previous examples are only a partial list of the solutions for addressing the root cause(s) of barrier/obstacles. The agile implementation team should select specific solutions to address those root causes.

Work Paper 24-19 is a quality control checklist for Step 4. The investigation should focus on determining whether a specific aspect of the step was performed correctly or incorrectly.

The agile implementation team should review these questions as a team. A consensus Yes or No response should be determined. “No” responses should be explained and investigated. If the investigation indicates that the particular aspect of the step was incorrectly performed, it should be repeated. (Note: Some teams prefer to review the quality control checklist before they begin the step to give them a fuller understanding of the intention of this step.)

Everybody favors compressing software testing time. What they object to is the method proposed to compress software testing time. They might also object to the allocations of resources for compressing software testing time, especially when the IT organization is behind in implementing business software projects. It is important to recognize that individuals may believe that their career is partially dependent on completing a specific project. Anything that might delay that effort would be viewed negatively.

Many organizations establish approval processes designed to delay the quick implementation of projects. These processes ensure that the appropriate safeguards are in place, that only those projects desired by management are, in fact, implemented. These safeguards also become barriers and obstacles to implementing ideas that can compress software testing time.

Individuals who want to compress software testing time must be aware of what these barriers and obstacles are. Some are people related, whereas others are administrative and organizational related. For all of these, they must look for the root cause and then develop a solution to address that root cause. This step has provided a process to do just that.

There are five common management cultures in IT organizations worldwide (see Figure 24-11), as follows:

Figure 24-11. The five management cultures.

The five cultures are generally additive. In other words, when an organization moves to a culture of “manage by process,” the organization does not stop managing people. When it moves to the “manage by competency” culture, it is addressing the people issue of work processes. Measurement is not effective until the work processes are stabilized at the “manage by competency” level.

From the discussion of the five IT management cultures, the agile implementation team must identify their organization’s IT culture and the barriers and obstacles associated with that culture. Five tasks are involved in completing this step. The results of the tasks should be recorded on Work Paper 24-20.

Agile processes depend on open and effective communication. Open and effective communication has these three components:

For an in-depth discussion of the challenges testers face, refer to Chapter 8, “Step 2: Developing the Test Plan.”

Work Paper 24-22 is a quality control checklist for Step 5. The investigation should focus on determining whether a specific aspect of the step was performed correctly or incorrectly.

The agile implementation team should review these questions as a team. A consensus Yes or No response should be determined. “No” responses should be explained and investigated. If the investigation indicates that the particular aspect of the step was incorrectly performed, it should be repeated. (Note: Some teams prefer to review the quality control checklist before they begin the step to give them a fuller understanding of the intention of this step.)

This step, like Step 4, has identified some of the barriers and obstacles that need to be addressed to compress software testing time. Whereas Step 4 identified the root cause of the people and organizational barriers, the root cause of the culture barriers is known; it is the culture itself. It is the culture that primarily determines what information is shared and to whom it is shared. Ineffective and closed communication inhibits the building of an agile software testing process. Knowing these culture and communication barriers, and how to address the barriers, is an important part of the plan to compress software testing time.

An implementable refers to something that can actually be realized in an organization. It may not be the most effective or most efficient idea, but it is doable. The implementable is not the idea that is easiest or the quickest to realize, but it is doable. It is not the idea that will compress software testing time by the most workdays, necessarily, but it is doable.

Experience from IT organizations that have been effective in compressing software testing time show there are four criteria for implementables, as follows:

Obviously, other criteria affect whether an idea is doable. However, the preceding four criteria are most commonly associated with effective ideas.

Ultimately the decision about what is implemented to compress testing time must be based on the judgment of the agile implementation team. The team has knowledge of the software testing process and the IT organization. They should be motivated to do the job, and responsible for the success of the effort. However, many organizations use a process that may help in determining which are the better ideas for implementation.

This process uses the four criteria previously mentioned: user acceptance, barrier free, attainability, and effectiveness. By using a simple scoring system, organizations can score an idea based on these four criteria; that score will help determine how doable an idea is.

The recommended scoring system is to allocate a maximum of three points for each of the four criteria. The agile implementation team evaluates each criterion on a scale of 0 to 3. For a specific idea, 3 is the most desirable state of the criteria, and a 0 is an unacceptable state for the criteria. For each idea, the score for each of the four criteria should be multiplied. Multiplication is used rather than addition because when you multiply by 0 (zero), you eliminate any idea that is awarded a 0 in any of the four criteria.

Listed here is a scoring guide for the four criteria:

To complete the scoring, the agile implementation team should first list the high-priority improvement ideas developed in Steps 1 through 3 and post them to Work Paper 24-23. For each idea, they should then determine the score for the four criteria used to select an improvement idea for implementation. These four individual criteria scores are multiplied to obtain a doable score. The result will be an overall score between 0 and 81 (zero meaning the idea will be discarded, and 81 is the best possible idea identified through performing Steps 1 through 3).

At the end of this process, the ideas for compressing the software testing process would be ranked from 0 to 81 based on the actual scores. The idea getting the highest overall doable score should be considered the first idea to implement. The idea with the second highest score should be the second idea to implement.

You need to recognize that this selection process is not statistically valid. It includes judgment, but it does help rank the ideas by the probability of success. Obviously, a score of 27 may not be significantly different from a score of 24. However, the score of 24 would be significantly different from a score of 54.

The purpose of this selection process is to help the agile implementation team select the best overall idea for implementation. They should look at the ideas with the highest overall doable score as the ones they should select first for implementation. However, judgment would still apply, and one with a slightly lower score might be determined the better one for implementation even though it has a lower effectiveness score than the idea with the highest score.

After the ideas have been selected, a plan needs to be developed for implementation. It is generally recommended that a single idea be implemented. Then its results are evaluated before the next idea is implemented. This assumes the ideas can be implemented in a relatively short time span. Another advantage to implementing ideas in a series rather than many simultaneously is that it makes it easier for the software testing staff to assimilate the change and support it.

A scoring process that ranks implementables is an effective way to select ideas. However, the agile implementation team may want to prioritize the better ideas. Prioritization might prove beneficial in jumpstarting the agile software testing process, by implementing the ideas that could quickly add some agility to the existing test process.

Three sets of guidelines are provided to the agile implementation team to help identify the best doable ideas. These are as follows:

If the agile implementation team determines that it wants to apply judgment to the scored implementables, they should document the basis of that selection process. Work Paper 24-24 can be used for that purpose. To complete this work paper, the following needs to be documented:

Work Paper 24-25 is a quality control checklist for Step 6. The investigation should focus on determining whether a specific aspect of the step was performed correctly or incorrectly.