This chapter introduces a seven-step process to build agility into your software testing process through time compression. Agility not only improves software testing processes; it also adds flexibility and enhances ease of use. Thus, agility leads to a “continuous process enhancement,” which significantly increases testing volumes because of satisfaction with the process.
Testers avoid defective and difficult-to-use processes in the worst case; in the best case, they customize the processes, which means that they are not followed as intended. Testers pick and choose those pieces from the process they like, and improvise on those components they do not like, which leads to each testing group having its own testing process. The net result is significant effectiveness and efficiency variability among testing projects. When testers focus on adding agility to the process, the process improves, and all testing groups are likely to follow the agile process.
Agile systems are designed to be quick and easy to use. Much effort has been expended on the development of agile software development processes. Unfortunately, those software development processes have not placed much emphasis on software testing, even though in many systems, testing consumes about one-half the total developmental resources.
Effective agile processes have the following three characteristics:
Well-defined objectives
Open information/communication channels
Task-performance flexibility
The best way to build an agile software testing process is to empower the actual testers to build one. Software testers can build an agile testing process by focusing on the following three goals:
Eliminating nonproductive aspects of testing
Identifying and incorporating best testing practices
Building the process using the practices that offer the greatest payback with the highest probability of successful implementation
Process variability is the enemy of agile software testing. The first step to reduce variability is to identify and measure variability. Variability means the range of resources needed to accomplish a task. For example, if the time to prepare a script of 100 actions takes an average of 100 hours, with the most efficient person doing it in 50 hours, and the least efficient doing it in 150 hours, the process variability is 100 hours for that task.
Flexibility, the desirable attribute of agile software testing, is relatively affected by excessive process variability. For example, if someone prepares a test script using a process with extensive variability, the testing team cannot rely on a task being completed when needed.
As discussed more fully in the following section, common business practices inhibit agility. Remember, therefore, that agility will result only under the following conditions:
IT management supports and allots resources to building agile systems.
Testers are empowered to focus on business objectives, rather than just follow test processes.
Significant emphasis is placed on communication and team building.
To build an agile testing process, you must have a “champion” and an “agile implementation team.” Ideally, the champion would be the director of the organization or the Chief Information Officer. It is best that the champion not have direct responsibility for managing or performing software testing.
The agile implementation team can be as few as three or as many as five people, but each person should be a stakeholder in the testing. At a minimum, the team should have these representative members:
Software manager
Development representative, particularly, a project leader who respects the importance of testing
User representative
If more than three individuals are included in the agile implementation team, the additional numbers should be software testers. Although it is not necessary for a member of the process/engineering standards committee to be a member of the agile implementation team, that committee should be kept informed of all activities (and provide input to the process as necessary).
The seven-step process for time compression discussed later in this chapter addresses some of the specific responsibilities and tasks of the agile implementation team. This team does not have to perform all of the tasks involved in building an agile software testing process. For example, if a testing sub-process is to be changed, they might delegate that task to one or two testers knowledgeable in that specific task. However, the agile implementation team should develop a plan, obtain the resources, set work priorities, and oversee the implementation process.
The following ten factors inhibit agility in software testing processes (and thus contribute to inefficient and ineffective testing):
Test processes are not mature processes, making them prone to defects. Mature test processes have minimal variability. Mature test processes are continuously improved. When testers follow a mature test process, the process itself causes few defects. These characteristics of a mature process rarely exist in today’s testing processes.
Test processes are not designed to facilitate changing requirements. Business requirements change frequently. Opportunities to capitalize on new ideas occur frequently. When creating new information systems, it is difficult to fully define requirements at the start of the project. These factors result in constantly changing requirements. Unfortunately, test processes are not designed to handle these changes.
Quality is not defined in most software testing plans. Quality is defined as both meeting requirements and satisfying customer needs. However, in most test plans, quality is not an attribute of the plan.
Test processes are not flexible. No single process is effective in testing all software systems. However, flexibility, to adjust to the challenges individual test teams face, is rarely built in to test processes.
Test processes are not objective-driven. One of the key principles of agile systems is that everyone understands the objectives. Many testers are mired in the process of creating and executing test conditions and rarely are informed of the business objectives of the system. Therefore, they do not understand how the day-to-day work helps accomplish business objectives and add value.
Lines of communication between testers and developers, users, and IT management are not defined. Flexibility requires continuous and effective communication. It is important that all stakeholders be involved in the testing process, which means that testers need communication lines to those stakeholders. In most organizations, testers are not encouraged to initiate such communication.
Testers do not get the same respect as developers. In many organizations, testers are not considered equal to developers. This inequality is expressed in the amount of training provided, the resources expended on tools and processes, and the willingness to send testers to outside seminars and conferences; in many organizations, testers even have a lower pay grade than developers.
Open and honest communication up and down the chain of command does not exist. In many organizations, testers are viewed as the inhibitors of getting software into production by a scheduled date. Developers don’t confide their problems (the errors they believe may exist) in testers; and likewise, testers often keep the type of test they are conducting secret from the developers. This lack of communication can easily increase the cost of development and testing.
Testers are not actively involved, or given the resources to be involved, in determining their own processes. Experience has shown that the best test processes are those developed by the process users. Unfortunately, most test groups are not provided the time or the training needed to define, upgrade, and continually improve their test processes.
Minimal effort is expended on creating effective and efficient test teams. One of the most important principles of agile processes is focusing on building and upgrading effective and efficient teams. Most test managers are not trained in team building and team management. Many testers do not view themselves as team members, but merely individuals performing specific tasks.
To decide whether improvement is necessary, you must first ask whether software testing is currently performed efficiently. Then you must ask whether software testing is currently performed effectively. An agile process focuses efficiency (which implies time compression). Experience shows that when testing is efficient, it will also be effective.
From a management perspective, effectiveness is easier to determine than efficiency. You can easily measure effectiveness by using a metric called Defect Removal Efficiency (DRE). DRE refers to the percentage of defects removed through testing. In other words, if software contains 100 defects, and testing finds 75 of those defects, the DRE metric is 75 percent. The higher the DRE metric, the more effective the testing process.
Another way to measure the effectiveness of software testing is to calculate operational measures that relate to availability of software for production purposes. To do so, you must consider the mean time to failure (MTTF), the mean time between failures (MTBF), and the mean time to repair (MTTR). You can readily obtain this information from operational activities (see Figure 23-1).
The MTBF is the amount of time (or number of events) between the last failure and the current failure. Of course, failure must be defined (e.g., incorrect processes and/or non-operational software). As part of your consideration of MTBF, you must factor in the amount of time it takes to make the process/software operational after failure.
If you calculate MTBF on a regular basis, you can monitor the number of defects that remain. These types of calculations are beyond the scope of this book; MTFB is used here just to illustrate testing effectiveness. Simplistically, if MTBF is unchanging, it implies a replenishment of defects. In other words, as failures are corrected, new failures are introduced. On the other hand, if MTBF is increasing, you can statistically predict the time when the next failure will be detected.
The bottom-line measure of testing effectiveness is availability of the software, meaning not only available for use, but available for correct processing. If testing is “perfect,” the system will be continuously available for correct processing.
An analysis of DRE, MTTF, MTBF, and MTTR provides insight into testing effectiveness. If testing does not appear to be effective, time-compression efforts will improve both testing and effectiveness.
Many organizations consider software testing an art form rather than an engineering discipline. And although some might think that such an approach to testing is more closely related to an agile process, exactly the opposite is true. The art-form approach, although believed to be highly creative, is inefficient and leads to frustration (among testers and other expecting testing deliverables).
Quality Assurance Institute studies indicate that only about half of software testing groups develop plans for testing. Of those, only about half actually follow the plan in detail. The result is that testers have to spend time developing the testing process, instead of performing actual testing responsibilities.
A software testing process that eliminates most of the rework and communication problems among testers does not eliminate flexibility. A timely (calendar-day efficient) agile process can offer the flexibility testers need to perform their testing tasks.
Many testers understand that IT management already tries to save testing time by giving them unreasonable deadlines. If a scheduled implementation date has been established, and dynamic testing can occur only after code is complete, testers feel the pressure of that deadline. Two solutions to this dilemma exist: ask for more time, or complete the process more efficiently. Agile processes facilitate speed.
Many organizations fail to compress testing time because of the following challenges:
No time/no resources. Almost all IT organizations and individual staff members have more work than they can currently handle. The misconception is that if effort is expended on trying to save testing time, then software projects currently needed by customers/users cannot be completed on time. The dilemma is this: Do you spend your time doing the work of the business or do you expend your effort to compress testing time?
Long-term process. Most believe that the time to change a process is in direct relationship to the size of the process. In other words, it requires a massive re-engineering effort to change the software testing process because it is a large process. Software testing is among the most complex process in an IT organization; therefore, changing an extensive and complex process is a long, time-consuming and expensive process.
Industry models have resulted from efforts to improve quality and productivity. These models include the Software Engineering Institute (SEI) Computer Maturity Model (CMMI), the International Standards Organization Model, and the Malcolm Baldrige National Quality Award Model. Optimistic estimates to fully implement any of these models is three years (usually longer). Because few IT directors have a guaranteed tenure of three or more years, they are reluctant to undertake such an effort.
No in-house skill sets. Even if IT managers want to compress testing time, many do not believe the current staff possesses the skill sets necessary to do it. Without those skill sets, either a new process needs to be acquired, implemented, and taught, or consultants must be engaged to come in and perform the necessary analysis and changes to compress testing time.
Poor past experience. Almost since the day the first computer was delivered to an organization, vendors have promised new tools and techniques that will compress testing time and improve productivity. Many of these “vendor miracles” have resulted only in building bad software systems faster. Many of the miracles never really touched the root cause of too much time spent on testing. For example, avoiding test planning is not the way to compress software testing time.
Agile approaches to compressing software testing time are needed. Conventional approaches have not worked. Although many have helped, they are long-term solutions in an industry that prefers the short term.
Industrial engineers have used agile methods (known in that sector as time and motion studies) to compress the time required to build manufactured products since the late 1920s. These methods work.
To successfully compress software testing time, organizations must do the following:
Make “compressing” a project. Asking someone to do a task, but not providing the time, resources, and procedures to do it will not work. Many view unbudgeted tasks as a means by which management can get more work for the same money. In addition, unless compressing testing time is a project, it will not be managed like a project, and therefore the probability of success is diminished.
Focus solely on compressing time. There are many reasons why an IT organization might want to undertake an improvement effort, of which compressing testing time is but one focus. Objectives might include improving quality and productivity, providing better documentation, and compressing testing time. Unfortunately, when an individual is given many goals, it is difficult to differentiate the important from the unimportant goals.
Experience shows that if many variables are in play, selecting the most relevant variable will pull the other variables in the same direction. In other words, if process improvement focuses on the correct, single variable, the other desirable variables most likely will also be achieved.
Manufacturing experience shows that the correct variable to select is “time compression.” By focusing on that variable, quality, productivity, and the morale and motivation of the testing staff will be improved.
Use a “time-compression” process. If you ask someone to do a job, you need to tell them how to do it. Telling someone to compress the time to complete testing without providing a process to perform that “compression” effort has only a minimal probability of success. If you do not provide an individual with a process, they first must develop that process and then execute it.
Use in-house best-of-the-best work practices. Some software testing projects, when completed, have a high degree of customer satisfaction, whereas others do not. Some software testing projects are performed efficiently; some are not. A quick analysis can readily identify those projects exhibiting high customer satisfaction and those projects that are very efficiently developed. Finding the best practices from the best projects and then having everyone use these best practices is the quickest way to implement time-compression solutions in the software testing process.
Time compression will elicit agility. The following section consolidates time-compression methods into an easy-to-use seven-step process that, when implemented, will result in an agile software testing process.
Wanting to compress software testing time and having the organizational motivation and resources in place to do so are two different things. It is like dieting: You might want to lose weight, but the serious question is whether you are really ready to apply the discipline needed to lose weight. The same applies to readiness for time compression.
The following five criteria are the most significant “time-compression” readiness criteria:
Management support
Software testing process in place (The process can have significant variability, but the general activities in the process should exist and be documented.)
A need for time compression (e.g., an obvious benefit from such for customers, users, and business priorities)
Surmountable barrier/obstacle identification
Requisite resources
Testers (in consensus with a small group of key stakeholders in the effort and individuals respected by the teams involved) can use Work Paper 23-1 to evaluate the preceding criteria. If any readiness criterion has fewer than three Yes responses out of the five questions you must ask for each readiness criterion, your organization is not ready for a time-compression process. Any readiness criteria receiving three or four Yes responses should be evaluated to determine whether the items receiving a No response (i.e., not ready for a time-compression improvement process) might warrant more readiness preparatory work before undertaking the time-compression process.
Table 23-1. Readiness Assessment for Compressing Software Testing Time
Readiness Criteria: Management Support | YES | NO | COMMENTS |
Does the IT culture support using work process to test software systems? | |||
Would IT management support and encourage the more-effective testers to document their best testing practices? | |||
Would IT management be willing to become personally involved in the efforts to compress software testing time? | |||
Would IT management reward those who invest time and effort to compress software testing time? | |||
Do the IT strategic and tactical annual work plans include goals and objectives for compressing software testing time? | |||
TOTAL | |||
Readiness Criteria: A Software Testing Process in Place | YES | NO | COMMENTS |
Does a software testing process exist? | |||
Do most of the software testing projects follow the software testing process from, at least, an intent perspective? | |||
Have the software testers been trained in using the process? | |||
Is the process divided into self-contained testing activities? | |||
If so, do each of these self-contained activities contain entrance and exit criteria? | |||
TOTAL | |||
Readiness Criteria: Need for Time Compression | YES | NO | COMMENTS |
Do the users/customers of IT software want a shorter testing time? | |||
Does IT management want a shorter software testing time? | |||
Do IT project personnel want a shorter software testing time? | |||
Is there a backlog of software testing projects waiting to be undertaken? | |||
Is the inability to get software testing projects completed on a timely basis negatively affecting the business? | |||
TOTAL | |||
Readiness Criteria: Surmountable Barrier/Obstacle Identification | YES | NO | COMMENTS |
Are the cultural barriers against compliance to work processes surmountable? | |||
Are political obstacles to time compression surmountable? | |||
Are organizational barriers to time compression surmountable? | |||
Are budget and schedule constraint barriers to time compression surmountable? | |||
Are management hot buttons and red flags related to time compression surmountable? | |||
TOTAL | |||
Readiness Criteria: Requisite Resources | YES | NO | COMMENTS |
Are the tools needed for time compression available (e.g., consensus techniques)? | |||
Are the necessary skill sets available? | |||
Is the staff time needed available? | |||
Are the resources of the process engineering/standards committee available? | |||
Because compressing software testing time is achieved through many small efforts, will resources be available over an extended period of time? | |||
TOTAL |
If those key individuals having the responsibility for testing believe, after this analysis, that the organization is ready to compress testing time, begin the seven-step process outlined in the following section. If the individuals responsible for software testing have any reservations about the readiness to proceed, readiness work should be undertaken prior to beginning the seven-step process.
To compress software testing time/effort, follow these seven steps:
Measure software process variability. The timeline is the series of activities that must be executed to develop a software system. In this step, you define an activity by documenting a workbench for each activity. The workbench indicates the objective, the standards (for example, exit criteria), as well as the procedures to perform and check the work leading to the completion of that workbench. From this timeline process, you can determine time-compression opportunities.
Maximize best practices. Identify the software testing projects that are most effective and those that are most efficiently developed (effectiveness meaning high customer satisfaction, and efficiency meaning minimizing resources to complete the testing).
Build on strength, minimize weakness. An analysis of the testing process will indicate those areas that contain the greatest testing strengths and those where weaknesses exist. Obviously, you must focus your time-compression efforts on the areas of weaknesses.
Identify and address improvement barriers. Barriers are obstacles to time-compression improvements. The barriers can be budgetary or process oriented (i.e., getting permission to act), or the barrier may come in the form of an obstructionist on the team or in management. You must decide whether you can reduce or eliminate the barriers. If you cannot, you should explore another time-compression activity.
Identify and address cultural and communication barriers. An IT organization’s culture dictates, in many ways, the means by which the organization attempts to achieve results. For example, in a “management-object culture,” adherence to process is difficult to achieve. Any time-compression effort must be implemented within the cultural restrictions of an IT organization.
Identify doable improvements. The totality of the information you gather in Steps 1 through 5 are used in this step to identify those time-compression methods and techniques that have the greatest probability of success.
Develop and execute an implementation plan. After you have identified the time-compression improvement to seek, you must put a plan needs into place to implement the improvement and make it operational. Resource allocation is part of this step.
Steps 6 and 7 are repeated continuously to achieve shorter and shorter testing cycles. This is a never-ending process. Periodically, probably annually, you should repeat Steps 1 through 5 to identify new opportunities as well new barriers/obstacles.
It is not good practice to compress a bad software testing process. In other words, to do bad testing faster should not be the objective of time compression. However, industrial engineering experience has shown that if the time-compression effort is focused exclusively on time compression, it will improve both effectiveness and efficiency.
This chapter has described a seven-step process for compressing software testing time. It has also provided a readiness assessment to help determine whether your organization is ready to undertake this process.