Chapters 2 through 5 explained how to establish a test environment. Now you’re ready to:
Understand the advantages of following a process
Understand the costs associated with testing
Introduce the seven-step process that will take you through organizing, planning, testing, and completing your testing project
Customize the seven-step process to the needs of your organization
Establish a process to manage the seven-step testing process
The process for software testing described in this chapter is based on the experience of more than 1,000 organizations affiliated with the Quality Assurance Institute.
There is no best process for testing software. However, the seven-step process described in this chapter incorporates the best aspects of many different processes. Understanding and using a process for testing software provides the following advantages:
Testing is consistent. With a process, testing can be performed in a consistent manner from test to test. The use of the process will reduce variability of testing and improve confidence in the test process.
Testing can be taught. When testing is performed by a process, the process is teachable. When testing is performed as an art or craft, one must study under a master tester to learn how to test. The test process breaks testing into steps and tasks that are easy to teach.
Test processes can be improved. By using processes, one learns advantages and disadvantages of the process. Disadvantages can be identified and the process changed to continually improve testing.
Test processes are manageable. With processes, the test manager can manage the process. Without the test process, the test manager must manage people. From a control perspective, it’s much easier to manage or control a process than an individual.
There are two general categories of testing: pre-implementation and post-implementation testing. The first encompasses those activities that occur prior to placing the application system in an operational status. The objective of pre-implementation testing is to determine that the system functions as specified and that defects in the system are removed prior to placing the system into production. The second type of testing occurs after the system goes into operation and is normally considered part of systems maintenance.
The cost of removing system defects prior to the system going into production includes:
Building the defect into the system
Identifying the defect’s existence
Correcting the defect
Testing to determine that the defect has been removed
Defects uncovered after the system goes into operation generate the following costs:
Specifying and coding the defect into the system
Detecting the problem within the application system
Reporting the problem to the project manager and/or user
Correcting the problems caused by the defect
Operating the system until the defect is corrected
Correcting the defect
Testing to determine that the defect no longer exists
Integrating the corrected program(s) into production
Testing should include the cost to test plus the cost of undetected defects. Few organizations consolidate all the named costs as testing costs; therefore, an organization rarely knows the true cost of testing. Testing is normally considered to be that process used to find defects and ensure that the system functions properly. However, as illustrated, the cost of building and correcting defects may far exceed the cost of detecting those defects.
The National Institute of Standards and Technology has estimated that testing, including the correction of defects prior to the application going into production, accounts for at least half of the total system development effort.
The high cost of system defects poses the following two challenges to organizations: how to quantify the true cost of removing defects, and how to reduce the cost of testing.
Quality Assurance Institute surveys indicate that there are in the range of 20 to 60 defects in many application systems per 1,000 source statements. These surveys indicate that approximately two-thirds of the defects per 1,000 lines of source code occur in the requirements and design phases of application systems. Thus, while the defects are normally caught in the test phase of the system development life cycle, they occur early in the development process. (Note that as development processes mature, the number of defects produced is reduced.)
The causes of the defects built into application systems include:
Improperly interpreted requirements. IT personnel misinterpret what the user wants, but correctly implement what the IT people believe is wanted.
Users specify wrong requirements. The specifications given to IT personnel are erroneous.
Requirements are incorrectly recorded. IT personnel fail to record the specifications properly.
Design specifications incorrect. The application system design does not achieve the system requirements, but the design as specified may be correctly implemented.
Program specifications incorrect. The design specifications are incorrectly interpreted, making the program specifications inaccurate, but the program can be properly coded to achieve the correct program specifications.
Program coding error. The program is not coded according to the program specifications.
Program structural or instruction error. The programming capabilities are improperly utilized, resulting in defects attributable to misuse of a program instruction or the method in which the instruction is used.
Data entry error. The system and/or program information is incorrectly entered into the computer.
Testing error. The test either detects an error where there is no error or fails to detect an existing error in the application system.
Error correction mistake. In the process of correcting an error, the corrected condition contains a defect.
Corrected condition causes another defect. In the process of correcting a defect, a defect occurs in the unchanged portion of the software.
The areas associated with the test process can usually be readily identified. It is the estimation of the costs associated with these areas that is difficult to obtain. However, until the total cost of testing is known, the cost of uncovering and correcting defects will be unknown.
There are two methods for developing a more realistic estimate of testing. The first is to ask IT personnel to identify all the preceding conditions and allocate their time and effort accordingly. Although this concept works in theory, in practice it is difficult to record the time and effort associated with incurring defects until that defect is actually known. Because the point of uncovering defects may be many weeks or months after the actual day they were built into the system, it may be difficult to go back and recover these costs.
The second, and more practical, approach is to record the number of defects encountered as a result of testing. As each defect is uncovered, it should be noted, as well as the point in the system development life cycle process where it was uncovered.
The actual cost to redesign and correct the system should then be recorded. These are the costs required for correcting the programs by some recompilation and change of documentation. The costs are then multiplied by a factor that represents the totality of the error and problems associated with the defect as follows:
Defects corrected during design requirements phase. The cost to correct will be the total cost associated with the correction of the defect.
Defects corrected during building the software. The cost to correct requirement defects should be multiplied by a factor of 10 because the additional costs are associated with removing the defective components of the software.
Defects corrected after the system goes into production. The cost to correct will be approximately 100 times the cost to correct the same defect prior to placing the software into production.
The economics of computer testing clearly demonstrate that the method to reduce the cost of defects is to locate those defects as early in the system development life cycle as possible. This involves beginning testing during the requirements phase of the life cycle and continuing testing throughout the life cycle. The objective of testing would then become to detect the error as early in the life cycle as possible.
The seven-step software testing process follows the “V” concept of testing (see Figure 6-1). The V represents both the software development process and the seven-step software testing process. Both processes commence at the same time and proceed concurrently through the end of the project. Note that step 7, post-implementation analysis, will occur for both the development process and the test process. The purpose of this analysis is to determine whether development and/or testing can be performed more effectively in the future.
A brief overview of the seven-step software testing process follows:
Organizing for testing
Define test scope. Determine which type of testing is to be performed.
Organize the test team. Determine, based on the type of testing to be performed, who should be assigned to the test team.
Assess development plan and status. This is a prerequisite to building the test plan that will be used to evaluate the software implementation plan. During this step, testers will challenge the completeness and correctness of the development plan. Based on the extensiveness and completeness of the project plan, the testers will be in a position to estimate the amount of resources they will need to test the implemented software solution.
Developing the test plan
Perform risk analysis. Identify the test risks.
Write the test plan. Forming the plan for testing will follow the same pattern as any software planning process. The structure of all plans should be the same, but the content will vary based on the degree of risk the testers perceive as associated with the software being developed.
Verification testing
Test software requirements. Incomplete, inaccurate, or inconsistent requirements lead to most software failures. The inability to get the right requirements during the requirements phase will increase the cost of implementation significantly. Testers, through verification, must determine that the requirements are accurate and complete and that they do not conflict with one another.
Test software design. This step tests both external and internal design through the verification techniques. The testers are concerned that the design will in fact achieve the objectives of the project as well as being effective and efficient on the designated hardware.
Test software construction. The method chosen to build the software from the internal design document will determine the type and extensiveness of tests needed. As the construction becomes more automated, less testing will be required during this phase. However, if software is constructed by a manual coding process, it is subject to error and should be verified. Experience has shown that it is significantly cheaper to identify defects during the construction phase than through dynamic testing during the validation testing step.
Validation testing
Perform validation testing. This involves the testing of code in a dynamic state. The approach, methods, and tools specified in the test plan will be used to validate that the executable codes meets the stated software requirements and the structural specifications of the design.
Record test results. Document the results achieved through testing.
Analyzing and reporting test results
Analyze the test results. Examine the results of testing to determine where action is required because of variance between “what is” and “what should be.”
Develop test reports. Test reporting is a continuous process. It may be both oral and written. It is important that defects and concerns be reported to the appropriate parties as early as possible so that the can be corrected at the lowest possible cost.
Acceptance and operational testing
Perform acceptance testing. Acceptance testing enables users of the software to evaluate the applicability and usability of the software in performing their day-to-day job functions. This tests what the user believes the software should perform, as opposed to what the documented requirements state the software should perform.
Test software installation. Once the test team has confirmed that the software is ready for production, the ability to execute that software in a production environment should be tested. This tests the interface to operating software, related software, and operating procedures.
Test software changes. While this is shown as step 6 in the context of performing maintenance after the software is implemented, the concept is also applicable to changes throughout the implementation process. Whenever requirements change, the test plan must change, and the impact of that change on software systems must be tested and evaluated.
Post-implementation analysis. Test improvements can best be achieved by evaluating the effectiveness of testing at the end of each software test assignment. Although this assessment is primarily performed by the testers, it should involve the developers, users of the software, and quality assurance professionals if the function exists in the IT organization.
The following are the objectives of the seven-step testing process:
Organizing for testing. This step has two objectives. The first objective is to define what is to be tested. This is the scope of testing. It is not the objectives for a specific application, but the scope of the testing that will be performed to determine whether the application objectives have been met. Scope includes such things as the test to determine whether or not user needs have been met. Do you perform both static and dynamic testing? Do you test security? Do you test internal control? Second, you need to determine who will perform the test. This involves establishing the test team and determining what the individuals on that team will test.
Developing the test plan. The plan will determine how testing will be performed. During planning, the specific objectives for testing will be determined. For example, in the payroll system, a test objective might be to test the calculation of the payroll taxes. Both the specific project objectives and the scope objectives will be used to develop the detailed test plan.
Verification testing. The objective of verification testing is primarily to ensure that you are building the right system. You will also perform tracing to ensure that no requirements are lost as you move between developmental phases, but the main objective is verifying that you have built the right system. During verification testing, you will challenge the requirements and design of the system. The objective is to remove defects as close to the point they occur as possible.
Validation testing. The objective of validation testing is to determine whether you built the system right. In other words, does the system perform as expected? During validation testing, you will create test data and scripts and run those in a dynamic mode against the software to validate that, in fact, the output is the expected output.
Analyzing and reporting test results. The objective of this step is to determine what you have learned from testing and then inform the appropriate individuals. This would include what works, what doesn’t work, as well as any suggestions that the testers might make. It would also include analyzing whether such things as internal controls and security are adequate.
Acceptance and operational testing. The objective of this type of testing is to determine if you can use the system now, and that as the system is changed over time, it is still effective and efficient. Acceptance testing might be performed immediately prior to the software going into production, or it may be performed whenever changes are made to the system. The step also addresses controlling changes to the software application.
Post-implementation analysis. The objective of this step is future-oriented. It attempts to determine whether testing was performed effectively, and if not, what changes could be made to the testing process so that future testing will be more effective and efficient. Note that this can be done individually on a single system or on multiple systems simultaneously. This step can be performed by testers or by another group, such as quality assurance personnel.
The seven-step process defined in this chapter is a generic process. It is designed to test the most complex software testing system. It is designed to be used by both developers and an independent test team. If the seven-step process is to be effective, it must be customized for the organization using that process.
In order to get buy-in from those responsible for testing, they should be involved in the customization process. It is recommended that a small team of well-respected testers be organized for the purpose of customizing the process.
To customize the process, the following steps must be undertaken:
Understand the seven-step process. This involves reading Part Three of this book and perhaps discussing the process among the group.
Customize for “who” tests. Testing can be performed by both developers and testers. In some organizations, those who develop the software also test the software. In other organizations, those who test may be independent of the development team. As the test process is reviewed, it will become obvious that some of the steps are designed for independent testers and will not be needed for developers if they perform their own testing. For example, independent testers may want to know that the project has allocated adequate time for testing. Since it is the estimate of the developers, if the developers tested they would not have to do that step.
Customize for the size and type of system to be tested. The seven-step process is designed to enable you to test any software system and many different types of systems. For some systems, those which may pose no risk to the organization, it is not necessary to perform risk analysis as a prerequisite to testing. For small systems, the scope of the test plan might be reduced, for example, eliminating sections such as when to stop testing.
Customize for “what” to test. The seven step process is a generic testing process. It does not address such things as the platform to be tested, the type of developmental process, or specific testing needs, such as testing internal controls. Part Four of this book includes many different types of testing that may need to be incorporated into the seven-step testing process. The customization team should become familiar with the materials in Part Four of this book plus any specific testing needs they may have for their organization. These needs should be incorporated into the testing process if appropriate.
Customize for in-house developed and/or contracted software. The generic seven-step testing process is designed to be used with both in-house developed and contracted or purchased software. However, depending on whether the software is developed in-house or contracted, the steps in the process may need to be modified. For example, in contracted software, the testers may not have access to source code. Thus, they will be limited to black box testing and precluded from white box testing.
Customize for vocabulary. It is important that the testing process use the vocabulary of the organization. For example, if your organization does not use the terms verification and validation, you will want to change those phrases to vocabulary suitable to your organization. For example, your organization may refer to verification testing as static testing and validation testing as dynamic testing.
Integrate testing into the development process. Development and testing should not be completely independent processes. It is important that development and testing be integrated into a process that will produce quality software. If static testing is to be performed, for example, conducting a requirement review, that should be incorporated into the overall process of building software. If not, the developers may not allot adequate time for their participation in that aspect of testing.
Customization is a very important component of getting the software testing process implemented, accepted, and used by testers. It is important that they be involved in customizing this process for their organization. Even if you decide to use the seven-step process as is, your decision should involve the users of the process.
The test manager uses the seven-step process to manage the entire testing process. The test manager should also manage by process, by facts, and by results.
Manage by process. The test manager uses the test process to manage day-to-day testing activities. In preparing for testing, the test manager selects the test team and defines the specific test objectives for the software system being tested. The test process will then enable testers to accomplish those objectives.
The test plan provides the detailed budget, schedule, and tasks to be accomplished during testing. The test plan, combined with the test objectives, is in fact a contract with the stakeholders of the tested system. It is what the testers agreed to do in performing testing. The test manager needs to manage that plan as though it were a contract and ensure that all aspects of the plan are accomplished.
Manage by facts. The test manager needs to develop metrics in order to monitor quantitatively some key aspects of testing. This is considered the dashboard for test management. The test manager should select between three and eight key metrics to help manage testing. These metrics might include:
Budget
Schedule
Requirements tested/not tested
Status of testing, including such things as requirements tested that were implemented incorrectly
Requirements tested and not corrected as of a certain time period (for example, 10 days)
Status of defects
Manage by results. The process of testing is performed in order to accomplish specific objectives. For many organizations, the objectives will be the criteria for software acceptance—for example, the following customer needs:
Transactions can be processed by someone with X skills.
Ninety-six percent of all input transactions are acceptable for processing.
Changes to a product’s price can be made within 24 hours.
The test manager should create a quantitative dashboard of key indicators that will enable him or her to accomplish the test objectives. The types of key indicators the test manager may want on the dashboard include
Leaving customers satisfied with the testing process (customers may be developers, users, and/or management)
Meeting testing objectives
Completing the test plan’s contractual components
Accomplishing special test needs, such as validating the adequacy of system security
Chapter 3 introduced the tester’s workbench, which forms the template describing the procedures your test team will perform within the seven testing steps. Chapters 7 through 13 use the following workbench:
Overview. A brief description of the step. This will expand on the overview given earlier in this chapter for each step.
Objective. A detailed description of the purpose of the step that you can use to measure your progress at each step.
Concerns. Specific challenges that testers will have to overcome to complete the step effectively.
Workbench. A description of the process that the testers should follow to complete the step.
Input. The documents, information, and skills needed to complete the step.
Do procedures. Detailed, task-by-task procedures that testers must follow to perform the step.
Check procedures. A checklist that testers use to verify that they have performed a step correctly. These procedures will be related to the test’s objective.
Output. The deliverables that the testers must produce at the conclusion of each step.
Guidelines. Suggestions for performing each step more effectively and for avoiding problems.
Manufacturing positions frequently are designed so that workers require minimal skills to perform their tasks effectively. Job descriptions for these positions are accompanied by detailed documentation so that any worker could perform that function correctly. Professional positions, however, require more advanced skills and are usually accompanied by far inferior documentation. (It’s assumed that the person coming into the position will bring in a certain level of knowledge.)
A surgeon, for example, has to undergo 12 years of training before becoming licensed. Although there are detailed do and check procedures for performing a given operation, much of the execution depends on the doctor’s skills. The same is true when the systems analyst defines end-user requirements. The systems analyst is guided by work papers, but much of the innovative work needed to properly define requirements depends on his or her years of experience and skill level.
Figure 6-2 illustrates the relationship between the tester’s competency and the tester’s workbench. The workbench assumes an average skill level on the part of the reader, incorporating these assumptions into its descriptions of procedures and tools. The skills that a professional tester should possess are defined in the common body of knowledge (CBOK) for a software testing professional. Developed by the Software Certification Organization Certification Board, the CBOK is the basis used for evaluating the competency of testers.
If the people involved in the tester’s workbenches do not possess the basic testing skills in the CBOK, one or more of the following recommendations should be pursued to improve testing skills:
Attend a basic course on software testing
Take the necessary courses from the Quality Assurance Institute to prepare for the Certified Software Tester Examination
This chapter presented the proposed seven-step testing process. The chapter also introduced issues surrounding the costs associated with testing. This will be helpful for justifying testing and attempting to test in a cost-effective manner. The chapter presented six areas for customizing the seven-step process to make it more effective in your organization. The chapter also addressed how test managers should manage the seven-step process.
Chapters 7 through 13 describe each of the seven steps, respectively. Each chapter includes all the necessary templates and checklists to perform those steps. It is important when reading a chapter to recognize the objective is to provide the templates, checklists, and tasks needed to perform those steps. This differs from many testing books, which focus on what to do; these seven steps focus on how to perform software testing.