The test results data includes but is limited to the following:

These are the test products produced by the test team to perform testing. They include but are not limited to the following:

This category includes a description of the individual defects uncovered during testing. Work Paper 11-1 can be used for recording and monitoring defects. This description includes but is not limited to the following:

The results of later investigations should be added to this information—for example, where the defect originated, when it was corrected, and when it was entered for retest.

Two types of efficiency can be evaluated during testing: software system and test. As the system is being developed, a process decomposes requirements into lower and lower levels. These levels normally include high- and low-level requirements, external and internal design, and the construction or build phase. While these phases are in progress, the testers decompose the requirements through a series of test phases, which are described in Steps 3 and 4 of the seven-step process.

Conducting testing is normally the reverse of the test development process. In other words, testing begins at the lowest level and the results are rolled up to the highest level. The final test report determines whether the requirements were met. Documenting, analyzing, and rolling up test results depend partially on the process of decomposing testing through a detailed level. The roll-up is the exact reverse of the test strategy and tactics.

The measurement team should include individuals who:

Representatives should come from management and personnel responsible for software development and maintenance. For an IT organization of fewer than 50 people, the measurement team should be between three and five members.

An inventory of existing measurements should be performed in accordance with a plan. If problems arise during the inventory, the plan and the inventory process should be modified accordingly. The formal inventory is a systematic and independent review of all existing measurements and metrics. All identified data must be validated to determine if they are valid and reliable.

The inventory process should start with an introductory meeting of the participants. The objective of this meeting is to review the inventory plan with management and representatives of the projects. A sample agenda for the introductory meeting would be to:

Creating an inventory involves the following activities:

To enable senior management to quickly access the status of each project, it is critical to develop a list of consistent measurements spanning all project lines. Initially, this can be challenging, but with cooperation and some negotiating, you can establish a reasonable list of measurements. Organizations with development and maintenance standards will have an easier time completing this step.

The objective of this step is to use the management criteria and measurement data developed in Steps 2 and 3 to define the process requirements for the management project reporting system. Major criteria of this specification include the following:

The objective of this step is to document the work process used to output the reports of the project data. The implementation will involve the following activities:

Monitoring the reporting process is very important because software tools are constantly being upgraded and manual supporting activities sometimes break down. The more successful the system, the better the chance that management will want to use it and perhaps expand the reporting criteria.

The two primary reports from this step are Summary Status and Project Status.

Summary Status Report

The Summary Status report (see Figure 11-2) provides general information about all projects and is divided into the following four sections:

• Report date information. The information in the report is listed as current as of the date in the top-left corner. The date the report was produced appears in the top-right corner.

• Project information. Project information appears in a column on the left side of the report. Each cell contains the project’s name, manager, phase, and sponsor.

• Timeline information. Timeline information appears in a chart that displays a project’s status over an 18-month period. It shows project status by measuring technical, budgeting, and scheduling considerations. The year and month (abbreviated with initials) appear along the top of the chart to indicate the month-by-month status of each project.

  Technical (T), scheduling (S), and budget (B) information also appears in the chart, and is specific to each project. These three considerations measure the status of each project:

  • Technical status (T) shows the degree to which the project is expected to function within the defined technical and/or business requirements.

  • Scheduling status (S) shows the degree to which the project is adhering to the current approved schedule.

  • Budgeting status (B) shows the degree to which the project is adhering to the current approved budget. Expenditures for the budget include funds, human resources, and other resources.

• Legend information. The report legend, which is located along the bottom of the page, defines the colors and symbols used in the report, including category and color codes. The following colors could be used to help to quickly identify project status:

  • A green circle could mean there are no major problems and that the project is expected to remain on schedule.

  • A yellow circle could indicate potentially serious deviation from project progression.

  • A red circle could mean a serious problem has occurred and will have a negative effect on project progression.

Figure 11-2. A Summary Status report.

Project Status Report

The Project Status report (see Figure 11-3) provides information related to a specific project component. The design of the report and use of color in your report enables the reader to quickly and easily access project information. It is divided into the following six sections:

• Vital project information. Vital project information appears along the top of the report. This information includes:

  • Date the report is issued

  • Name of the executive sponsoring the project

  • Name of project manager

  • Official name of project

  • Quick-status box containing a color-coded circle indicating the overall status of the project

• General Information. This section of the report appears inside a rectangular box that contains general information about the project. The work request number and a brief description of the project appear in the top half of the box. The lower half of the box shows the phase of the project (e.g., planning, requirements, development, and implementation), as well as important project dates and figures, which include:

  • Project start date, determined by official approval, sponsorship, and project management

  • Original target date for project completion

  • Current target date for project completion

  • Phase start date of the current phase

  • Original target date for completion of the current phase

  • Current target date for completion of the current phase

  • Original budget allotted for the project

  • Current budget allotted for the project

  • Expenses to date for the project

• Project/Activities. The Project/Activities chart measures the status according to the phase of the project.

  Future activities for the project are indicated by a bar, which extends to the expected date of project completion, or the current target date, identified by the abbreviation Tgt.

• Essential Elements. The Essential Elements section indicates the current status of the project by comparing it to the previous status of the project. The chart could use the color-coded circles and list considerations that allow the reader to quickly gather project statistics. These considerations ask:

  • Is the project on schedule?

  • Do the current project results meet the performance requirements?

  • Are the project costs within the projected budget?

  • Are the project costs over budget?

  • What is the dollar amount of the project budget overrun?

  These questions can be answered by comparing the previous report results (on the left side of the chart) to the current report results (on the right side of the chart).

  This section of the report also includes a graph that compares projected costs to actual costs. The projected cost line appears in one color; the actual cost line appears in another. The dollar amounts appear on the left side of the graph, and the time line appears along the bottom of the graph. This graph shows you whether the project is adhering to the current approved budget.

• Legend information. The report legend, which is located along the bottom of the page, defines the colors and symbols used in the report, including category and color codes.

• Project highlights information. The project highlights appear in a rectangular box located at the bottom of the report. This section may also contain comments explaining specific project developments.

Figure 11-3. A Project Status report.

The function/test matrix shows which tests must be performed to validate the software functions, and in what sequence to perform the tests. It will also be used to determine the status of testing.

Many organizations use a spreadsheet package to maintain test results. The intersection of the test and the function can be color-coded or coded with a number or symbol to indicate the following:

Testers should complete Work Paper 11-1 each time they uncover a defect. This information should be maintained electronically so that test managers and/or software users can review it. The information collected about each defect can be as simple or as complex as desired. For simplification purposes, it is suggested that the following guidelines be used:

The information from Work Paper 11-1 should be used to produce the function/test matrix, as shown in Table 11-1.

The report is designed to show the results of performing a specific test on a function. Therefore, no interpretation can be made about the results of the entire software system, only about the results of individual tests. However, if all the tests for a specific function are successful, testers can assume that function works. Nevertheless, “working” means that it has met the criteria in the test plan.

The purpose of this report is to show the percentage of functions, including the functions that have been fully tested, the functions that have been tested but contain errors, and the functions that have not been tested.

A sample of this test report is illustrated in Figure 11-4. It shows that approximately 45 percent of the functions tested have uncorrected errors, 35 percent were fully tested, and 10 percent were not tested.

Figure 11-4. A Functional Testing Status report.

The report is designed to show the status of the software system to the test manager and/or customers. How the status is interpreted will depend heavily on the point in the test process at which the report was prepared. As the implementation date approaches, a high number of functions tested with uncorrected errors and functions not tested should raise concerns about meeting the implementation date.

The purpose of this report is to show the status of testing and the probability that the software will be ready on the projected date.

Figure 11-5 shows an example of a Functions Working Timeline report. This report assumes a September implementation date and shows from January through September the percent of functions that should be working correctly at any point in time. The “Actual” line shows that the project is doing better than anticipated.

Figure 11-5. A Functions Working Timeline report.

If the actual performance is better than planned, the probability of meeting the implementation date is high. On the other hand, if the actual percent of functions working is less than planned, both the test manager and development team should be concerned and may want to extend the implementation date or add resources to testing and/or development.

The purpose of this report is to show whether the number of defects is higher or lower than expected. This assumes that the organization has sufficient historical data to project defect rates. It also assumes that the development process is sufficiently stable so that the defect rates are relatively consistent.

The example chart for the Expected versus Actual Defects Uncovered Timeline report in Figure 11-6 shows a project beginning in January with a September implementation date. For this project, almost 500 defects are expected; the expected line shows the cumulative anticipated rate for uncovering those defects. The “Actual” line shows that a higher number of defects than expected have been uncovered early in the project.

Figure 11-6. An Expected versus Actual Defects Uncovered Timeline report.

Generally, an actual defect rate varies from the expected rate because of special circumstances, and investigation is warranted. The cause may be the result of an inexperienced project team. Even when the actual defects are significantly less than expected, testers should be concerned because it may mean that the tests have not been effective and a large number of undetected defects remain in the software.

The purpose of this report is to list the backlog of detected but uncorrected defects. It requires recording defects as they are detected, and then again when they have been successfully corrected.

The example in Figure 11-7 shows a project beginning in January with a projected September implementation date. One line on the chart shows the cumulative number of defects uncovered during testing, and the second line shows the cumulative number of defects corrected by the development team, which have been retested to demonstrate that correctness. The gap represents the number of uncovered but uncorrected defects at any point in time.

Figure 11-7. A Defects Uncovered versus Corrected Gap Time Line report.

The ideal project would have a very small gap between these two timelines. If the gap becomes wide, it indicates that the backlog of uncorrected defects is growing, and that the probability the development team will be able to correct them prior to implementation date is decreasing. The development team must manage this gap to ensure that it remains narrow.

The purpose of this report is to show the breakdown of the gap presented in Figure 11-7 by defect type—that is, the actual number of defects by the three severity categories.

The Average Age of Uncorrected Defects by Type report example shows the three severity categories aged according to the average number of days since the defect was detected. For example, it shows that the average critical defect is about 3 days old, the average major defect is about 10 days old, and the average minor defect is about 20 days old. The calculation is to accumulate the total number of days each defect has been waiting to be corrected, divided by the number of defects. Average days should be working days.

Figure 11-8 shows a desirable result, demonstrating that critical defects are being corrected faster than major defects, which are being corrected faster than minor defects. Organizations should have guidelines for how long defects at each level should be maintained before being corrected. Action should be taken accordingly based on actual age.

Figure 11-8. An Average Age of Uncorrected Defects by Type report.

The purpose of this report is to explain how defects are distributed among the modules/units being tested. It lists the total cumulative defects for each module being tested at any point in time.

The sample Defect Distribution report, shown in Figure 11-9, shows eight units along with the number of defects for each. The report could be enhanced to show the extent of testing that has occurred on the modules (for example, by color-coding the number of tests or by incorporating the number of tests into the bar as a number).

Figure 11-9. A Defect Distribution report.

This report can help identify modules that have an excessive defect rate. A variation of the report could list the cumulative defects by test—for example, defects uncovered in test 1, the cumulative defects uncovered by the end of test 2, the cumulative defects uncovered by test 3, and so forth. Frequently, modules that have abnormally high defect rates are those that have ineffective architecture and, thus, are candidates to be rewritten rather than for additional testing.

The purpose of this report is to normalize the defect distribution presented in Figure 11-9. The normalization can be by function points or lines of code. This will enable testers to compare the defect density among the modules/units.

The Normalized Defect Distribution report example in Figure 11-10 shows the same eight modules presented in Figure 11-9. However, in this example, the defect rates have been normalized to defects per 100 function points or defects per 1,000 lines of code, to enable the reader of the report to compare defect rates among the modules. This was not possible in Figure 11-9, because there was no size consideration. Again, a variation that shows the number of tests can be helpful in drawing conclusions.

Figure 11-10. A Normalized Defect Distribution report.

This report can help identify modules that have excessive defect rates. A variation of the report could show the cumulative defects by test: for example, the defects uncovered in test 1, the cumulative defects uncovered by the end of test 2, the cumulative defects uncovered by test 3, and so forth. Frequently, modules that have abnormally high defect rates are those that have ineffective architecture and, thus, are candidates for rewrite rather than additional testing.

This is a summary action report prepared by the test team. The information contained in the report should be listed as necessary to the test manager and/or the development manager so that they can properly direct the team toward a successful implementation date.

The Testing Action report example (see Figure 11-11) lists four pieces of information helpful to most test managers: total number of tests behind schedule, uncorrected critical defects, major uncorrected defects more than five days old, and the number of uncovered defects not corrected.

Figure 11-11. A Testing Action report.

These items are examples of what could be included in the Testing Action report. Most are included in the other reports, but this report is a summation, or a substitute, for the other reports.

The test manager should carefully monitor the status of testing and take action when testing falls behind schedule.

As testers complete an individual project they should issue an Interim Test report. The report should discuss the scope of the test, the results, what works and does not work, and recommendations (see Figure 11-12).

Any report about testing should indicate the test’s scope; otherwise, the reader will assume that exhaustive testing has occurred, which is never the case. Testing is a risk-oriented activity in which resources should be expended to minimize the major risks. Exhaustive testing is not possible, practical, or economical. Thus, testing is never designed to ensure that there are no defects remaining in the software, and the scope will explain what the testers accomplished.

The recommendations section is a critical part of the report, because the reader is usually removed from the project being tested and the technical recommendations provided by the testers can help with the reader’s business decision. For example, testers may indicate that there is a 50/50 probability that the system will terminate abnormally in production because of dating problems. A business decision might then be made to put the software into operation, but develop effective backup recovery procedures in case the termination occurs.

This report focuses on individual projects. When different testers test individual projects, they should prepare a report on their results. Refer to Figure 11-12 for a sample report format.

Integration testing tests the interfaces between individual projects. A good test plan will identify the interfaces and institute test conditions that will validate interfaces between software systems. Given this, the integration report follows the same format as the Individual Project Test report, except that the conditions tested are the interfaces between software systems.

Chapter 8 presented a system test plan standard that identified the objectives of testing, what was to be tested, how it was to be tested, and when tests should occur. The System Test report should present the results of executing that test plan (see Figure 11-13). If test data is maintained electronically, it need only be referenced, not included in the report.

Testing has two primary objectives. The first is to ensure that the system as implemented meets the software requirements. The second objective is to ensure that the software system can operate in the real-world user environment, which includes people with varying skills, attitudes, time pressures, business conditions, and so forth. This final report should address these issues. See Chapter 12 for conducting and reporting the results of acceptance testing.