Chapter 5. Test Team Management

Automated test tools are effective in providing solutions only when the problem at hand is adequately understood. Once the problem is understood, the challenge becomes identifying the individuals who will be responsible for solving it. Test efforts are complex and require a diverse set of skills. The test team needs to include members with the specific expertise necessary to comprehend the scope and depth of the required test effort and to develop a strategy to execute and implement the test program.

To execute a successful test program, the test team composition needs to be designed, the test team assembled, and the team members’ roles and responsibilities defined and assigned. Experienced people are needed to conduct test planning and write test procedures for manual and automated testing. To ensure that everyone on the test team is aware of what needs to be done and who will take the lead on a task, it is important to document the roles and responsibilities assigned to the various test team members.

The roles and responsibilities should be communicated both verbally and in written form and made available to everyone on the team. By identifying the assigned roles of each test team member on the project, everyone will have a clear understanding as to the individual responsible for each particular area of the project. Also, new team members can quickly determine whom to contact if a problem arises.

To get a clear picture of the individuals who are needed to support a particular task, a task description should be created. Once the scope of the task is understood, then assigning the specific talent to the task becomes easier. To help ensure successful execution of the task, work packages can be developed and distributed to the individuals involved in the task. Such packages typically detail the task organization, technical approach, task schedule, allocation of hours for each individual, and applicable standards and processes.

Along with members of the quality assurance department, the test team typically takes responsibility for an assortment of test and quality-related tasks, including the enforcement of development standards and code inspection standards on the project. If such project-specific standards are not available, the test team should inquire about organization-level standards that can be adopted or tailored to the project. Other sources for applicable standards, depending on the organizational needs, include the Software Engineering Institute (SEI), IEEE, ISO, and military standards organizations, among others. By the same token, the test team should be responsible for knowledge transfer of testing processes, methodologies, and test tools. For further detail on test team roles and responsibilities, see Section 5.5.

When test activities will span the entire development life cycle, the test team is in a position to facilitate communication among project development and test groups. This goal can be accomplished by conducting meetings to discuss and analyze the testing process, discuss the life-cycle approach to testing, provide test tool overviews, and conduct training.

Test teams need to inform project team members about the test approach defined for the project, allowing these personnel to be aware of the full scope of the test activities and to support the test process appropriately. This orientation can be both formal and informal in nature. For example, the test team might provide a presentation on the capabilities of the test tool being used in the test process. Test program orientation might also include test team participation in technical interchange meetings with end-user customers, if this type of communication is not already a standard practice.

As emphasized in the ATLM, the test team should attend requirements and design walkthroughs so that it can verify that requirements are testable, code is designed for testability, and traceability between requirements and design is possible. The test team may also decide to post test tool and process information on the organization’s intranet, together with appropriate links, such as those referring project personnel to the test tool vendor’s Web site.

The test team needs to work closely with the development group. It is beneficial to identify one lead contact from the development team and to have the test team work one on one with the development group whenever feasible. The test team should also be proactive in soliciting developer involvement in test activities. Lead test engineers, for example, should have a vision of test end goals in mind and communicate these goals to the development team.

Depending on the organization, the test team can be structured in a variety of ways. The potential organizational structures are discussed in the following section.

5.1 Organizational Structure of a Test Team

The structure of the test team will vary from organization to organization. Its depth and makeup also depend on the kind of product being tested and the mission of the test team. Several different organizational structures may be imposed on the test team that will be responsible for performing test activities and for introducing automated testing to a project. The flexibility that a test manager may have in designing a test team, however, will likely depend upon the organization’s culture and its influence on project team composition. Software testing is performed most often by testing staff who are divided among application groups (44%) rather than by a single, centralized software test department (39%) [1].

Some organizations hire and assign test engineers in response to the birth and retirement of projects. Within these organizations, long-term testing capability is not consciously managed. Job descriptions, which are developed for hiring purposes, are based entirely upon the particular needs of a specific project. Once the project goes away, the test engineers who worked on the project may leave, taking with them the experience gained with the application-under-test, test tools, lessons learned, and process and procedural insights. For purposes of discussion, we will call these structures stovepipe test teams.

Other organizations view professional software testing capability as a strategic investment. These organizations deploy centralized test teams. They seek to attract professional test engineers by offering career development paths within a centralized software test department, training and development in automated test tools, mentoring opportunities with more senior test engineers, increased job security by having the test engineer rotate between projects, and increased professional development opportunities achieved by the diversity of technology and tools experienced over a multitude of projects.

Besides test team organization strategies based along distributed or centralized organizational concepts, test team structure may also differ based on the team’s mission. Some organizations view testing as a last-ditch, stop-gap effort occurring at the end of the development cycle and classify it as an independent verification and validation (IV&V) effort. Test engineers assigned to an IV&V test team perform an acceptance test of the software application and review software documentation to validate product quality. Other test teams are seen as a nucleus of internal quality, testing, and process engineering, with consultants being made available to the various projects within the organization. In these organizations, testing is carried out by system methodology and test (SMT) groups.

Table 5.1 suggests sample compositions for each of these different types of test organizations.

Table 5.1. Test Team Profiles

5.1.1 Stovepipe Test Team

A manager responsible for a particular project will likely interview candidates for the test team and make the hiring decision. Given all of the various project start-up activities that demand the project manager’s attention, the test team may not be assembled until well into the project. The test plan document for the project may therefore be developed by the project manager prior to the assignment of test engineers.

A stovepipe test team typically includes two to five test engineers. These test team members do not belong to a separate testing organization entity but instead report to a task manager on the project or the actual project manager. One of the test engineers will act as a test lead, and the other test engineers will perform test development and execution activities.

Once in place, the test team will likely be introduced to the test tool, which was chosen during early project conception, or asked to perform a quick review of test tools so that the right tool can be procured and installed. In some situations, the project manager may inform the test team that no budget has been provided for a test tool, hand a copy of the test plan to the lead test engineer, and ask the team to begin developing test procedures.

The test lead develops a test design for the development of test procedures and then makes assignments for this development effort. After the test effort ends, the engineers may find themselves in the precarious position of not knowing where they will be assigned next. Once the project has concluded, no formal mechanism is available to ensure that the test effort’s lessons learned are retained. Likewise, the organization has no formal way to transfer test process, methodology, or automated tool knowledge and expertise.

5.1.2 Centralized Test Team

Some organizations view professional software testing capability as a strategic investment. They recognize that testing professionals possess skills with automated test tools, skills with network operations, functional knowledge of certain types of applications, and skills in software programming. In summary, they see the software testing professional as very versatile and as an important asset for the organization.

The centralized test team boasts a roster of test professionals who collectively support a number of projects. Each test engineer may work on one or more projects at a given time. As the test team may include 10 to 30 test engineers, the centralized test organization may need to be headed up by a test director. The test director has the full-time responsibility of maintaining the professionalism and technical expertise of the group. He or she must ensure that test activities are being performed correctly and within schedule on several projects. To fulfill this mission, the test director needs to ensure that the centralized test organization retains the services of test professionals with a variety of technical skills and a mix of skill levels.

During the start-up phase for a new project, the centralized test team can often supply personnel on a part-time basis to perform a number of activities that otherwise would be left to the project manager. These activities might include test effort sizing, development of test engineer job descriptions, test engineer interview support, test automation decision support (see Chapter 2), test tool evaluation and selection support (see Chapter 3), and test tool introduction support (see Chapter 4). The centralized test team generally provides an individual to develop the test plan for the project.

The centralized test team may be able to assign one or more full-time test engineers to the project from the outset. These engineers can help with the start-up activities and initiate full life-cycle support (the full range of tasks required for this step are discussed in Section 5.2).

The organization may find that the centralized test team offers a great deal of flexibility by being able to perform on a variety of projects. For instance, a test engineer could be assigned to a new project on a part-time basis to support the review of software requirements to ensure that the requirements are stated in testable terms. This part-time help may prove valuable when the test engineers slated for the project have not yet been hired or started working. In other situations, the centralized test team may provide additional test engineers during the project’s peak activity periods. On still other occasions, the project may require the services of a test engineer with a specialized test expertise for a limited period of time.

The fact that an organization has a centralized test team may in itself motivate a talented software test professional to join the organization. The test engineer will likely view the centralized test team as an organizational hierarchy within which it is possible to advance. He or she may also appreciate the mentoring opportunities possible with the team’s senior engineers. Other perceived benefits associated with the establishment of a centralized test team may include the availability of training, the diversity of technology and tools across projects, information sharing and technology transfer among the test engineers, and increased job security.

In some organizations, the centralized test team may be part of a larger organization responsible for implementing key software management and performance disciplines on projects. The larger organization may be referred to as a systems engineering support (SES) center or department. The SES organization commonly is responsible for implementing the level 2 and 3 key process areas outlined within the Software Engineering Institute’s (SEI’s) Capability Maturity Model (CMM). (For more information on the SEI’s CMM, see http://www.sei.cmu.edu/.) Its composition would likely include personnel who would perform tasks outside the test engineering discipline, such as those related to requirements management, risk management, quality assurance, and configuration management. (The composition of the centralized test team given in Table 5.1 focused only on the testing discipline; it does not reflect support for these other areas.)

Once the project has concluded, the test engineers performing on the project would report back to the centralized test team for their next assignments. Lessons learned from the test effort on the project are likely to be retained within a database maintained by the centralized test team. Likewise, the team probably maintains a repository of test processes, methodologies, procedures, test tool evaluation results, and test automation code libraries.

5.1.3 IV&V Test Team

Another possible structure for a test team is the independent verification and validation (IV&V) group. This type of team may consist of personnel who are assigned to a separate group within the software development organization or it may be a group that exists outside of the software development organization, such as a subcontractor. Often the IV&V group will decide whether a new software build is ready and acceptable for release. Its responsibilities are to participate in walkthroughs, verify that standards are being followed, perform quality assurance reviews of software documentation, and carry out other testing tasks. The IV&V group usually focuses on system testing (see Chapter 7 for more details on system testing) and is not concerned with the application’s internal processing.

The IV&V group is not responsible for the development of a project test plan but rather for ensuring that the test plan is thorough and complete, complies with standards, and supports appropriate system requirements and derived test requirements. It could therefore be seen as the first and most demanding user, meaning that business knowledge combined with technical knowledge is a must for an IV&V group member.

This type of organizational structure is fitting for a company that, for example, performs a significant amount of work in a particular business such as financial, logistics management, or space satellite control systems. The IV&V group structure may also prove advantageous for a large commercial software development organization or a company that maintains and possibly markets large financial services software programs. Within such an environment, it is beneficial for the organization to retain its investment by holding on to the expertise of the IV&V personnel.

Once a project has concluded, IV&V specialists report to a IV&V group manager or some other organization manager for their next assignments. Lessons learned from the IV&V test effort on the project may be retained, and the organization may maintain a repository of IV&V processes and procedures.

5.1.4 Systems Methodology and Test Team

In some organizations, the test technology transfer responsibility is assigned to a separate group within the same organization called the system methodology and test (SMT) team. This group’s reporting chain is kept separate from the immediate project organization. This test group is often responsible for carrying out test program start-up activities for a multitude of applications being simultaneously developed by the organization.

An SMT team is generally viewed as a group of internal consultants within the organization. The SMT staff is responsible for promoting knowledge transfer of methodologies and standards, promulgating guidelines for development and testing, developing and refining test techniques, performing evaluations and training on automated testing tools, and introducing test tools on projects. Team members should work one on one with the various project development team leads to perform the knowledge transfer and other activities.

The SMT team should include software professionals who have proven testing capabilities and a cerebral talent for conceptualizing, organizing, and planning. The process and methodology work require a disciplined approach to ensure that procedural steps explicitly cover all required actions and that the output of one step satisfies the input requirements for the next step. The capabilities required of SMT team members include an understanding of the complete testing life cycle and the software skills necessary to support test design, development, automation, and execution activities.

SMT personnel are usually heavily involved in project start-up activities as well as test planning and design activities, but take lesser roles during test development and execution. Their duties are not to perform testing on a project, but rather to consult, train, and mentor the project personnel who will actually carry out test development and execution.

Once a project’s test development effort begins to take off, the SMT test engineer generally returns to more routine SMT activities, which may include researching new test methodologies and tools, attending test tool conferences, maintaining the organization’s software and test process assets, maintaining a software project “lessons learned” database, and maintaining a repository of test tool evaluation results and test automation code libraries.

5.1.5 Test Team Summary

The structure of a test team depends on several variables, including the culture of the organization. The most important consequence of test team organization pertains to the likelihood of continual improvement in process maturity and software test capability. Structures that persist after a project end have the means to retain and improve upon the organization’s processes, procedures, and tool knowledge, as well as to transfer this knowledge to new projects. Table 5.2 describes the advantages and disadvantages for different test organizations.

Table 5.2. Test Team Comparison

The top 10 ingredients for a successful test group (all organization types) are as follows:

1. Business knowledge. Test engineers need to have business knowledge and work closely with the users and customers of the system.
2. Technical knowledge. Applications are more complex and, to understand the technical intricacies of the application, both automated test tools and a technical background are required.
3. Task separation. Business tasks are kept separate from technical tasks.
4. Resource management. Business and technical resources can be combined.
5. Relationship with development team. Test engineers work hand in hand with the developers.
6. Early life-cycle involvement. The test team becomes involved from the beginning of the development life cycle.
7. Defined test methodology. Methods, standards, and processes need to be in place, implemented, or modified as necessary.
8. Flexibility/adaptability. Each application is different. A testing strategy that has proven itself successful on one project might fail on another.
9. Metrics. The team can learn which metrics to collect and then use these metrics to improve the testing process. Metrics are collected throughout the development life cycle.
10. Process improvement. The team can strive for continuous improvement of the defined test methodology.

5.2 Test Program Tasks

Table 5.3 describes the different types of test tasks that can be performed. The work breakdown structure (WBS) depicted in the table can be used in conjunction with timekeeping activities to develop a historical record of the effort expended to perform the various activities on projects. The maintenance of such historical records is valuable in the test effort sizing exercise, which seeks to estimate the test effort required for a new project. The test program work breakdown structure will be helpful in determining the earned value metric of the test program, as discussed in Chapter 9.

Table 5.3. Test Program Work Breakdown Structure

Test teams may wish to further break down elements 8.7 and 9.3 in Table 5.3 so as to delineate test procedure/script development and execution according to the various testing subtypes. The possible test subtypes may include functional requirements testing, server performance testing, user interface testing, performance testing, program module complexity analysis, program code coverage testing, system load performance testing, boundary testing, security testing, memory leak testing, and response time performance testing, among others.

5.3 Test Effort Sizing

The test effort applied to a given project depends on a number of variables, such as the culture or test maturity of the organization, the scope of test requirements defined for the project, the skill levels of the individuals performing testing, and the type of test team organization supporting the test effort. The Test Maturity Model (TMM) addresses the level of test effort required on a project in relation to the testing maturity of the organization. The human resources costs vary according to the organization’s test maturity, as defined by TMM [2].

Level 1 Test Maturity

At test maturity level 1, testing is often limited to debugging. A programmer writes and debugs the product’s software until everything appears to work correctly. Because only the programmer is involved, testing costs often remain hidden within the cost of development. Likewise, the potential benefits of better test practices are hidden in field-support and product-upgrade costs. Historical costs pertaining to level 1 testing may be difficult to ascertain.

Level 2 Test Maturity

In level 2 software test programs, testing is recognized as a separate function. Test plans and scripts are generally developed by experienced product users or support personnel. These individuals may or may not have test automation (programming) experience. In any case, the individuals performing the testing must understand the software requirements and design specifications well enough to develop a comprehensive test plan and associated test scripts. Once the test scripts exist, they are provided to test engineers, who run them and record the results.

At level 2, the test team may consist of a group of more junior, relatively inexperienced end users or individuals with relevant functional knowledge. These individuals are given the job of trying to break the system as well as making sure that it works correctly. At level 2, the test effort may also include the services of one or more high-level support people who coordinate test writing, supervise the test engineers, and edit the results. Although a one-time set-up cost is incurred in implementing a capture/playback tool, this investment more than pays for itself when a number of test cycles are involved. During the later cycles, test scripts are reused and automatically played back, providing a huge labor saving with regard to script development and execution.

Levels 3–5 Test Maturity

At higher levels of testing maturity, the test engineer responsible for developing the test plan should participate in product development meetings with design engineers to help build testability into the product. The test engineer’s programming background combined with his or her familiarity with the product will promote the subsequent creation of efficient tests that address the weakest aspects of the product. If the test tool has white-box test capabilities, the test engineer uses his or her knowledge of the system’s internal workings to specify tests for functions that cannot be tested manually. The test plan serves to document the results of the test design. The test design, in turn, provides the guidance necessary for test engineers to develop the test script programs.

The test script development effort may be performed by either a team of test engineers or the application programmers. The level of programming experience required to develop test scripts depends on the test tool being used and the complexity of the tests. Generally, the most versatile tools employ scripts that are written in a common programming language, such as C. Other tools may use simplified languages. In any case, at least one member of the test team must have experience in developing a structured set of programming instructions. Automated tools are used to automatically generate test logs, document defects, and produce test status outputs. These tools provide significant labor savings with regard to test execution and management.

5.3.1 Test Team Sizing Methods: An Overview

Several methods are commonly used to determine the size of the test team needed to support a test effort. Historically, software development programs have focused on estimating the required development effort and overall project effort. Efforts related to product quality assurance, such as software testing, have then been sized in relation to the amount of anticipated development effort or overall project effort.

Commercial estimation tools such as COCOMO, PriceS, and SLIM require the input of various parameters associated with development size, productivity input, and the scope of project management activities. The accuracy of the output generated by these commercial tools, not surprisingly, reflects the quality of the input data. Few of these tools account for the growing importance and complexity of product-quality-assurance-related disciplines such as software testing by incorporating these considerations in the set of input factors used to generate resource and cost estimates.

The level of test effort required to support a given project depends on a number of variables, which can be used as input to complex estimation models to develop test team resource estimates. Other, simpler estimation models can be employed when values for a number of input parameters are lacking. Given the predominant focus within the industry on estimating the scope of the software development effort, attempts to size a test program often rely on the results of software development estimation, as reflected in the development ratio method.

5.3.2 Development Ratio Method

A quick and easy way to gauge the level of effort required to support a test program is to determine how many test engineers are needed based upon the number of software developers assigned to the project. The size of the test team is calculated by designating a desired ratio of developers on a project to test engineers. The term “developers” in this case includes those personnel dedicated to performing design, development, compilation, and unit-level test activities. Even though some developers’ roles may extend beyond traditional development activities, for purposes of using this ratio method, the classification of the developer is limited to these specific activities. This classification excludes those personnel performing (on a full-time basis) functional analysis, requirements management, configuration management, quality assurance, process improvement, project management, software test, training material development, and user manual documentation.

The developers-to-test engineers ratio will vary depending on the type of software development effort, as noted in Table 5.4. The ratios in Table 5.4 (which are derived from the authors’ experience) also assume that the scope of the test effort will involve functional and performance testing at the integration and system test levels. The values in the columns labeled “Developers Planned” and “Test Team Size” are expressed in terms of number of people.

Table 5.4. Development Ratio Method

Some mission-critical software projects may employ a higher number of test engineers than developers. The ratio between the number of application developers and test engineers, therefore, may reflect the roles defined for developers and test engineers. The figures in Table 5.4 assume that test engineers are involved only in executing testing life-cycle activities and do not perform any direct development work.

5.3.3 Percentage Method

Another quick way to estimate the level of effort required to support a test program is to use the percentage method, outlined in Table 5.5. This method considers the number of people planned to support a project in calculating the size of the test team. The test team size factor values given in Table 5.5 assume that the scope of the test effort will involve functional and performance testing at the integration and system test levels.

Table 5.5. Percentage Method

Also represented in Table 5.5 is sizing factor for a product assurance (PA) team. Organizations performing software development in accordance with maturity guidelines outlined within the CMM would require the services of individuals who support the various key process areas promulgated by the CMM. The PA team composition indicated in Table 5.5 includes both the test team personnel and personnel who undertake requirements management, configuration management, quality assurance, and process improvement. In this context, “process improvement” means that personnel supervise the effort to tailor organization processes to the specific project, provide project personnel with requisite training, and collect and analyze project performance metrics.

5.3.4 Test Procedure Method

Another way to estimate the level of effort required to support a test program is to use the number of test procedures planned for a project. The organization will need to develop a historical record of its development projects and their associated development sizing values, number of test procedures required, and the resulting test effort measured in terms of man-hours. Development sizing values may be documented in terms of lines of code (LOC), lines of code equivalent, function points, or number of objects produced.

Once this historical record exists, the test team could determine the past relationship between the sizing values and the number of test procedures developed, and then estimate the number of test procedures required for a new project. Once the number of test procedures was estimated, the test team could determine the historical relationship between the number of test procedures and the number of test team man-hours expended. This factor would then be used to estimate the number of man-hours or full-time equivalent personnel needed to support the test effort on the new project. The historical values are assumed to reflect the culture or test maturity of the organization and a correlation is presumed between the number of test procedures applied to a project and the scope of test requirements.

Table 5.6 gives an example of the use of the test procedure method, where a test team has estimated that a new project will require 1,120 test procedures. The team reviewed historical records, finding that a previous test effort involving 860 test procedures required a total of 5,300 hours. On this previous effort, the ratio of hours to test procedures was 6.16. The 5,300 hours were expended over a nine-month period giving a full-time equivalent of 3.4 test engineers for the project. Given the historical ratio of 6.16 for number of hours/number of test procedures, the test team calculates that it will take 6,900 hours to complete testing on the 1,120-test procedure project. As the new project is scheduled to be developed over 12 months, the test team calculates that 3.3 test engineers will be needed. This figure is calculated by dividing the total number of hours (6,900) by the number of hours per person (2,080) for the given period of performance (12 months).

Table 5.6. Test Procedure Method

5.3.5 Task Planning Method

Another way to estimate a test program’s level of effort required involves the review of historical records with regard to the number of man-hours expended to perform test program tasks for a similar type of test effort. The test team would need to perform time recording in accordance with a work breakdown structure like that depicted in Table 5.3. Historical records would then be developed that highlight the effort required to perform the various tasks. Next, the estimated number of test procedures (1,120) for the new project is compared with the historical sizing baseline, as shown in Table 5.7. The historical baseline indicates that the total number of man-hours for a project involving 860 test procedures is 5,300 hours, which represents a factor of 6.16. This factor is then used to estimate the number of hours required to perform a test effort involving 1,120 test procedures. This same historical comparison appeared in Table 5.6.

Table 5.7. New Project Man-Hours Estimation

The test team then reviews the historical makeup of the expended hours associated with the various test program tasks included in the work breakdown structure depicted in Table 5.3. Table 5.8 sums the hours needed for each WBS element.

Table 5.8. Task Planning Method

The test team develops a preliminary estimate of the hours for each WBS element using the historical percentage factor. Given that the new project has been mandated to use a particular automated test tool, the test effort is revised as indicated in the far right-hand column in Table 5.8. The test team had also been advised that the project would not be provided with any funding to cover test process improvement activities. The far right-hand column in Table 5.8 is therefore further adjusted.

Next, the test team computes the test team size based on the adjusted man-hour estimate of 6,403 hours as depicted in Table 5.9. The test team size is calculated to be equivalent to 3.1 test engineers over the 12-month project. In the event that the test team is staffed with exactly three full-time test personnel throughout the duration of the test effort, the team will need to achieve a slightly higher level of productivity than that of previous test teams so as to complete the test effort within the given schedule. It could implement a different staffing approach, applying two full-time test personnel and the fractional time of two other test engineers. The fractional time could be split a number of ways to include 50% of one person and 60% of a second person. Under this plan, it would be best for the two fractional-time personnel to be consistently used to perform a particular type of testing or test a particular functionality.

Table 5.9. Test Team Size

5.3.6 Test Effort Sizing Factors

The following factors should be considered when developing test effort sizing estimates:

1. Organization. This factor includes the culture or test maturity of the organization.
2. Scope of test requirements. Tests to be performed can include functional requirements testing, server performance testing, user interface testing, program module performance testing, program module complexity analysis, program code coverage testing, system load performance testing, boundary testing, security testing, memory leak testing, response time performance testing, and usability testing. Chapter 8 provides more information on the different kinds of testing.
3. Test engineer skill level. This factor encompasses the technical skill levels of the individuals performing the testing.
4. Test tool proficiency. The use of automated testing introduces a new level of complexity that a project’s test team may not have previously experienced. Test script programming is a required expertise that may be new to the test team, and few members of the test team may have had experience in performing coding. Even if the test team has had experience with one kind of an automated test tool, the tool required on the new project may be different.
5. Business knowledge. Test team personnel must have familiarity with the application business area.
6. Test team organization. This factor includes the type of test team organization supporting the test effort. Section 5.1 provides more information on test team organization types.
7. Scope of test program. An effective automated test program amounts to a development effort complete with strategy and goal planning, test requirements definition, analysis, design, and coding.
8. Start of test effort. Test activity and test planning should be initiated early in the project, which means that test engineers should be involved in analysis and design review activities. These reviews can serve as effective testing components, preventing analysis/design errors. Such involvement allows the test team to more completely understand the project’s requirements and design, develop the most appropriate test environment, and generate a more thorough test design. Early involvement not only supports effective test design, which is a critically important activity when utilizing an automated test tool, but also enables early detection of errors and prevents migration of errors from requirements specification to design, and from design into code.
9. Number of incremental software builds planned. Many industry software professionals assume that the use of automated test tools makes the software test effort less significant in terms of man-hours or less complex in terms of planning and execution. In reality, savings accrued from the use of automated test tools will take time to generate. At the first use of a particular automated test tool by a test team, very little savings may be realized. Instead, savings are realized in subsequent builds of a software application.
10. Process definition. Test team utilization of defined (documented) processes can improve the efficiency of test engineering operations. A lack of defined processes has an opposite effect, translating into a longer learning curve for junior test engineers.
11. Mission-critical applications. The scope and breadth of testing for mission-critical software applications, where a software failure poses a risk to human life or is crucial to the success of an organization, will be greater than that for software applications that do not pose a high risk. For example, testing of software that will control a heart monitor in a hospital setting is more critical than testing of the performance of game software.
12. Test development/execution schedule. Short timeframes to perform test development and execution may interject inefficiency into test engineering operations and require that additional test engineering effort be applied.

5.4 Test Engineer Recruiting

A test manager who faces the challenge of staffing and executing a test program on a project needs to recruit the right kind of test engineering talent. The manager likely has several questions in mind. What kind of person makes a good test engineer? What kind of skills should the test engineer have? How do I know which test engineer candidates are the best for the job?

Good software developers will have been trained and groomed to have a mindset to make something work and to develop a work-around solution, if necessary. Test engineers, on the other hand, need a capacity to be able to make things fail and a developer’s mentality to develop work-around solutions, if necessary, especially during the construction of test scripts.

Test engineers should be analytical, attentive to detail, and organized, and, given the complexities of automated testing, possess a creative and planning-ahead mindset. Because test engineers must work closely and cooperatively with software developers, they should be both assertive and poised when working through trouble reports and issues with developers. In addition, test engineers need a broad range of technical skills and experience across multiple platforms, operating systems, layers of supporting applications, interfaces to other products and systems, databases, and application languages. It is beneficial for them to be familiar with the script programming language of the primary automated test tool.

5.4.1 Test Engineer Qualities

The test engineer will be asked to carry out the various test program tasks listed in Table 5.3. This software professional should therefore be comfortable performing a variety of different tasks, often in parallel. The test engineer needs to be able to pick up new technology quickly. She should also have good conceptual skills to be able to understand the technical intricacies of the application.

The test engineer may perform testing on a mainframe application on one project and then test a client-server application on the next project. An automated test expert must be familiar not only with development techniques, but also with network, database, and middleware issues.

The skills required by the test team will depend on the kinds of tests performed and the test tools applied. For GUI testing, the test engineer may need to be familiar with Visual Basic, MS Access, SQL Server, and Windows-based operating systems. For server testing, he or she may need the ability to use C, C++, SQL, UNIX, and/or UNIX scripting and an understanding of remote procedure calls (RPCs). Applications operating in client-server environments depend on RPCs to bridge the gap between the processes in the different layers. Because the processing can be distributed across multiple platforms, the data must be delivered to the target platform in a form that can be read on that platform. Using RPCs makes the delivery and translation processes transparent to both client and server. The client program can receive the results of a server call as if the server were a compatible local procedure.

Qualities and skills to consider when recruiting for a test engineer are listed below. It may be worthwhile to document the skills desired of test team members within the application’s test plan.

1. Adaptability—able to perform in a variety of different technical environments, and familiar with different processes, tools, and methodologies.
2. Quick learner—enjoy performing a variety of different kinds of tasks, learning new things, and touching many different products.
3. Conceptual skills—aptitude for conceptualizing complex activities and articulating thoughts and ideas.
4. Organizational skills—understand complex test requirements and be able to formulate test planning and design approaches to support requirements; able to perform multiple responsibilities concurrently.
5. Problem solving—be able to develop work-around solutions to problems encountered during test development and execution.
6. Creativity—mindset to be able to think of a multitude of ways to exercise a system or application to ensure that it works in all circumstances; able to identify all conditions under which the software or system could fail.
7. Analytical/programming skills—training, experience, and skill to be able to develop automated test scripts.
8. Application business area knowledge—familiarity or understanding of the required functionality of the business application.
9. Diplomatic/cooperative—able to work closely and effectively with software developers; includes strong verbal communication skills.
10. Software professional—proficient at exercising the system and skillful in identifying and communicating problems to the developer.
11. Technical expertise—ability to set up and evaluate test tools; develop and maintain test data; control the test configuration and environment; and understand network, database, and middleware issues.
12. Test experience—level of test program experience. An effective test program incorporating the automation of software testing will involve a development life cycle of its own. The test engineer should have experience with test strategy and goal planning, test requirements definition, and test design, development, and execution.
13. Detail-oriented—pays attention to detail so as to identify difficult-to-find glitches and has a strong interest in improving the quality of a software product.
14. Process-oriented—skilled in the ability to understand inputs, logical sequences of steps, and expected output.
15. Writing/grammar skills—ability to effectively evaluate and improve requirements specifications and software design documentation.

5.4.2 Test Team Composition

To be able to recruit test engineers for a test program, it is necessary to understand the target composition of the test team. The test team, as a whole, will be responsible for fulfilling all test requirements for a project and performing all test program tasks. The effective execution of a test program requires a team with enough resident expertise to leverage the adopted test process and the applied test tools. The team will need enough familiarity and experience with the test tool to adequately plan, prepare, and execute test. The composition of the test team should be outlined within a test team profile like that depicted in Table 5.10.

Table 5.10. Test Team Profile

The test team profile in Table 5.10 portrays the composition of a team responsible for a test program involving the use of an automated test tool called QA Partner. The project involves the development of a client-server health care patient-scheduling and resource-management application operating on Windows client computers and servers running UNIX. The application is being developed in Visual Basic and C++ with a SQL Server back end. The scope of testing on the particular project involves functional requirements, server performance, the user interface, memory allocation, and system load testing.

The team described in Table 5.10 consists of a test manager, test lead, three test engineers, and a junior test engineer. This test team profile suggests that the test manager would need to have at least six years of software testing experience, including one to two years of test leadership experience. Ideally, the test leadership experience would include personnel supervision. In reality, the test manager’s experience might include both software development and software testing experience. Ideally, the test manager would also have at least one year of experience with the primary test tool planned for the project as well as some exposure to other test tools on the market. It would also be beneficial for this manager to be familiar with various software tools that support the management of the test effort or help team members navigate their way around the testbed environment.

While the test manager is responsible for the overall effort and focuses on long-term and strategic concerns, the test lead is responsible for the technical execution of the test program. The test team profile in Table 5.10 suggests that the test lead should have at least four years of test experience and at least two years of experience with the QA Partner test tool. The ideal skill set for the test lead includes abilities in the use of the Purify test tool and several programming languages, as well as familiarity with SQL Server relational databases.

The example test team profile also requires three test engineer positions. While all three would perform general test activities, two would ideally have experience in the relevant business area and one would have a proficiency in network engineering and administration. This proficiency might be reflected in network experience as well as credentials as a certified network engineer (CNE).

Rounding out the test team might be a very junior test engineer—right out of college or having only one or two years of software development experience. The junior test engineer supports the test team in a utility capacity. He would initially perform simple tasks, such as verifying the user manual and help files. The junior test engineer would gradually become more familiar with the test life cycle, the test tools being applied on the project, and the business application itself. It would be beneficial for this team member to have a variety of software skills, including training and academic experience in new software tools and programming languages.

5.4.3 Job Requisition

Armed with an understanding of the desired qualifications for a test engineer candidate as well as a mental framework for the composition of the test team, the test manager can initiate the recruiting process. For most organizations, a form must be completed to advertise for an open job position. This form may be referred to as a job requisition, position description, or recruiting request. Its purpose is to define the nature of the position, by indicating the required skills and credentials necessary to perform the job. The form is usually distributed within the organization and released to equal-opportunity employment offices and state or local employment agencies. For purposes of discussion, this chapter will call this form a job requisition.

The contents of the job requisition are fairly standard. The job requisition usually contains a posting date, position title, work location, and position identification number. It outlines duties to be performed in the position as well as required and desired skills. It will usually stipulate the level of education and number of years of experience required.

For each position identified within the test team profile outlined in Table 5.10, a job requisition would need to be created. A sample job requisition for the test lead position is provided in Figure 5.1.

Figure 5.1. Job Requisition

5.4.4 Recruiting Activities

The ability of an organization to effectively recruit a quality test engineer differs based upon the type of test team employed by the organization. Organizations that use the stovepipe test team organization (see Section 5.1) recruit and hire test professionals on a project-by-project basis. In this type of scenario, the organization responsible for the project assigns a project manager to oversee the start-up and execution of the project. During project start-up, this manager investigates and evaluates tools for the project, develops job requisitions for the types of personnel desired, interviews candidates for the various positions, and makes personnel hiring decisions. This project manager may have had limited experience with professional software testing programs and may not know how to define, recruit, or recognize the types of skills needed for the test team.

Other test team organizations, such as the centralized and SMT types outlined in Section 5.1, possess infrastructures that support the managed rotation of test engineering personnel between projects. These test organizations can generally draw on a network of test engineer professionals in the event that the team needs to be augmented with an additional test professional. By definition, these test organizations have professional test engineers on staff who are available to put together a job requisition, provide input for a recruitment advertisement, and interview test engineer candidates. They enjoy several advantages over stovepipe test organizations when it comes to attracting the candidate to join the organization, as noted in Section 5.1. Given that the demand for professional software test engineers has exceeded the supply in recent years—a trend that is expected to continue—successful recruiting and retention of test engineers may require a special effort. Employers may need to offer comprehensive compensation packages, training opportunities, flexible work hours, attractive job titles, and signing bonuses. Another important consideration is the availability of modern software engineering development and support tools, as well as the use of current technology computer equipment.

Centralized and SMT organizations often seek to attract professional test engineers by offering career development paths within a centralized software test department, training and development in automated test tools, mentoring opportunities with more senior test engineers, increased job security by having the test engineer rotate between projects, and increased professional development opportunities achieved by the diversity of technology and tools experienced over a multitude of projects.

5.4.5 Locating Test Engineers

Regardless of the type of test team organization, the individual responsible for screening and hiring test engineers must know how to locate job candidates. He or she must also be able to distinguish a superior test engineer candidate from a mediocre test engineer candidate during an interview.

Résumés for test engineers can be solicited or located via several means. One approach is to review the organization’s recruiting or résumé repository. Alternatively, one can solicit résumés in a job advertisement in a newspaper or magazine. A more proactive and potentially more cost-beneficial avenue is to query Internet résumé sources or to advertise open positions with test tool user groups or test-related newsgroups. Employee referral programs, when offered, are beneficial to solicit résumés of test engineers as well, and special promotions, such as those offering trips to exotic places, may be helpful.

Following the effort to retrieve or solicit test engineer résumés, the hiring manager needs to whittle the pile of prospective résumés down to a handful or so of those that most closely resemble the manager’s specific requirements. The manager must then screen the candidates in person.

5.4.6 Test Engineer Interviews

In preparation for test engineer interviews, the hiring manager should develop a list of relevant questions intended to confirm that the candidate has the resident expertise at the level of proficiency required. Once these questions have been prepared, they should be distributed to all staff members participating in the interview process.

The individuals conducting an interview should document or summarize the responses to each question. The interviewer should also jot down notes pertaining to observations about the candidate. This documentation will be helpful later when making a decision about whether to offer the position to a candidate or when trying to decide between two candidates. Where possible, the candidate should be interviewed by two or more people.

Some introductory questions to consider at the start of the interview are as follows:

1. It’s good to start the interview off with an open-ended question that allows the candidate to talk for a while. During this timeframe, the interviewer can assess the candidate’s communication skills and ability to organize his or her thoughts. Example: “Could you summarize your test background and interest in the test profession?”
2. A good second question will ask that the test engineering candidate to talk about their problem-solving ability. Example: “Could you describe how you’ve overcome technical problems and the kinds of results that you have had?”
3. The test engineer should be familiar with the test life cycle. Example: “Could you outline your perspective of the test engineering life cycle?”

Specific topics to consider when interviewing a test engineer are described below. The interviewer should ask the candidate about his or her experience regarding each topic. Depending on the qualification level of the test engineer being hired, these questions can be reduced or increased.

1. Analyzing system requirements for testability. Give examples of testable and nontestable requirements, and have the candidate define which requirements are deemed to be testable and which ones are not (for example, “the system shall allow for the addition of an unlimited amount of accounts” is a nontestable requirement).
2. Understanding the testing life cycle. The test engineer should know that the testing life cycle takes place in parallel with the development life cycle.
3. Deriving test requirements and test strategies. Ask the candidate to give examples.
4. Using an automated testing tool.
5. Evaluating an automated test tool.
6. Modifying automated test scripts. The candidate should have some development experience.
7. Verifying automated test tool compatibility with the application-under-test.
8. Finding work-around solutions to test tool incompatibility problems.
9. Planning for test tool introduction on a project.
10. Planning test activities.
11. Planning, tracking, and managing test environment setup activities.
12. Understanding the importance of baselining the testbed and test environment.
13. Identifying the kinds of tests to be performed.
14. Developing a test design.
15. Developing test data and refreshing the test database during test execution.
16. Performing data validation tests.
17. Inserting comments when recording scripts with an automated test tool.
18. Performing test readiness reviews.
19. Being able to break the system or make the application fail, and identifying and communicating the problem to the developer.
20. Documenting, tracking, and obtaining closure on trouble reports.
21. Fostering a close working relationship between developers and test engineers.
22. Performing test activities within the technical environment planned for the project.
23. Demonstrating the technical skills required for the position as reflected in the published job requisition.
24. Performing a variety of different kinds of tasks, learning new technologies, and performing multiple responsibilities concurrently.
25. Understanding the required functionality of the business application.
26. Working closely and effectively with software developers and users.
27. Understanding that users need to be involved from the beginning of and throughout the system development life cycle.
28. Understanding common network, database, and middleware issues.
29. Being familiar with the project-specific operating system, database, network, and development language.
30. Understanding metrics collection and metrics analysis.

5.4.7 Distinguishing the Best Candidate

Aside from reviewing the candidate’s personal qualities and his or her test engineering and technical skills, a hiring manager can take several steps to ensure that the test engineering candidate will successfully perform in the particular position:

1. Assess how the candidate asked questions and took the initiative during the interview.
2. Determine whether the candidate listened well and showed interest in the position and the company.
3. Listen for comments and responses that indicate that the candidate is a team-oriented person.
4. Listen for comments and responses that indicate that the candidate is a hard worker and committed to doing an excellent job.
5. Assess for how well the candidate is able to make important distinctions and share important insights with regard to an effective test process, design, and execution.
6. Assess how well the candidate answers questions. Are the answers brief and to the point, or do they go off on tangents, becoming incoherent or disorganized?
7. Inquire about how well the candidate performs programming. For junior candidates, find out what kind of grades they received in academic programming courses. A computer science degree offers some assurance that the candidate is analytical, intuitive, and persistent enough to successfully perform as a test engineer. Programming skills are important because they enable the test engineer to modify scripts so as to make them reusable and maintainable, minimizing the time required to regenerate test scripts.
8. Note whether the candidate speaks with respect about the testing profession.
9. Note whether the candidate speaks with respect about a previous boss, college professors, and members of college projects teams. Although technical skills are important, it is also crucial to select people who will work effectively with others, as the test engineer will need to advise developers about defects. The test engineer must be patient enough to stick with the project and with the organization through obstacles and tough challenges encountered on the project.
10. Don’t hire a test engineer who didn’t make it as a developer. Remember, a bad developer does not make a good test engineer.
11. Consider how well the test engineer’s personality fits with the rest of the test team. The test team needs to be populated with both leaders and warriors. A team mix including all of one type and none of the other may present a problem. If too many strong-headed or stubborn technicians (warriors) work on the project, they will likely be at odds with one another and the resulting friction may hamper test performance. If the project includes too many leaders, disagreements may arise regarding which direction to take, and the test team may lack enough technician strength to persevere through tough challenges.

5.5 Roles and Responsibilities

This section describes the major roles and responsibilities of individuals who perform test activities as well as those who typically have a close working relationship with test engineers. These roles and responsibilities should be tailored to the particular project and documented within the project test plan.

The number of test engineering roles required on a project may outnumber the number of actual test team positions. As a result, a test engineer may be responsible for more than one role—that is, she may “wear different hats.” The different test engineering roles required for the various test program tasks were listed in Table 5.3. To ensure that test roles are properly carried out, several practices may need to be considered, such as using consultants on a part-time or short-duration basis. Consideration should also be given to assigning an existing test engineer within the organization to act as a mentor for a more junior test engineer. In addition, the organization should cross-train test engineers across projects, different technical environments, and different test tools.

Table 5.11 lists the responsibilities and roles of the participants in the testing process.

Table 5.11. Testing Team Responsibilities and Roles

Test teams that plan to use automated test tools should have personnel on staff who have software development skills, as automated testing differs from manual testing in the way that test scripts are developed, executed, and managed. As a result, the skills and activities pertinent to test engineers who will perform manual testing differ from the skills and activities of test engineers who will perform automated software testing. Manual test roles are therefore listed separately in the defined roles and responsibilities in Table 5.11.

It is important that the correct roles be assigned to the appropriate person. Assess and leverage current strengths and assign accordingly. Organize and manage the test effort to take maximum advantage of the specialized skills of each member of the team and the specific purpose of each element of the test program.

Chapter Summary

• The test team needs to possess the specific expertise necessary to comprehend the scope and depth of the required test effort and be able to develop a strategy to execute and implement the test program. Test team composition needs to be designed, the test team assembled, and the roles and responsibilities of the test team defined.

• Several organizational structures are possible for the test team. Potential approaches include stovepipe test team, centralized test team, IV&V test team, and systems methodology and test team organizations.

• The most important consequence of test team organization pertains to the opportunities for continual improvement in process maturity and software test capability. Test team structures that persist after a project ends can readily retain and improve upon the organization’s processes, procedures, and tool knowledge, as well as transfer this knowledge to new projects.

• The different types of test tasks involved in a test program are commonly outlined within a work breakdown structure. This structure is then used in conjunction with timekeeping activities to develop a historical record of the effort expended to perform various test activities.

• Several methods are commonly employed to determine the size of the test team required to support a test effort. Size estimation approaches include the development ratio, percentage, test procedure, and task planning methods.

• A test engineer needs to possess the ability to discover defects. He needs a developer’s mentality to develop work-around solutions to incompatibility problems that may arise when using an automated test tool. The ideal test engineer is analytical, attentive to detail, and organized and, given the complexities of automated testing, has a creative and planning-ahead mindset. A test engineer also needs to be both assertive and poised when working through issues with software developers.

• Given the complexities of the test effort associated with a client-server or multitier environment, test engineers need a broad range of technical skills. They should possess experience across multiple platforms, operating systems, layers of supporting applications, interfaces to other products and systems, databases, and application languages. They also need to know the script programming language of the primary automated test tool.

• To recruit test engineers for a test program, the hiring manager should understand the target composition of the test team. The effective execution of a test program requires that a test team have enough resident expertise to leverage the adopted test process and the applied test tools. The composition of the test team should be outlined within a test team profile.

• In preparation of test engineer interviews, the hiring manager should develop a list of questions intended to confirm that the candidate has the appropriate expertise at the level of proficiency required. All individuals involved in the interview process should document or summarize each candidate’s responses to each question.

• The major roles and responsibilities of individuals who perform test activities on a project need to be defined and documented within the project test plan.

References

1. “Survey Provides Software Testing Industry Snapshot.” Software Testing Newsletter Fall/Winter 1995–1996.

2. Burnstein, I., Suwanassart, T., Carlson, C.R. Developing a Testing Maturity Model, Part II. Chicago: Illinois Institute of Technology, 1996.