Appendix C. Test Engineer Development
In part of their attempt to do more with less, organizations want to test their software adequately, but as quickly and thoroughly as possible. To accomplish this goal, they are increasingly turning to automated testing. The shift to automated testing has had a major effect on the software industry and test professionals interested in furthering their career in software testing would be well advised to take note. Changes in the software industry stemming from the growth of automated testing include a greater demand for test engineers who are also software professionals and the migration away from manual testing methods. Additionally, the job market is rich with entry-level test positions for new computer science and information systems college graduates.
In the past, the tasks of performing software testing and the effort to develop software were often viewed as totally different responsibilities requiring vastly different skill sets. Today’s test engineer, however, needs to be a software professional with programming skills and knowledge of networks, databases, and operating systems. The division of skills between software test engineers and software developers will continue to become less significant.
The evolution of automated test capabilities, together with recent emphasis on quality standards and software maturity guidelines, has given birth to new career opportunities for software engineers. Within the job market, there has been an explosion in the demand for automated software test professionals. Many software engineers are choosing careers in the automated test arena because of the different kinds of tasks involved and the variety of applications for which they are introduced. Additionally, experience with automated test tools can provide a career lift. It provides the software engineer with a broader set of skills and perhaps a competitive career development edge.
The test effort associated with today’s two-tier and multitier client-server and Web environments is complex, so test engineers need a broad range of technical skills. Test engineers and test teams, therefore, need experience across multiple platforms, multiple layers of supporting applications, interfaces to other products and systems, different types of databases, and application languages. In an automated test environment, the test engineer also needs to know the script programming language of the primary automated test tool.
Given the breadth and depth of skills required by the test engineer to perform in a test engineering role, what might a test engineer expect in terms of a career path? Table C.1 portrays a possible test career progression.
Table C.1. Test Career Progression
In the first several years, the person could take on either a test engineering or a programmer analyst role. With more experience and the development of additional skills, he or she could move into a team or test/programming lead position. Eventually, the individual might assume a management position.
As a test engineer, you might ask, “What should I do to improve my chances of moving up through these progressive steps?” You might also ask, “What might a career development program plan look like?” Table C.2 outlines a six-step test engineer career development program. This development program is aimed at identifying the different kinds of skills and activities at each stage or level on which test engineers should focus time and attention so as to improve their capabilities. If you are already performing in a management capacity, then Table C.2 serves as a guideline on how to approach training and development for your staff.
Table C.2. Career Development Program
C.1 Technical Skills Stage
The first career development stage focuses on the development concerns for an individual performing in an entry-level or a junior test engineering position. The focus of development during this stage is the improvement of technical skills. Test engineers will need to enhance their skills by becoming more familiar with test tools and test techniques, as well as with programming languages, operating systems, networks, and databases.
If new to the workforce, the test engineer will need to become familiar with the overall test engineering life cycle and will need to learn the business areas of applications under test. If the test engineer has several years of experience and is making a transition to a test career, then the person may have already developed this business area knowledge. Commonly, individuals performing in the areas of software development, system administration, network management, and software quality assurance rotate into the software test discipline. Often out of necessity, these individuals become responsible for software testing because of the variety of work involved in software testing, the opportunity to work with automated test tools, and the chance to apply programming skills during the performance of test script development.
To become more knowledgeable on test tools and test techniques, the test engineer can read periodicals and journals such as software development magazines, read books on software development and testing, and surf the Web for test information. Other avenues to learn more about testing and to obtain answers to questions include local test tool user groups, test and software conferences, and quality assurance or process improvement-related seminars and associations.
It is important to develop skills and understanding of automated test tools. Obtain evaluation copies of automated test tools and experiment with the tools to become familiar with how they work and what they can do. If your organization has a license for an automated software test tool but may not be using it on a project that you are supporting, then experiment with the test tool.
Test engineers can augment their technical knowledge by enrolling in technical training courses offered by universities, community colleges, and technical training centers. Training on test tools and test methodology is valuable. Some organizations that offer such training are listed in Table C.3. Also see http://www.autotestco.com/ for updates to this list.
Table C.3. Test Training Organizations
The test engineer’s employer may also offer training. The types of technical training courses that are helpful for the test engineer include those addressing software programming, network engineering, database modeling and development, operating systems, and other technical issues.
As a first priority, the test engineer should become more proficient in the use of any test tool being used on a current project or a tool planned to be used on an upcoming project. It is also worthwhile, where possible, to receive training on tools other than one with which the test engineer is already comfortable. This kind of training helps the test engineer develop the ability to look at test requirements and test engineering challenges from a multitude of different perspectives. With these different perspectives, the test engineer has more latitude to explore creative solutions to test program work.
C.2 Test Process Stage
The second career development stage focuses on understanding the software development and test life-cycle process. The test engineer must first become familiar with the organization’s overall test life-cycle process. The Automated Test Life-cycle Methodology (ATLM) can be adopted to help define an organization’s methodology for implementing and performing automated testing. The test engineer must also develop a comprehensive understanding of the different test strategies and the types of test tools that are available.
Beneath the organization’s overall test life-cycle process, the test team should develop lower-level defined processes (test procedures) for performing test activities. The organization’s process asset repository should include a standard (template) test plan, the top-level test life-cycle process, and the detailed test procedures, together with applicable development and test standards.
The overall software development process has been defined as “the set of activities, methods, practices and transformations that integrate managers and software engineers in using technology to develop and maintain software” [1]. Evidence suggests that improvement in the overall software development process to include the test life-cycle process provides increased productivity and product quality [2]. Sixty-nine percent of companies don’t have a measurable, repeatable software testing process. Seventy-four percent of companies say the best way to meet software quality demands is to “build a process that sets goals and [begins] testing in the beginning of the development life cycle” [3].
For test process assets to be meaningful, they must be used. For test process assets to be used, they must be effectively implemented. Implementation requires that processes not only be defined, but also be established, incorporated into the system, made the norm, and eventually adopted as a standard. Implementation also means that test engineers are trained on the test process assets and assume ownership of them.
One effective approach for implementing the test process is to initiate a process brief program. The process brief program represents a timely and efficient way to achieve process definition. It requires that test team personnel develop presentations for each defined process area. Each presentation follows a prescribed format, with the test engineer outlining essential process elements such as entry/exit criteria, inputs, process steps, process participants, relevant tools, and outputs. The test team realizes several goals by undertaking this approach. Test engineers performing presentations take ownership and learn the process, other test team members are trained on the defined process, and improvements on the process become possible because the process has been both defined and baselined.
The test team should first require that a process brief be performed on the overall test process for the organization. Within this top-level process definition, 5 to 10 primary test steps (procedures) would be defined. Test engineers would then be assigned responsibility for the defined test procedures, and process briefs would be constructed for each test procedure.
The process brief is designed to simplify process definition by having the test engineer prepare presentation slides that detail the primary ingredients of the standard process definition. That is, the process brief consists of simple text and diagrams. The time required to develop the presentation slides should be minimal.
The process brief serves as a vehicle to define a test process; it also represents a way to obtain management and project team consensus on the defined process. In addition, the conduct of the brief serves to train other test team personnel on the process. Table C.4 gives a suggested format for the process brief slide presentation. The structure for the process context diagram (slide 3 of the process brief) is depicted in Figure C.1.
Table C.4. Process Brief Format
Figure C.1. Process Context Diagram Format
The test process stage of test engineer career development should also focus on activities other than the knowledge of the test process and involvement in process briefs. The test engineer should participate in requirements, design, and code inspections, walkthroughs, and reviews. He or she should begin mentoring other, more junior engineers within the test team. Project and test managers need to avoid the temptation of steering junior test engineers toward test process and test tool documentation, when these less experienced personnel request help. Mentoring is more beneficial to the junior test engineer, and the test engineer providing the mentoring generally learns from the experience as well.
During the test process stage of development, the test engineer will need to take his or her test automation skills to the next level. The test engineer needs to view test automation activities as requiring their own, separate development effort, complete with strategy and goal planning, test requirements definition, analysis, design, and coding. He or she must learn to appreciate the fact that effective automated test development requires careful design and planning, just like any software application development effort.
The test engineer needs to develop test automation programming techniques so as to create test scripts that perform such tasks as testing of different data values, testing of a large number of different user interface characteristics, or volume testing. Automated test scripts need to be developed that invoke looping constructs to exercise a script repeatedly or that call conditional statements to exercise a statement only under a specific condition. In addition, test scripts need to be developed that take advantage of application programming interface (API) calls or use .dll files, use files, and libraries. Eventually, the test engineer needs to apply his or her evolving skills to help establish test script programming standards and frameworks.
C.3 Team Effort Stage
In the team effort stage of development, the test engineer needs to improve his or her verbal communication and writing skills. This person must develop the ability to effectively organize thoughts, concepts, and ideas within written correspondence, such as e-mail messages and reports, and during speaking engagements, such as meetings and presentations. Without these communication skills, the test engineer’s career progression may be needlessly delayed. The team lead, test lead, and test management positions all require that the individual be able to communicate assignments, conceptualize and discuss complex test strategies, and talk through difficult technical challenges. Strong communications skills are also necessary to be able to assure supervisors, through written status reports, that proper actions are being taken to mitigate test program risks and resolve technical problems.
The test engineer should inquire about communication skill development courses to improve his or her verbal and writing skills. It is also important to gain experience by giving formal presentations in front of people. Where possible, the test engineer should join clubs and organizations, such as Toastmasters (www.toastmaster.org), that offer opportunities for the test engineer to polish his or her public speaking skills.
In addition to developing communication skills, the test engineer should become involved in the exercise of sizing a test program. The test team member might initially volunteer to help develop a work breakdown structure for a new test effort. The work breakdown structure provides a way of identifying the various types of test activities that are expected to be performed on a test effort (see Section 5.2). The test engineer can also offer to help with timekeeping activities, which maintain a historical record of the effort expended to perform the various test activities on projects. The maintenance of historical records helps support the test effort sizing exercise, by allowing test team members to better judge the amount of effort that will be required for a new project. The test engineer should become familiar with the different sizing methods for determining the necessary size of the test team (see Section 5.3.1). Where possible, the test engineer should learn about and become involved in test task scheduling, tracking, and reporting.
The personal software process (PSP) is an education program that supports development of sizing and estimation skills together with task scheduling, tracking, and reporting skills. Test engineers should read Watts Humphrey’s A Discipline for Software Engineering, which explains the PSP program [4]. Test engineers can also obtain classroom and on-site training on the PSP through an organization called Advanced Information Services (AIS) [5] For more information on PSP training offered by AIS, see the Web site http://www.advinfo.net/.
Another way to continue to progress as a test professional during this stage is to become more involved in the test activities outlined in Section 5.3. The test engineer, for example, might become involved in new activities in which he or she has not yet gained experience, such as support for the test bed environment, development and maintenance of test data, and the maintenance of the test script reuse library.
In addition, the test engineer can further develop his or her skills in life-cycle support tools, such as test tools, requirements management tools, defect/issue tracking tools, and configuration management tools during this stage. It is important that the test engineer understand the application and capabilities of these tools. It is also beneficial for him or her to become familiar with ways to integrate and share data between these tools.
C.4 Technical Stewardship Stage
In the technical stewardship stage of development, the test engineer needs to begin performing management support activities. Management support activities include test engineer recruitment, development of relationships with test tool vendors, staff supervision, customer relations, and oversight of test tool evaluations and introductions. It is important during the technical stewardship stage that the test engineer take responsibility for developing test effort estimations as well as performing task scheduling, tracking, and reporting activities.
The test engineer must provide technical leadership on each test program. This leadership includes providing guidance to subordinate test engineers with regard to test planning, design, development, and execution activities. The test engineer must stay current with the latest test approaches and test tools and must be able to transfer this knowledge to the rest of the test team. He or she also needs to display leadership in the area of test process implementation and test process improvement. Test process leadership may involve reviewing test metrics, lessons learned, and the results of test program benefits surveys in a meeting setting with the rest of the test team. It could also be evidenced by the assignment of subordinate test engineers to the tasks of reviewing existing test procedures and developing new test process briefs that improve the test procedures.
To effectively demonstrate the necessary leadership skills, the test engineer might need to undergo personal effectiveness training. This training includes courses and seminars that improve the test engineer’s ability in the areas of leadership, team building, and public speaking. The goal for the test engineer is to deal with people more effectively, while also being more capable in overcoming tough technical challenges. Important skills in this stage include diplomacy, creativity, and conceptualization.
At this stage, the test engineer can begin to stand out as an exceptional software professional by making clear distinctions about problems and remaining focused on options and solutions. The ability to make clear distinctions stems from an improved ability to listen and understand what is being said, listen for how it is being said, and sometimes listen for what is not being said. The test engineer needs to be very honest when dealing with fellow project personnel—but not brutally honest. “Being honest” implies that the test engineer can ask questions and state observations in a constructive, nonthreatening, and respectful way.
Staying current with the latest test approaches and test tools may require that the test engineer attend software, quality, and test conferences. Where possible, he or she can practice public speaking skills by offering to make presentations at these conferences. During the development of a presentation topic and the preparation of presentation slides, the test engineer stands to become more knowledgeable on the particular topic. During the actual presentation and discussions following the presentation, this individual will learn even more.
C.5 Test/Project Management Stage
During this stage, the test engineer needs to evolve from a technical leadership role to one in which he or she becomes responsible for a test program and perhaps even a complete development project. In a test management role, the test engineer must ensure that each project test plan is complete and accurate. (See Chapter 6 for more information on test plan development.) The test manager is also responsible for enforcing the cohesive integration of test and development activities.
The test manager needs to ensure that the acquisition of required hardware and software is performed correctly and within the necessary timeframe, that the test environment is developed and maintained properly, that test product configuration management is performed, and that the test process is defined and maintained. Likewise, he or she must ensure that the test program remains on schedule and, in doing so, obtain accurate measures of test progress.
Given the increased number of personnel being supervised, the test manager needs to be concerned about employee motivation. It is a good practice to acknowledge good performance, publicly where possible. The test manager should also use a bonus program to reward personnel who put in extra effort to ensure the success of a test program and promote the credibility of the test team.
As the manager and leader for the test team, this individual is responsible for cultivating a good reputation for the entire test team. The test team must be viewed as an organization that performs professionally and provides value to each project. By maintaining its good reputation, the test team will be able to influence business decisions made by senior management, such as whether to purchase a new test tool. The test manager must also be careful to protect test team members from unwarranted or harsh criticism. Issues and criticisms should go through the manager rather than having the particular test engineer be approached in person.
Because the test manager may be responsible for directing and juggling test efforts across more than one project, he or she must be able to rely upon skilled and capable test leads to assist on each project. The test manager, therefore, needs to make good hiring decisions and then motivate and further develop each test lead. The ability to delegate is an important skill that involves the establishment of an honest and trusting relationship with each test lead. In fact, having a competent staff to perform the various test program activities is essential.
A primary concern for the test manager involves the need to maintain an overall level of test capability. Thus the test manager must identify potential single points of failure and take corrective action. A single point of failure involves a situation where only one member of the test organization has a particular test tool or technology skill. When this skill is vital to the credibility of the test team and the ability to successfully perform on projects, then the loss of the individual represents a significant risk.
To develop a clear picture of the different kinds of test tool and technology skills that reside within the test team, the test manager should develop a test team capability profile. Such a profile lists the name of each member of the test team together with the skills in which he or she has proficiency. Somewhere on the profile sheet, the test manager needs to list the top 10 or so most critical test tool and technology skills required of the test team. Where deficiencies exist, the test manager should implement mentoring relationships, on-the-job training, and formal training mechanisms to bolster the team’s capability. It is worthwhile to ensure that the test team is at least two deep in the most important test tool and technology skill areas.
If the test manager is to move up further within the organization, an advanced college degree may be necessary. The manager should investigate the post-graduate education programs available from colleges and universities in the vicinity. He or she is in a position where close coordination and dealings with senior management and possibly with a client or customer are required. In this environment, an understanding of office politics becomes crucial to job security and career advancement.
One significant aspect of office politics is the notion that the test manager may be technically correct on a subject, but present the information to senior management in an ineffective way. The test manager needs to be sensitive to the political environment and savvy enough to remain focused on the desired end result. It is important to respect others and allow them to hold different opinions. Your opinion may be correct given all the knowledge and information that you have at your disposal. Another person’s opinion may be correct given all the knowledge and information that he or she has. The test manager may lose a few battles, but must understand that the cherished end result is to win the war.
C.6 Business/Product Management Stage
In the next stage of development, the test engineer evolves out of a test management role and into being concerned with the business health and well-being of a part of the organization or for a product line. In this environment, the manager must begin to shift his or her focus from concerns within the organization and to activities and concerns involving people and organizations outside of the organization. Business opportunities must now be identified and partnerships with outside organizations cultivated and reaped.
As a business or product manager, the individual must be able to develop a vision for the future of the business or product line and to obtain and organize resources necessary to support the vision. He or she must now focus less on how to get the job done, but instead be more concerned about identifying the kind of job that needs to get done. The individual now needs to be surrounded by people who can understand the kinds of jobs that need to be done and who know what is necessary to complete each job.
References
1. Curtis, B. “Software Process Improvement: Methods for Modeling, Measuring and Managing the Software Process.” Tutorial notes, 15th International Conference on Software Engineering, May 17–21, 1993.
2. Curtis, B., Kellner, M., Over, J. “Process Modeling.” Communications of the ACM, 1992; 35:75–91.
3. The 1996 CenterLine Survey.
4. Humphrey, W. A Discipline for Software Engineering, Reading, MA: Addison Wesley, 1995.