It has been said, “If you do not know where you are going, all roads lead there.” Traditionally, many IT organizations annually develop a list of improvements to incorporate into their operations without establishing a goal. Using this approach, the IT organization can declare “victory” any time it wants.
This chapter will help you understand the importance of following a well-defined process for becoming a world-class software testing organization. This chapter will help you define your strengths and deficiencies, your staff competencies and deficiencies, and areas of user dissatisfaction.
The objective of this chapter is threefold: to define a world-class software testing model, to provide a self-assessment process for your software testing organization to measure yourself against the world-class model, and to provide some planning considerations for moving to a world-class level.
The roadmap to become a world-class software testing organization is a simple three-step process, as follows:
Define or adopt a world-class software testing model.
Determine your organization’s current level of software testing capabilities, competencies, and user satisfaction.
Develop and implement a plan to upgrade from your current capabilities, competencies, and user satisfaction to those in the world-class software testing model.
This three-step process requires you to compare your current capabilities, competencies, and user satisfaction against those of the world-class software testing model. This assessment will enable you to develop a baseline of your organization’s performance. The plan that you develop will, over time, move that baseline from its current level of performance to a world-class level. Understanding the model for a world-class software testing organization and then comparing your organization will provide you with a plan for using the remainder of the material in this book.
Software testing is an integral part of the software-development process, which comprises the following four components (see Figure 1-1):
Plan (P): Devise a plan. Define your objective and determine the strategy and supporting methods to achieve it. You should base the plan on an assessment of your current situation, and the strategy should clearly focus on the strategic initiatives/key units that will drive your improvement plan.
Do (D): Execute the plan. Create the conditions and perform the necessary training to execute the plan. Make sure everyone thoroughly understands the objectives and the plan. Teach workers the procedures and skills they need to fulfill the plan and thoroughly understand the job. Then perform the work according to these procedures.
Check (C): Check the results. Check to determine whether work is progressing according to the plan and whether the expected results are being obtained. Check for performance of the set procedures, changes in conditions, or abnormalities that may appear. As often as possible, compare the results of the work with the objectives.
Act (A): Take the necessary action. If your checkup reveals that the work is not being performed according to the plan or that results are not what you anticipated, devise measures to take appropriate actions.
Testing involves only the “check” component of the plan-do-check-act (PDCA) cycle. The software development team is responsible for the three remaining components. The development team plans the project and builds the software (the “do” component); the testers check to determine that the software meets the needs of the customers and users. If it does not, the testers report defects to the development team. It is the development team that makes the determination as to whether the uncovered defects are to be corrected.
The role of testing is to fulfill the check responsibilities assigned to the testers; it is not to determine whether software can be placed into production. That is the responsibility of the customers, users, and development team.
There is no generally accepted model for a world-class software testing organization. However, analyzing the best testing organizations among the more than 1,000 IT organizations affiliated with the Quality Assurance Institute (QAI) enabled QAI to identify the attributes of the best software testing organizations (see Figure 1-2). Organizations that follow this model report more effective and efficient testing than those that do not.
The world-class software testing model includes
Test environment. The conditions that management has put into place that both enable and constrain how testing is performed. The test environment includes management support, resources, work processes, tools, motivation, and so forth.
Process to test a single software project. The standards and procedures testers use to test.
Tester competency. The skill sets needed to test software in a test environment. The three self-assessments that follow are for the above three attributes of a world-class software testing organization.
Note
The three self-assessments in this chapter correspond to the preceding three attributes of a world-class software testing organization.
The world-class model of a software testing organization focuses on stakeholder satisfaction. This assumes a greater role for a world-class software testing organization than just testing against documented software requirements. Chapter 2 defines the many roles that software testing can adopt; however, those roles include much more than testing documented software requirements. They include testing for quality factors such as ease of use, meeting testing schedules and budgets, and minimizing the risks involved with any software project.
According to the world-class model, the following parties have a vested interest in software testing:
Software customer. The party or department that contracts for the software to be developed.
Software user. The individual or group that will use the software once it is placed into production. (Note: This may be the customer or it may be parties other than the customer.)
Software developer. The individual or group that receives the requirements from the software user or assists in writing them, designing, building, and maintaining the software, as needed.
Development tester. The individual or group that performs the test function within the software development group.
IT management. The individual or group with responsibility for fulfilling the information technology mission. Testing supports fulfilling that mission.
Senior management. The CEO of the organization and other senior executives who are responsible for fulfilling the organization mission. Information technology is an activity that supports fulfilling that mission.
Auditor. The individual or group responsible for evaluating the effectiveness, efficiency, and adequacy of controls in the information technology area. Testing is considered a control by the audit function.
Project manager. The individual responsible for managing the building, maintaining, and/or implementing of software.
The test mission, strategy, and environment must be focused on stakeholder satisfaction. The mission defines the testing objectives; the strategy defines how the mission will be accomplished; and the environment provides the culture, processes, and tools that are conducive to effective and efficient software testing.
The test processes are those step-by-step procedures that the testers will follow to accomplish their assigned tasks. Test processes executed by trained and competent testers enable those testers to accomplish the defined test mission.
The test processes need to be improved continually for two reasons: to make them more effective and efficient to use, and to incorporate updated approaches into testing new technologies and software development methodologies.
The responsibility for ensuring that the execution of the test processes meets the defined test mission lies with management. Management must ensure that testers are following and can accomplish the test plan, and that the plan will, in fact, accomplish the test objectives. If not, management should modify the plan to meet those objectives.
Management and testers need tools to enable them to fulfill their responsibilities. Two very important tools are the testing strategic dashboard and the testing tactical dashboard. The testing strategic dashboard includes key indicators such as user satisfaction, staff competency, and the percent of tests completed. The testing tactical dashboard includes test indicators such as the number of requirements tested and percent correct, defects uncovered, defects corrected and uncorrected, and the schedule and budget status.
Management must ensure that if you meet the testing tactical key indicators, you will, in fact, meet the objectives defined by the strategic key indicators.
You can customize the world-class model for software testing by defining the attributes of each of its components (refer to Figure 1-2). The material in this book explains the attributes of all the components: stakeholder satisfaction, test mission, test management and enabling competencies are discussed in Part 2. The test processes are explained in Parts 3 and 4. Test process improvement is described in Part 5 of this book.
As you read those parts of the book, you can customize those attributes based on the mission of your organization. For example, in describing a tester’s competency, skill sets for testing COTS software and outsourced software will be listed. However, if your organization does not use COTS software or does not outsource the development of software, you would not need those skills in your testing staff. Likewise, if your testers are not responsible for testing security, you would not need a test processes for testing security.
The three self-assessments included in this chapter are based on the model in Figure 1-2. However, it is recognized that few testing organizations need all these testing capabilities and competencies. Therefore, you need to develop the model that is suited to your test mission.
This section presents the following three self-assessment categories to enable you to compare your testing organization against the world-class model:
Assessing the test environment. This includes user satisfaction, management support, environment, planning, tools, test processes, measurement, quality control, and training.
Assessing the process for testing individual software projects. This category of assessment will assess your testing process against the seven-step process for testing individual software projects presented in Part 3 of this book.
Assessing the competencies of software testers. This self-assessment will be based on the 2006 Common Body of Knowledge (CBOK) developed by the Certification Board of the Software Certifications Organization. Each of the recommended ten competencies for software tester will be assessed. A more detailed assessment to be used in individuals to compare their specific test competencies against the 2006 CBOK is provided in Chapter 5.
During the past 25 years, the Quality Assurance Institute (QAI) has been studying what makes software testing organizations successful. As a result, QAI has identified the following eight criteria:
Test environment planning
Management support
Use of test processes
Test tools
Quality control
Test measurement
User satisfaction
Test training
When these eight criteria are in place and working, the result is normally a world-class testing organization.
The assessment process developed by QAI has five items to address within each of the eight criteria. The more of those items that are in place and working, the more likely that criteria will contribute to world-class testing. Figure 1-3 shows a cause-effect diagram indicating the areas to address, called drivers, which results in a world-class testing organization.
Software testing organizations can use the results of this assessment in any one of the following three ways:
To determine their current testing environmental status versus the environment of a world-class testing organization. The responses to the items address will indicate an organization’s strengths and weaknesses compared to the environment of a world-class testing organization.
To develop the goal/objectives to accomplish becoming a world-class testing organization. QAI’s world-class criteria indicate a profile of the environment of a world-class testing organization. Achieving those objectives can lead you to become a more effective software testing organization.
To develop an improvement plan.
By doing the assessment, you will develop a Footprint Chart that shows where improvement is needed. Those criteria in which you are deficient become the means for improving the environment of your software testing organization.
This practice involves the following four tasks:
Build the assessment team.
Complete the assessment questionnaires.
Build the footprint chart.
Assess the results.
The assessment team should combine people who in totality possess the knowledge of how your organization manages software testing. Before the team is established, the areas to address should be reviewed to determine the makeup of the team. It is recommended that a matrix be prepared with the seven assessment criteria on one dimension and the recommended assessment team on the other. The matrix should indicate which assessment team member is knowledgeable about each of the seven assessment criteria.
Once all seven criteria have been associated with an assessment team member, it can be concluded that the team is adequate to perform the assessment.
The assessment questionnaire in Work Paper 1-1 consists of eight categories, with five items to address for each category. A Yes or No response should be made, as follows:
A Yes response means all of the following:
Criteria items are documented and in place.
Criteria items are understood by testers.
Criteria items are widely used, where applicable.
Criteria items have produced some possible results.
A No response means any of the following:
No formal item in place.
Criteria items are applied differently for different test situations.
No consistency as to when used or used very seldom.
No tangible results were produced.
Table 1-1. Self-Assessment on Software Testing Environment
ITEMS TO ADDRESS ON TEST ENVIRONMENT PLANNING | YES | NO | COMMENTS | |
|---|---|---|---|---|
1. | Does your IT organization have a policy on software testing? | |||
2. | Does your software testing organization have a test strategy? | |||
3. | Does your software testing organization have software processes and tools to support that testing strategy? | |||
4. | Does your software testing approach include both verification and validation testing (i.e., testing the software in both a static and executable mode)? | |||
5. | Does your testing strategy address the various roles that testing can assume, and determine which of those roles will be incorporated into your organization’s testing strategy (e.g., testing user needs in addition to software specifications)? | |||
ITEMS TO ADDRESS ON MANAGEMENT SUPPORT | YES | NO | COMMENTS | |
1. | Does management provide the resources necessary (including calendar time) to adequately train, plan, conduct, and evaluate results for software testing assignments? | |||
2. | Are testers involved from the inception through termination of software projects to ensure that testing concerns are continuously addressed? | |||
3. | Does management allocate as many resources to the test processes and tools as it does to the development process and tools? | |||
4. | Does management spend as much time on test planning and test execution as it does on development planning and development execution? | |||
5. | Is management knowledgeable and sufficiently trained in test theory, processes, and tools to effectively manage test planning and execution, and understand and effectively act on test results? |
| ||
ITEMS TO ADDRESS ON THE USE OF TEST PROCESSES | YES | NO | COMMENTS | |
1. | Do testers follow processes to plan tests, prepare test data, execute tests, and develop and report test results? | |||
2. | Can testers correctly interpret documented test processes so that the test procedures can be followed as intended? | |||
3. | Do the processes provided for testing cover all the activities that are needed to perform effective testing? | |||
4. | Has a plan been developed and put in place to mature the test processes so that they become more effective and efficient and are performed on time? | |||
5. | Do the owners/users of the test processes (the testers) build the processes used for testing? | |||
ITEMS TO ADDRESS ON TEST TOOLS | YES | NO | COMMENTS | |
1. | Do testers use an automated tool to generate and reuse test data? | |||
2. | Are test tools selected in a logical manner? | |||
3. | Can testers use test tools only after they have received adequate training in how to use them? | |||
4. | Is test tool usage specified in the test plan? | |||
5. | Has a process for obtaining assistance in using test tools been established, and does it provide testers with the needed instructional information? | |||
ITEMS TO ADDRESS ON TEST TRAINING | YES | NO | COMMENTS | |
1. | Does a career training plan for testers exist, and is it in use to develop a tester from an unskilled state to a master tester state? | |||
2. | Are testers adequately trained in test processes before using those processes for testing? | |||
3. | Are testers trained in the theory of testing, risk analysis, the various approaches to testing, and so forth so that they understand “why” they perform certain test tasks? | |||
4. | Are testers trained in statistics so that they understand the level of confidence they can provide a user by different test approaches and how to interpret test results? | |||
5. | Are testers trained in how to measure process performance, and do they use the results of that measurement to improve the test processes? | |||
ITEMS TO ADDRESS ON USER SATISFACTION | YES | NO | COMMENTS | |
1. | Do users get the information they need to track test progress and assess results prior to placing software into production? | |||
2. | Are surveys conducted to determine user satisfaction with test planning, test execution, test results, communications, and so forth? | |||
3. | Do users participate in tests that determine whether the software is acceptable for use? | |||
4. | Are users presented with a plan for testing, and do they “approve” (i.e., agree) that if that plan is followed, they will consider testing to be satisfactory? | |||
5. | Are the user support activities (such as data entry, output usage, terminal usage, manual usage, and so forth) validated as part of testing? |
| ||
ITEMS TO ADDRESS TO TEST MEASUREMENT | YES | NO | COMMENTS | |
1. | Does a set of test measures and metrics exist, and are they used to measure the efficiency and effectiveness of software testing? | |||
2. | Has a measurement process been installed to measure the efficiency of the test processes? | |||
3. | Is compliance to the budget and schedule measured and variances addressed effectively? | |||
4. | Is tool usage measured to assess the contribution received from automated testing? | |||
5. | Is the percentage of defects removed versus the total defects eventually attributable to a development phase measured? | |||
ITEMS TO ADDRESS TO TEST QUALITY CONTROL | YES | NO | COMMENTS | |
1. | Are defects made by testers during testing recorded and effectively addressed? | |||
2. | Is the test plan reviewed/inspected during/after completion by peers for adequacy and compliance to test standards? | |||
3. | Does the test plan include the procedures that will be used to verify that the plan is executed in accordance with the plan? | |||
4. | Are regular reports prepared that show the full status of testing individual software systems? | |||
5. | Are the individual quality control reports periodically summarized to show the efficiency and effectiveness of testing in the entire information services organization? |
The assessment team should read aloud each item and then discuss how that item is addressed in their testing environment. The results should be recorded on Work Paper 1-1. The assessment team may also wish to record comments that clarify the response and/or to provide insight in how that area may be improved.
For this task, you should transcribe the results of Work Paper 1-1 onto Work Paper 1-2. To do so, total the number of Yes responses for each criterion. Then place a dot on Work Paper 1-2 on the line representing the number of Yes responses. For example, if you have three Yes responses for test training, you should place a dot on the test training line at the intersection of the line representing three Yes responses. A dot should be marked on the line representing all seven criteria for the number of Yes responses. Then connect the dots with a line, resulting in what is called a “footprint” of the status of your testing environment versus the environment of a world-class testing organization.
You should make the following two assessments regarding the footprint developed on the Work Paper 1-2:
Assess the status of each criteria versus what that criteria should be in the world-class testing environment. To do this, you need to look at the number of Yes responses you have recorded for each criterion versus a world-class organization, which would have five Yes responses. For example, three Yes responses for test training would indicate that improvements could be made in your test training process. The two items that received No responses are indications of where improvements are needed to move your test training activities to a world-class level.
Interpret your testing environment footprint chart. The footprint in your Work Paper 1-2 provides an overview of your testing environment. Given the footprint, your assessment team should attempt to draw some conclusions about your testing environment. Three examples are given to help in drawing these conclusions, as shown in Figures 1-4, 1-5, and 1-6.
The following list of questions, if responded to positively, would indicate that the assessment has been performed correctly:
To assess the capabilities of your existing test processes, follow the same procedure that you used to assess your test environment. Note that you should use the same team for both assessments. The only change you will need is to substitute self-assessment questionnaires in assessing the test environment process with the self-assessment questionnaires for assessing the test processes included in this section.
The assessment of test processes will be divided into the following seven categories:
Preparing for a software testing project
Conducting test planning
Executing the test plan
Conducting acceptance testing
Analyzing test results and preparing reports
Testing the installation of software
Post-test analysis
Note that these seven categories of test processes correspond to a seven-step software testing process presented in Part 3 of this book. Thus, each assessment will help you determine your strengths and weaknesses in each of the seven steps of the proposed software testing process.
To conduct this self-assessment, answer the questionnaire in Work Paper 1-3 and post your results to Work Paper 1-4, as described in the preceding section.
Table 1-3. Self-Assessment on Test Processes
ITEMS TO ADDRESS ON PREPARING FOR A SOFTWARE TESTING PROJECT | YES | NO | COMMENTS | |
|---|---|---|---|---|
1. | Have the objectives and requirements for this software system being developed been defined? | |||
2. | Are the requirements testable? | |||
3. | Have adequate time and resources been allotted for both development and testing? | |||
4. | Has the process to be used for testing software been defined? | |||
5. | Are the testers familiar with the methodology that will be used to develop the software? | |||
ITEMS TO ADDRESS ON TEST PLANNING | YES | NO | COMMENTS | |
1. | Have the risks associated with the software been defined? | |||
2. | Have the test objectives been defined? | |||
3. | Do the testers have a well-structured process to follow to develop the test plan? | |||
4. | Have the constraints that will be imposed on testing been defined? | |||
5. | Does the test plan include a matrix that relates the test objectives to the tests that will be conducted? | |||
ITEMS TO ADDRESS ON TEST EXECUTION | YES | NO | COMMENTS | |
1. | Is there a process to follow to design test data? | |||
2. | Will verification testing be performed during the requirements phase of development? | |||
3. | Will verification testing be performed during the design and build phases of development? | |||
4. | Is a process in place to record and track defects? | |||
5. | Will test execution be performed in accordance with a plan included in the test plan? | |||
ITEMS TO ADDRESS ON ACCEPTANCE TESTING | YES | NO | COMMENTS | |
1. | Have the users defined acceptance criteria? | |||
2. | Do the users have a planning process to follow in developing an acceptance test plan? | |||
3. | Do the users have the competencies needed to conduct acceptance testing? (Note that the competencies may include professional software testers involved in acceptance testing) | |||
4. | Will acceptance testing simulate real-world processing conditions? | |||
5. | Prior to acceptance testing, has the user determined the actions that will be taken based on the software meeting or not meeting the acceptance test criteria? | |||
ITEMS TO ADDRESS ON TEST ANALYSIS AND REPORTING | YES | NO | COMMENTS | |
1. | Will test reporting be tied to the testing plan as defined in the test plan? | |||
2. | Will test reporting follow the test plan’s reporting standards? | |||
3. | Will both interim and final test reports be issued? | |||
4. | Will reporting report back on status of the function/test matrix included in the test plan? | |||
5. | Will the test report include an analysis and recommendation by the software test team? | |||
ITEMS TO ADDRESS ON TESTINGSOFTWARE INSTALLATION | YES | NO | COMMENTS | |
1. | Does a software configuration plan exist and is that plan effective and operational? | |||
2. | Does version control exist as part of the software configuration management plan? | |||
3. | Does the installation plan include the appropriate training and use of personnel? | |||
4. | Have all the interfaces to other software systems been identified and addressed in the installation process? | |||
5. | Will the installed software be tested to ensure its correct prior to moving to an operational status? | |||
ITEMS TO ADDRESS ON POST-TEST ANALYSIS | YES | NO | COMMENTS | |
1. | Will an analysis of the testing process be conducted after the software is placed into an operational status? | |||
2. | Will that analysis include the operational results of the software? | |||
3. | Will that analysis identify good and bad testing practices? | |||
4. | Does that analysis include a set-up matrix that will be used to quantitatively assess the effectiveness of testing? | |||
5. | Is there a process to incorporate the results of a post-test analysis into a process to improve the software testing process? |
This practice will enable you to assess your testing competencies against the ten skill categories in the Common Body of Knowledge (CBOK) for the Certified Software Tester (CSTE) certificate. At the conclusion of the assessment, you will develop a Footprint Chart that shows your competencies against the skill categories needed to become a CSTE. You can use the results to design a program for improving your personal test competencies.
Figure 1-7 shows a cause-effect diagram indicating the areas of competency assessment. In the diagram these are called the drivers that result in becoming a fully competent software tester. The drivers are, in fact, the ten CBOK skill categories.
This practice involves performing the following four tasks:
Understand the CSTE CBOK.
Complete the assessment questionnaires.
Build the footprint chart.
Assess the results.
Before you can effectively evaluate your software test competencies, you need to understand the 2006 CSTE CBOK. The final version of the 2006 CSTE CBOK is available through the Software Certification Organization. The discussion draft version of the 2006 CSTE CBOK is included in Chapter 5 as a detailed skill-assessment questionnaire. This step requires you to read through the CBOK and to obtain clarifications of the material as necessary. The best source for these clarifications is the CSTE CBOK study guide, which is available from the Quality Assurance Institute (www.QAIworldwide.org).
The assessment questionnaire in Work Paper 1-5 contains ten knowledge categories with 5 items in each category, for a total of 50 items to assess. For each item, a Yes or No response should be made. The meanings of the Yes and No responses are as follows:
A Yes response means all of the following:
You have had formal training, experience, or self-study supporting this skill item.
You have actively used the skill in your personal or work life.
You have accomplished some positive result using this skill item.
A No response means any of the following:
You do not understand the theory and concepts supporting the skill item.
You have never used the skill item in a personal or work situation.
You have used the skill item but you have never achieved any positive results.
Table 1-5. Self-Assessment on Tester Competency
ITEMS TO ADDRESS FOR SOFTWARE TESTING PRINCIPLES AND CONCEPTS | YES | NO | COMMENTS | |
|---|---|---|---|---|
1. | Are you familiar with the technical terms used to describe various testing techniques, tools, principles, concepts and activities? | |||
2. | Do you have knowledge of the different levels of testing, such as unit testing? | |||
3. | Do you have an understanding of the multiple roles of software testers, including testing against specifications and testing to meet users’ needs? | |||
4. | Do you understand the “V” concept of testing? | |||
5. | Do you understand the tester’s workbench, meaning that you understand the process by which the testing task is performed? | |||
ITEMS TO ADDRESS FOR BUILDING THE TEST ENVIRONMENT | YES | NO | COMMENTS | |
1. | Do you understand the concepts of policies, standards and procedures and their integration into test processes? | |||
2. | Do you understand how to select processes for performing the test activities? | |||
3. | Do you understand how to adapt a test environment to different software development methodologies? | |||
4. | Do you understand a process for acquiring and deploying test tools? | |||
5. | Do you understand what management must do in order to create a work environment in which testers are motivated to do the right thing in an efficient and effective manner? | |||
ITEM TO ADDRESS FOR MANAGING THE TEST PROJECT | YES | NO | COMMENTS | |
1. | Do you possess the necessary communication skills to effectively manage a test project? | |||
2. | Do you possess the personal effectiveness skills, such as negotiation, to effectively manage the test project? | |||
3. | Do you have the test administration skills, such as budgeting and scheduling, to effectively administer the test project? | |||
4. | Do you have the skills to ensure that the test plan and processes used in the project will be in line with the organizational goals, user business objectives, release cycles, and different development for methodologies? | |||
5. | Do you have the skills needed to develop working relationships with users and other stakeholders in the testing process? | |||
ITEMS TO ADDRESS FOR TEST PLANNING | YES | NO | COMMENTS | |
1. | Do you understand the methods for performing risk analysis? | |||
2. | Do you know how to estimate the magnitude of risks? | |||
3. | Do you know how to develop a test plan that meets industry test plan standards? | |||
4. | Are you competent in software configuration management, change management, and version control? | |||
5. | Can you develop test objectives and acceptance criteria for a project being tested? | |||
ITEMS TO ADDRESS FOR EXECUTING THE TEST PLAN | YES | NO | COMMENTS | |
1. | Do you have the skills necessary to design test data and test scripts? | |||
2. | Can you develop a test cycle strategy that will determine the number of test cycles to be conducted and what type of testing will occur during these cycles? | |||
3. | Do you know the type of information that must be recorded to effectively document test results? | |||
4. | Do you understand the process that testers should follow in recording and monitoring the resolution of defects? | |||
5. | Do you understand what is necessary to test changes introduced to software testing after you have started testing? | |||
ITEMS TO ADDRESS FOR TEST ANALYSIS AND REPORTING | YES | NO | COMMENTS | |
1. | Do you understand the difference between a measure and a metric? | |||
2. | Do you know how to report results of testing that is consistent with the IT industry test reporting standards? | |||
3. | Are you familiar with, and can you calculate the more common metrics used in testing, such as defect removal efficiency? | |||
4. | Do you know the type of information that must be gathered during testing to enable test reports to provide the information projects need to assess their readiness to be placed into operation, such as code coverage and requirements coverage? | |||
5. | Do you have a knowledge of the tools needed to develop effective test reports, such as statistical analytical tools? | |||
ITEMS TO ADDRESS FOR USER ACCEPTANCE TESTING | YES | NO | COMMENTS | |
1. | Do you understand the differences between the system test and acceptance test? | |||
2. | Can you create “use case” test conditions? | |||
3. | Do you understand that the user’s role and the software tester’s role in acceptance testing? | |||
4. | Can you develop, in conjunction with users, an acceptance test plan that is consistent with the industry standards for acceptance test plan? | |||
5. | Do you know how to develop user acceptance criteria that are verifiable? | |||
ITEMS TO ADDRESS FOR TESTING SOFTWARE DEVELOPED BY OUTSIDE ORGANIZATIONS | YES | NO | COMMENTS | |
1. | Do you know the difference between software developed in-house and software developed by outside organizations? | |||
2. | Are you familiar with the process that would enable you to effectively test commercial off-the-shelf (COTS) software? | |||
3. | Are you knowledgeable in a process that would enable you to assess the software testing capabilities of an outside organization being considered for outsourcing? | |||
4. | Are you knowledgeable in the process that would enable you to test new versions of software acquired from outside organizations? | |||
5. | Do you know the risks/concerns associated with acquiring COTS software? | |||
ITEMS TO ADDRESS FOR TESTING SOFTWARE CONTROLS AND THE ADEQUACY OF SECURITY PROCEDURES | YES | NO | COMMENTS | |
1. | Are you knowledgeable in the vocabulary of internal control and security? | |||
2. | Are you knowledgeable in the industry-accepted model for internal control? | |||
3. | Are you knowledgeable in how to test systems of internal control in software business applications? | |||
4. | Do you understand the relationship between risk and control? | |||
5. | Are you knowledgeable in how to test the adequacy of security in a business application software system? | |||
ITEMS TO ADDRESS FOR TESTING NEW TECHNOLOGIES | YES | NO | COMMENTS | |
1. | Do you understand how to test new application architecture? | |||
2. | Do you know how to test new application business models? | |||
3. | Do you know how to test new communication methods? | |||
4. | Do you know how to test new hardware technologies? | |||
5. | Do you know how to evaluate the effective integration of new technologies into an organization’s IT policies and procedures? |
Prior to answering each question, you should think through the meaning of the question. This may require referring back to the CSTE study guide. Using the Yes/No response criteria, you need to come to a consensus on whether a Yes/No response should be indicated for the skill item. The result of your assessment should be recorded on the appropriate questionnaire.
You need to progress sequentially through the self-assessment questionnaires. Note that you may wish to make notes on the questionnaire to clarify your response or to indicate ideas on how you could improve your competency in that skill item.
To build the footprint chart, transcribe the results of Work Paper 1-5 onto Work Paper 1-6. To do so, total the number of Yes responses for each of the ten knowledge categories. Then place a dot on Work Paper 1-6 on the lines corresponding to the knowledge category. For example, if you have three Yes responses for the Test Planning category, you should place a dot on the Test Planning line at the intersection of the line representing the three Yes responses. After you have placed all ten dots, draw a line to connect them. This line, called a footprint, represents the status of your testing competencies versus those specified in the CSTE CBOK.
You should make the following two assessments regarding the footprint you developed on Work Paper 1-6:
Compare your results for each knowledge category versus what the knowledge category should be as indicated in the CSTE CBOK. Any rating less than five Yes responses indicates a potential area of improvement in that knowledge category. An analysis of the CBOK knowledge categories will be helpful in determining where to focus improvement, as will studying the CSTE guide to identify areas for potential improvement.
Compare your testing competencies against your current job responsibilities. The footprint provides an overview of your current competencies. Using your current job description, develop another footprint, which you believe is needed to achieve your current job responsibilities. Any deficiencies should be your first objective for improvement; your second for improvement would be to achieve the skill competencies needed to become a CSTE.
A positive response to the following questions indicates that you have correctly performed the competency assessment: (Note: Any negative response to the following five questions would reduce the value in using this self-assessment to measure an individual tester’s competency.)
The objective of the action plan is to move software testing from where it is (the baseline) to where it should be (the goal). There is no one way to develop this plan. Some organizations want to implement the plan so it is on a “pay as you go basis.” Other organizations are willing to invest in developing a significantly improved test process knowing that the payback will come after the process is developed and deployed.
The practices outlined in this book correspond to the three self-assessment footprints. If your organization is deficient in one or more components of the footprints, refer to the related chapter in this book that will help you develop your improvement plan, as shown in the following table:
ASSESSMENT NUMBER | ASSESSMENT CRITERIA | CHAPTER | ||
|---|---|---|---|---|
1 | Test environment assessment: | |||
Test Environment Planning | ||||
Management Support | ||||
User Satisfaction | ||||
Use of Process | ||||
Test Tools | ||||
Test Training | ||||
Test Measurements | ||||
Test Quality Control | ||||
2 | Test Process Assessment: | |||
Preparing for a Software testing Project | ||||
Test Planning General | ||||
Planning for specialized areas: | ||||
The Impact of Software Developmental Methodology When Testing | ||||
Testing Client/Server Systems | ||||
Testing Rapid Application Development | ||||
Testing the Adequacy of Internal Control | ||||
Testing Off-the-Shelf Software | ||||
Testing in a Multi-Platform Environment | ||||
Testing Security | ||||
Testing a Data Warehouse | ||||
Testing Web-based Systems | ||||
Test Execution | ||||
Acceptance Testing | ||||
Test Analysis and Reporting | ||||
Testing Software Installation | ||||
Post Test Analysis | ||||
Improving the test processes | ||||
3 | CSTE Knowledge Category | |||
1 Software Testing Principles and Concepts | ||||
2 Building the Test Environment | ||||
3 Managing the Test Project | ||||
4 Test Planning | ||||
5 Executing the Test Plan | ||||
6 Test Analysis and Reporting | ||||
7 User Acceptance Testing | ||||
8 Testing Software Developed by Outside Organizations | ||||
9 Testing Software Controls and the Adequacy of Security Procedures | ||||
10 Testing New Technologies | ||||
This chapter described how to assess your software testing processes and the competency of your testers. The chapter also briefly addressed specialized testing responsibilities and the need to improve your testing process and/or make your testing process more agile.
The chapter was built around a simple, three-step improvement process: determining your desired software testing performance, measuring your current performance against that performance goal, and developing a plan to move from where you are to a world-class testing organization.