Knowledge Category 1: Software Testing Principles and Concepts The “basics” of software testing are represented by the vocabulary of testing, testing approaches, methods, and techniques, as well as the materials used by testers in performing their test activities.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Testing Techniques

Understanding the various approaches used in testing, including static (e.g., desk checking), white-box (logic-driven), black-box (requirements-driven), load testing, coverage testing, and regression testing. Also included are the methods for designing and conducting tests.

2

Levels of Testing

Identifying testing levels such as unit, performance, string, integration, systems recovery, acceptance, parallel, performance, and interface testing.

3

Testing Different Types of Software

The changes in the approach to testing when testing different development approaches such as batch processing, client/server, Web-based, object-oriented, and wireless systems.

4

Independent Testing

Testing by individuals other than those involved in product/system development.

5

Vocabulary

The technical terms used to describe various testing techniques, tools, principles, concepts, and activities.

6

The Multiple Roles of Software Testers

The objectives that can be incorporated into the mission of software testers. This would include the testing to determine whether requirements are met, testing effectiveness and efficiency, testing user needs versus software specifications, and testing software attributes such as maintainability, ease of use, and reliability.

7

Testers Workbench

An overview of the process that testers use to perform a specific test activity, such as developing a test plan or preparing test data.

8

The V Concept of Testing

The V concept relates the build components of the development phases to the test components that occur during the test phases.

Knowledge Category 2: Building the Test Environment The test environment comprises all the conditions, circumstances, and influences surrounding and affecting software testing. The environment includes the organization’s policies, procedures, culture, attitudes, rewards, test processes, test tools, methods for developing and improving test processes, management’s support of software testing, as well as any test labs developed for the purpose of testing software and multiple operating environments. This category also includes ensuring the test environment fairly represents the production environment.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Knowledge of Test Process Selection and Analysis

Concepts of Test Processes—The concepts of policies, standards, and procedures, and their integration into the test process.

Test Process Selection—Selecting processes that lead to efficient and effective testing activities and products.

Acquisition or Development of a Test Bed/Test Lab/Test Processes—Designing, developing, and acquiring a test environment that simulates the “real” world, including the capability to create and maintain test data.

Quality Control—Testing quality control to ensure that the test process has been performed correctly.

Test Process Analysis—Analyzing the test process to ensure

Continuous Improvement—Identifying and making improvements to the test process using formal process improvement processes.

Adapting the Test Environment to Different Software Development Methodologies—Establishing the environment to properly test the methodologies used to build software systems, such as waterfall, Webbased, object-oriented, agile, and so forth.

Competency of the Software Testers—Providing the training necessary to ensure that software testers are competent in the processes and tools included in the test environment.

2

Test Tools

Tool Development and/or Acquisition—Understanding the processes for developing and acquiring test tools.

Tool Usage—Understanding how tools are used for automated regression testing, defect management, performance/load testing; understanding manual tools such as checklists, test scripts, and decision tables; using traceability tools, code coverage, and test case management.

3

Management Support for Effective Software Testing

Creating a tone that encourages testers to work in an efficient and effective manner.

Aligning test processes with organizational goals, business objectives, release cycles, and different developmental methodologies.

Knowledge Category 3: Managing the Test Project Software testing is a project with almost all the same attributes as a software development project. Software testing involves project planning, project staffing, scheduling and budgeting, communicating, assigning and monitoring work, and ensuring that changes to the project plan are incorporated into the test plan.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Test Planning, Scheduling, and Budgeting

Alignment—Ensuring the test processes are aligned with organizational goals, user business objectives, release cycles, and different development methodologies.

Test Performance—Monitoring test performance for adherence to the plan, schedule and budget, reallocating resources as required, and averting undesirable trends.

Staffing—Acquiring, training, and retaining a competent test staff.

Management of Staff—Keeping staff appropriately informed, and effectively utilizing the test staff.

Differences Between Traditional Management—Using a hierarchical structure versus quality management using a flattened organization structure.

2

Personal and Organizational Effectiveness

Communication Skills

Personal Effectiveness Skills

Project Relationships—Developing an effective working relationship with project management, software customers, and users.

Recognition—Showing appreciation to individuals and teams for work accomplished.

Motivation—Encouraging individuals to do the right thing and do it effectively and efficiently.

Mentoring—Working with testers to ensure they master the needed skills.

Management and Quality Principles—Understanding the principles needed to build a world-class testing organization.

3

Leadership

Meeting Chairing—Organizing and conducting meetings to provide maximum productivity over the shortest time period.

Facilitation—Helping the progress of an event or activity. Formal facilitation includes well-defined roles, an objective facilitator, a structured meeting, decision-making by consensus, and defined goals to be achieved.

Team Building—Aiding a group in defining a common goal and working together to improve team effectiveness.

Knowledge Category 4: Test Planning Testers need the skills to plan tests. Test planning assesses the business and technical risks of the software application and then develops a plan to determine if the software minimizing those risks. Test planners must understand the development methods and environment to effectively plan for testing.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Prerequisites to Test Planning

Identifying Software Risks—Demonstrating knowledge of the most common risks associated with software development.

Identifying Testing Risks—Demonstrating knowledge of the most common risks associated with software testing.

Identifying Premature Release Risk—Understanding how to determine the risk associated with releasing unsatisfactory, untested software products.

Risk Contributors—Identifying the contributors to risk.

Identifying Business Risks—Demonstrating knowledge of the most common risks associated with the business using the software.

Risk Methods—Understanding of the strategies and approaches for identifying risks or problems associated with implementing and operating information technology, products, and processes; assessing their likelihood, and initiating strategies to test for those risks.

Risk Magnitude—Demonstrating the ability to calculate and rank the severity of a risk quantitatively.

Risk Reduction Methods—Understanding the strategies and approaches that can be used to minimize the magnitude of a risk.

Contingency Planning—Planning to reduce the magnitude of a known risk.

2

Test Planning Entrance Criteria

Success Criteria/Acceptance Criteria—Understanding the criteria that must be validated to provide user management with the information needed to make an acceptance decision.

Test Objectives—Understanding the objectives to be accomplished through testing.

Assumptions—Establishing the conditions that must exist for testing to be comprehensive and on schedule.

Issues—Identifying specific situations/products/processes that, unless mitigated, will impact forward progress.

Constraints—Limiting factors to success.

Entrance Criteria/Exit Criteria—Understanding the criteria that must be met prior to moving software to the next level of testing or into production.

Test Scope—Understanding what is to be tested.

Test Plan—Understanding the activities and deliverables to meet a test’s objectives.

Requirements/Traceability—Defining the tests needed and relating them to the requirements to be validated.

Estimating—Determining the resources and timeframes required to accomplish the planned activities.

Scheduling—Establishing milestones for completing the testing effort and their dependencies on meeting the rest of the schedule.

Staffing—Selecting the size and competency of the staff needed to achieve the test plan objectives.

Test Check Procedures—Incorporating test cases to ensure that tests are performed correctly.

Software Configuration Management—Organizing the components of a software system, including documentation, so that they fit together in working order.

Change Management—Modifying and controlling the test plan in relationship to actual progress and scope of system development.

Version Control—Understanding the methods to control, monitor, and achieve change.

Knowledge Category 5: Executing the Test Plan This category addresses the skills required to execute tests, design test cases, use test tools, and monitor testing.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Test Design and Test Data/Scripts Preparation

Specifications—Ensuring test data scripts meet the objectives included in the test plan.

Cases—Developing test cases, including techniques and approaches for validation of the product. Determination of the expected result for each test case.

Test Design—Understanding test design strategies and attributes.

Scripts—Developing the online steps to be performed in testing; focusing on the purpose and preparation of procedures; emphasizing entrance and exit criteria.

Data—Developing test inputs; using data generation tools; determining the data set or sub-sets to ensure a comprehensive test of the system; determining data that suits boundary value analysis and stress testing requirements.

Test Coverage—Achieving the coverage objectives in the test plan to specific system components.

Platforms—Identifying the minimum configuration and platforms on which the test must function.

Test Cycle Strategy—Determining the number of test cycles to be conducted during the test execution phase of testing; determining what type of testing will occur during each test cycle.

2

Performing Tests

Execute Tests—Performing the activities necessary to execute tests in accordance with the test plan and test design—including setting up tests, preparing test data base(s), obtaining technical support, and scheduling resources.

Compare Actual Versus Expected Results—Determining whether the actual results meet expectations.

Documenting Test Results—Recording test results in the appropriate format.

Use of Test Results—Understanding how test results should be used and who has access to them.

3

Defect Tracking

Defect Recording—Recording defects to describe and quantify deviations from requirements/expectations.

Defect Reporting—Reporting the status of defects, including severity and location.

Defect Tracking—Monitoring defects from the time of recording until satisfactory resolution has been determined and implemented.

4

Testing Software Changes

Static Testing—Evaluating changed code and associated documentation at the end of the change process to ensure correct implementation.

Regression Testing—Testing the whole product to ensure that unchanged functionality performs as it did prior to implementing a change.

Verification—Reviewing requirements, design, and associated documentation to ensure they are updated correctly as a result of the change.

Knowledge Category 6: Test Status, Analysis, and Reporting Testers need to demonstrate the ability to develop status reports. These reports should show the status of the testing based on the test plan. Reporting should document what tests have been performed and the status of those tests. To properly report status, testers should review and conduct statistical analysis on the test results and discovered defects. The lessons learned from the test effort should be used to improve the next iteration of the test process.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Metrics of Testing

Using quantitative measures and metrics to manage the planning, execution, and reporting of software testing.

2

Test Status Reports

Code Coverage—Monitoring the execution of software and reporting on the degree of coverage at the statement, branch, or path level.

Requirement Coverage—Monitoring and reporting the number of requirements tested, and whether they are correctly implemented.

Test Status Metrics—Understanding the following metrics:

3

Final Test Reports

Reporting Tools—Using word processing, database, defect tracking, and graphic tools to prepare test reports.

Test Report Standards—Defining the components that should be included in a test report.

Statistical Analysis—Demonstrating the ability to draw statistically valid conclusions from quantitative test results.

Knowledge Category 7: User Acceptance Testing The objective of software development is to meet the true needs of the user, not just the system specifications. Testers should work with the users early in a project to clearly define the criteria that would make the software acceptable in meeting the user needs. As much as possible, once the acceptance criteria have been established, they should integrate it into all aspects of development.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Concepts of Acceptance Testing

Understanding the difference between system test and acceptance test.

2

Acceptance Test Planning Process

Defining the acceptance criteria.

Developing an acceptance test plan for execution by user personnel.

Testing data using use cases.

3

Acceptance Test Execution

Executing the acceptance test plan.

Developing an acceptance decision based on the results of acceptance testing.

Signing off on successful completion of the acceptance test plan.

Knowledge Category 8: Testing Software Developed by Outside Organizations Many organizations do not have the resources to develop the type and/or volume of software needed to effectively manage their business. The solution is to obtain or contract for software developed by another organization. Software can be acquired by purchasing commerical off-the-shelf software (COTS) or contracting for all or parts of the software development to be done by outside organizations.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Understanding the difference between testing software developed in-house and software developed by outside organizations.

2

Understanding the election process for selecting COTS software.

3

Verifying that testers are able to

4

Using the same approach as used for in-house software, but may need to be modified based on documentation available from the developer.

5

Understanding the following objectives:

Knowledge Category 9: Testing Software Controls and the Adequacy of Security Procedures The software system of internal control includes the totality of the means developed to ensure the integrity of the software system and the products created by the software. Controls are employed to control the processing components of software, ensure that software processing is in accordance with the organization’s policies and procedures, and according to applicable laws and regulations.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Principles and Concepts of a Software System of Internal Control and Security

Vocabulary of Internal Control and Security—Understanding the vocabulary of internal control and security, including terms such as risk, threat, control, exposure, vulnerability, and penetration.

Internal Control and Security Models—Understanding internal control and security models (specifically, the COSO [Committee of Sponsoring Organizations] model).

2

Testing the System of Internal Controls

Perform Risk Analysis—Determining the risk faced by the transactions/events processed by the software.

Determining the controls for each of the processing segments for transactions processing, including

Determining whether the identified controls are adequate to reduce the risks to an acceptable level.

3

Testing the Adequacy of Security for a Software System

Evaluating the adequacy of management’s security environment.

Determining the types of risks that require security controls.

Identifing the most probable points where the software could be penetrated.

Determining the controls at those points of penetration.

Assessing whether those controls are adequate to reduce the security risks to an acceptable level.

Knowledge Category 10: Testing New Techniques Testers require skills in their organization’s current technology, as well as a general understanding of the new information technology that might be acquired by their organization.

FULLY COMPETENT

PARTIALLY COMPETENT

NOT COMPETENT

1

Understanding the Challenges of New Technologies

New application architecture

New application business models

New communication methods

New testing tools

2

Evaluating New Technologies to Fit into the Organization’s Policies and Procedures

Assessing the adequacy of the controls within the technology and the changes to existing policies and procedures that will be needed before the new technology can be implemented effectively. This would include:

Testing new technology to evaluate actual performance versus supplier’s stated performance.

Determining whether current policies and procedures are adequate to control the operation of the new technology and modify to bring in currency.

Assessing the need to acquire new staff skills to effectively implement the new technology

1

Software Testing Principles and Concepts

8

2

Building the Test Environment

12

3

Managing the Test Project

16

4

Test Planning

27

5

Executing the Test Plan

19

6

Test Status, Analysis and Reporting

8

7

User Acceptance Testing

5

8

Testing Software Developed by Outside Organizations

6

9

Testing Software Controls and the Adequacy of Security Procedures

11

10

Testing New Technologies

8

Total

120

Multiply Total By

3

2

1

Multiplied Total

Total the Sum in Each of the Three Columns

Divide by 120

Software Testing Competency Score

1

Software Testing Principles and Concepts

2

Building the Test Environment

3

Managing the Test Project

4

Test Planning

5

Executing the Test Plan

6

Test Status, Analysis and Reporting

7

User Acceptance Testing

8

Testing Software Developed by Outside Organizations

9

Testing Software Controls and the Adequacy of Security Procedures

10

Testing New Technologies