Contents
Part I: What Is Automated Testing?
1. The Birth and Evolution of Automated Testing
1.2 Background on Software Testing
1.3 The Automated Test Life-Cycle Methodology (ATLM)
1.3.1 Decision to Automate Test
1.3.3 Automated Testing Introduction Phase
1.3.4 Test Planning, Design, and Development
1.3.5 Execution and Management of Tests
1.3.6 Test Program Review and Assessment
1.4 ATLM’s Role in the Software Testing Universe
1.4.1 ATLM Relationship to System Development Life Cycle
1.4.2 Test Maturity Model (TMM)—Augmented by Automated Software Testing Maturity
1.4.3 Test Automation Development
2.1 Overcoming False Expectations for Automated Testing
2.1.1 Automatic Test Plan Generation
2.1.3 Immediate Test Effort Reduction
2.1.4 Immediate Schedule Reduction
2.1.6 Universal Application of Test Automation
2.1.7 One Hundred Percent Test Coverage
2.2 Benefits of Automated Testing
2.2.1 Production of a Reliable System
2.2.2 Improvement of the Quality of the Test Effort
2.2.3 Reduction of Test Effort and Minimization of Schedule
Case Study: Value of Test Automation Measurement
2.3 Acquiring Management Support
3. Automated Test Tool Evaluation and Selection
3.1 The Organization’s Systems Engineering Environment
3.1.1 Third-Party Input from Management, Staff, and End Users
3.1.2 Tool Criteria Reflecting the Systems Engineering Environment
3.1.3 Level of Software Quality
3.1.4 Help Desk Problem Reports
3.1.7 Long-Term Investment Considerations
3.2 Tools That Support the Testing Life Cycle
3.2.1 Business Analysis Phase Tools
3.2.2 Requirements Definition Phase Tools
3.2.3 Tools for the Analysis and Design Phase
3.2.6 Other Testing Life-Cycle Support Tools
3.3.1 Improvement Opportunities
3.4 Evaluation Domain Definition
Part II: Introduction of Automated Testing to a Project
4. Automated Testing Introduction Process
4.1.2 Goals and Objectives of Testing
Case Study: Test Objectives and Strategies
4.2.1 Review of Project-Specific System Requirements
4.2.2 Application-Under-Test Overview
4.2.3 Review of Project Schedule
4.2.4 Test Tool Compatibility Check
4.2.5 Demonstration of the Tool to the Project Team
4.2.6 Test Tool Support Profile
4.2.7 Review of Training Requirements
5.1 Organizational Structure of a Test Team
5.1.4 Systems Methodology and Test Team
5.3.1 Test Team Sizing Methods: An Overview
5.3.2 Development Ratio Method
5.3.6 Test Effort Sizing Factors
5.4.6 Test Engineer Interviews
5.4.7 Distinguishing the Best Candidate
5.5 Roles and Responsibilities
Part III: Test Planning and Preparation
6. Test Planning: Smart Application of Testing
6.2.2 Critical/High-Risk Functions
6.2.3 Test Goals, Objectives, and Strategies
6.2.7 Test Requirements Definition
6.3 Test Requirements Management
6.3.1 Requirements Management Tools
6.3.2 Assessing the Test Requirements Risk
6.3.4 Requirements Traceability Matrix
6.4 Test Program Events, Activities, and Documentation
6.5.1 Test Environment Preparations
6.5.2 Test Environment Integration and Setup
6.6.1 Test Completion/Acceptance Criteria
7.1 Test Requirements Analysis
7.1.1 Development-Level Test Analysis (Structural Approach)
7.1.2 System-Level Test Analysis (Behavioral Approach)
7.2.1 Test Program Design Models
7.2.2 White-Box Techniques (Development-Level Tests)
7.2.3 Black-Box Techniques (System-Level Tests)
7.2.4 Test Design Documentation
7.3.1 Test Procedure Definition
7.3.2 Automated Versus Manual Test Analysis
7.3.3 Automated Test Design Standards
Case Study: Naming Conventions
7.3.4 Manual Test Design Guidelines
8.1 Test Development Architecture
8.1.2 Environment Readiness Checks
8.1.3 Automation Framework Reuse Analysis
8.1.4 Test Procedure Development/Execution Schedule
8.1.5 Modularity-Relationship Analysis
8.1.6 Explanation of the Sample Modularity-Relationship Matrix
8.1.7 Calibration of the Test Tool
8.1.8 Compatibility Work-Around Solutions
Case Study: Incompatibility Work-Around Solution
8.1.9 Manual Execution of Test Procedures
8.1.10 Test Procedure Inspections—Peer Reviews
8.1.11 Test Procedure Configuration Management
8.2 Test Development Guidelines
8.2.1 Design-to-Development Transition
8.2.2 Reusable Test Procedures
Case Study: Navigation Using Tabs or Mouse Clicks
Case Study: Testing Bitmaps Using a Capture/Playback Tool [2]
8.2.3 Maintainable Test Procedures
Case Study: Automating Documentation
Case Study: Automated Random Testing
Example 1. Determine Available Free Memory
Example 2. Identify System Resources in Use
Example 1. [6]: Sort the array contents by means of a .dll function.
Example 2. Sort a field according to class and x-position
8.3.1 Table-Driven Test Automation
8.3.2 PC Environment Automated Setup Script
8.3.3 Automated Recording Options
Case Study: Smoke Test Application
8.3.10 Help Function Verification Script
8.3.11 Timed Message Boxes Function
8.3.12 Advanced Math Functions
Part IV: Test Execution and Review
9.1 Executing and Evaluating Test Phases
9.1.1 Unit Test Execution and Evaluation
9.1.2 Integration Test Execution and Evaluation
9.1.3 System Test Execution and Evaluation
9.1.4 Test Results Analysis of Regression Tests
9.1.5 User Acceptance Test Execution and Evaluation
9.2 Defect Tracking and New Build Process
9.3 Test Program Status Tracking
9.3.1 Earned Value Management System
Case Study: System Test Status Tracking
9.3.2 Test Metrics Collection and Analysis
10. Test Program Review and Assessment
10.1 Test Program Lessons Learned—Corrective Actions and Improvement Activity
10.2 Test Program Return on Investment
Case Study: Test Program Return on Investment
Case Study: Quantify Tool Return on Investment [1]
Quantify Tool’s Return on Investment
A.1 Requirements Testing Approach
Make the Requirement Measurable
B. Tools That Support the Automated Testing Life Cycle
B.2.2 Configuration Management Tools
B.2.4 Technical Review Management
B.2.5 Documentation Generators
B.3 Requirements Definition Phase
B.3.1 Requirements Management Tools
B.4.2 Structure Charts, Flowcharts, and Sequence Diagrams
B.4.3 Test Procedure Generators
B.5.1 Syntax Checkers/Debuggers
B.5.2 Memory Leak and Runtime Error Detection Tools
B.5.4 Static and Dynamic Analyzers
B.5.5 Unit and Integration Test Tools
B.6.1 Code (Test) Coverage Analyzers and Code Instrumentors
B.6.2 Usability Measurement Tools
B.8.3 GUI Application Testing Tools
B.8.4 Load/Performance Testing Tools
C.4 Technical Stewardship Stage
C.5 Test/Project Management Stage
C.6 Business/Product Management Stage
D.2 Roles and Responsibilities
D.2.2 Project Roles and Responsibilities
D.4.1 Test Environment Configuration
D.5.4 Configuration Management
Appendix D.A Test Procedure Development Guidelines
Appendix D.B Test Verification Summary Matrix
Appendix D.C Test Procedures and Test Scripts
E.4 Test Tool Compatibility Checks
E.6 Baselined System Setup and Configuration
E.7 Software Installations in the Test Environment Baseline
E.8 Overall Test Program Objectives
E.10 Test Procedure Design and Development Standards
E.11 Automated versus Manual Test Analysis
E.13 Test Team Communication with Other Teams
E.16 Defect Documentation and Reporting
E.17 Automated Test Advocates and Experts
E.20 Test Tool Improvement Suggestions