From a tester’s perspective, the software development methodology used significantly affects software testing. Suppose, for instance, that one could develop a continuum of software development methodologies from no methodology to a perfect methodology. The perfect methodology produces no defects, thus negating the need for testing. In contrast, with no methodology, one must exhaustively test to provide an opinion as to whether to place the software in operation.
The reliability of the software development process also depends significantly on whether the development team complies with the process. In many organizations, compliance with the process is not required. Those organizations use a system development methodology as a guideline, selecting those components the project staff wants to use and deleting those components they do not want to use. The benefits of using the system development methodology are significantly reduced when the methodologies are not complied with during development.
This section covers the following six topics, each of which can affect the amount of testing required for software development:
Methodology types
Defining requirements
Methodology maturity
Project staff competency
Project staff experience
Configuration-management controls
This section explains why these six components affect the amount of testing required. Following this discussion, another section allows the tester to assess these six components and thus estimate the amount of testing required.
A software development methodology provides guidelines for how to build software. In the early days of computing, software project managers had two responsibilities: to develop a process for building software, and to follow that process to project completion. Because all project leaders need a software development methodology, standardized processes were developed. Some organizations create their own software development methodology, whereas others acquire the methodology for building software from suppliers.
This section discusses the following topics:
The “Overview” subsection identifies the challenges a project manager faces, the guidelines for choosing a methodology, and the project manager’s responsibilities when applying a specific methodology.
The “Methodology Types” subsection concisely describes several common software development methodologies.
The “Software Development Life Cycle” subsection covers six common phases in the software development life cycle (including responsibilities, external development services, and documentation within those phases).
Many software project managers use the development methodology as a guideline rather than a process that requires strict adherence. If these processes are not followed as written, the developers have to create the process or parts of the process to build software.
Therefore, software project managers should follow the developmental methodology for these reasons:
To focus the entire effort on building software, not on building a developmental process
To achieve consistency, which results because all projects use the developmental methodology
To improve the developmental methodology itself, which is possible only if the organization gains experience (positive and negative) by following the methodology and learning from that experience
When developmental methodologies are used only as guidelines, improvements to the methodologies are unnecessary. However, when compliance is required, defect-prone components are quickly replaced by more effective developmental processes.
Thousands of different methodologies are used to build software. No single developmental methodology has proven to be best. Most work, but some may not fit the needs of the specific project. This creates a dilemma for the software project manager.
The solution is for an organization to standardize on one or a few development methodologies. For example, they may have a development methodology for a waterfall development effort, and another methodology for a prototype effort. Because of the unique characteristics of specific projects, a standardized methodology may not be effective for the whole project.
An agile development methodology is widely promoted. Although no definition of an agile development methodology has been generally accepted, it means flexibility in using the development methodology. In other words, the users have options when they are applying an agile development methodology.
Most organizations believe that agile development is effective only in small projects with a small number of developers. Currently, this is how most agile methodologies are used. Some, however, believe that agile techniques can be used for larger projects.
Many different methodologies are available for developing software. The project manager must select the developmental methodology most appropriate to the project needs. Each methodology has advantages and disadvantages. A concise description of the six most common developmental methodologies follows.
The waterfall methodology is the oldest software development methodology. It follows a logical progression of defining requirements, designing the system, building the system, testing the system, and placing the system in operation. It is primarily used for the development of systems that do batch processing. The waterfall methodology assumes that at the end of the requirements phase, requirements for development are known.
The prototyping methodology assumes that the user does not have a rigid definition of requirements. Prototyping applies in an “I’ll know it when I see it” environment. Prototyping produces an operational methodology of the user’s requirements as currently defined. The user can then see the system in operation and make changes for another generation or prototype of the system. The development of prototype versions continues until the user believes it contains the requirements desired by the user. Most agree that the prototype should not be put into operation as is, because a prototype methodology lacks the necessary controls to ensure correct and high-quality processing.
The objective of rapid application development (RAD) is a quick turnaround time with emphasis on requirements definition, which is accomplished by heavy user involvement throughout all developmental phases. RAD makes extensive use of development tools. Much of its effectiveness and reduced cycle time is due to rapid movement from one phase of development to the next.
The spiral methodology focuses on objectives, alternatives, constraints, and risks. The spiral methodology, like RAD, heavily involves the user. At each part of the spiral development, the risks, constraints, and alternative methods are evaluated. The objective is to use the best possible system from a business perspective, although development may require a longer development cycle than RAD.
The objective of the incremental methodology is to develop the system in parts (or increments). It is effective when single parts of the system can be used immediately without the full system being developed or when a particular part is critical to the success of the overall system, but there’s some uncertainty as to whether that part can be effectively developed. The incremental method can be effective in reducing the risk of investing and building an entire system when the outcome is questionable.
The V methodology is mostly associated with software testing. It develops two processes: one for building the system, and one for testing the system. The two processes are then interrelated as a V methodology. The V shows development on one side and testing on the other side. For example, during the requirements stage of development, the software acceptance testing side is developed. The V methodology assumes that approximately one-half of the total development effort will be spent on testing. The V then integrates testing so that testing is more effective and defects are uncovered earlier in the developmental process.
Although no single generally accepted developmental methodology exists, all development methodologies share some attributes. The most common shared attributes are the phases or steps required, the individuals involved in development and their roles and responsibilities, and the deliverables produced.
The following example represents a large software development process. Project managers should have a general understanding of these common attributes. However, project managers should not expect that all systems can be developed in accordance with this specific software development life cycle (SDLC) methodology. This methodology is for explanatory purposes only. The following SDLC topics are discussed:
Six common phases of the SDLC
Roles and responsibilities
The phases described in this guide are intended to clarify the broad functions or activities that occur during the development of an automated system. The following six phases cover activities commonly performed.
The initiation phase begins with the recognition of a problem and the identification of a need. During this phase, the need is validated, and alternative functional concepts to satisfy the need are explored, recommended, and approved. The decision to pursue a solution must be based on a clear understanding of the problem, a preliminary investigation of alternative solutions (including non–computer-based solutions), and a comparison of the expected benefits versus the cost (including design, construction, operation, and potential risks) of the solution. At this stage, the sensitivity of the data controlled by the SDLC under consideration should be evaluated.
In this phase, the functional requirements are defined, and detailed planning for the development of an operable SDLC is begun. Functional requirements and processes to be automated are documented and approved by appropriate senior management before an SDLC development effort is started. Requirements identification is iterative, as is the analysis of potential risk, and involves those who identify and solve problems. It is critical that internal control and specific security requirements be identified during this process. Requirements may be, and commonly are, modified in later phases as a better understanding of the problem is gained.
Also during the definition phase, a Project Plan is prepared that describes the unique SDLC methodology to be used during the life of the particular project. It specifies a strategy for managing SDLC development, certification, and accreditation. It defines goals and activities for all subsequent phases, and includes resource estimates during each phase, intermediate milestones, and methods for design, documentation, problem reporting, and change control. Resource planning for VV&T (verification, validation, and testing) should be included here.
The activities performed during this phase result in a specification of the problem solution. The solution provides a specific high-level definition, including information aggregates, information flows and logical processing steps, and all major interfaces and their inputs and outputs. The purpose is to refine, resolve deficiencies in, define additional details in, and package the solution. The design specifications describe the physical solution (algorithms and data structures) in such a way that it can be implemented in code with little or no need for additional analysis.
Organizations should define and approve security requirements prior to acquiring or starting formal development of the applications. The VV&T goals are also identified during this phase, and a plan for achieving these goals is developed. The Project Plan and Risk Analysis are reviewed and revised as required given the scope and complexity of the solution formulated.
This phase results in programs that are ready for testing, evaluation, certification, and installation. Programming is the process of implementing the detailed design specifications into code. Completed code then undergoes unit testing, as described in the revised VV&T Plan (also completed in this phase), and integration and system testing in Phase 5. User and Maintenance Manuals are prepared during this phase, as is a preliminary Installation Plan that specifies the approach to, and details of, the installation of the SDLC.
In this phase, integration and system testing of the SDLC occurs. For validation purposes, the system should be executed on test data, and the SDLC field tested in one or more representative operational sites.
The purpose of this final life cycle phase is to:
Implement the approved operational plan, including extension or installation at other sites.
Continue approved operation.
Budget adequately.
Control all changes and maintain or modify the SDLC during its remaining life.
Problem reporting, change requests, and other change-control mechanisms are used to facilitate the systematic correction and evolution of the SDLC. In addition, periodic performance measurement and evaluation activities are performed to ensure that the system continues to meet its requirements in a cost-effective manner in the context of a changing system environment. These reviews may be conducted by either the quality assurance (QA) staff or the audit unit or both.
Software project managers must recognize that organizational structures vary significantly from organization to organization. The functions covered in this section are described as “job-title-related” functions so that organizations can look at them as specific job titles, if they have an equivalent job, or as functions that must be performed regardless of whether the specific job exists. The list is not meant to be all-inclusive, nor does it preclude smaller organizations or organizational units from combining participants or roles.
The rationale for describing the participants is to identify the role of each key participant. The project manager should verify that the respective SDLC participants have each performed his or her appropriate role. The following list describes all the participants:
Sponsor/user. The sponsor/user is responsible for initially identifying the need that the SDLC must meet. The sponsor/user must identify various alternative solutions to the problem and determine the feasibility and cost/benefit of the various alternatives. The sponsor/user also conducts or oversees a Risk Analysis to assess the potential vulnerabilities of the system or application under development. The analysis must be continually updated or revised during the SDLC to ensure the inclusion of appropriate internal controls and security safeguards.
The sponsor/user is ultimately responsible for accepting (accrediting) the system as complete, meeting its requirements, and being ready for operational use. Depending on the particular system, the sponsor/user may be located at various levels in the organization.
Project manager. The project manager is responsible for seeing that a system is properly designed to meet the sponsor/user’s needs, and is developed on schedule. The project manager is responsible for seeing that all system documentation is prepared as the system is being developed. If the system is developed either in-house or by a contractor, the project manager is responsible for certifying that the delivered system meets all technical specifications, including security, and obtaining technical assistance from the IT manager as necessary.
System security specialist (SSS). This individual is responsible, at the program or operational level, for ensuring that a system complies with the organization’s computer/system security policy. The SSS approves design reviews to ensure that 1) the design meets approved security specifications and system tests, and 2) administrative, physical, and technical requirements are adequate prior to installation of the system.
Internal control specialist (ICS). This individual is responsible, at the operational level, for seeing that a system complies with the organization’s internal control policy. The ICS ensures that a system meets basic standards for documentation, recording of transactions, execution of transactions, separation of duties, access to resources, and all other internal control requirements.
Contracting officer. The contracting officer is responsible for awarding and managing vendor contracts to provide part or all of the system development activity that is not in-house. The contract might also provide for the procurement of system software required by a new application. The contracting officer, in either case, is responsible for seeing that the vendor or contractor complies with the terms of the contract and that the deliverables are provided on time.
Information technology (IT) manager. The IT manager is the technical individual responsible for the automatic data processing (ADP) installations and operations of an organization’s programs (i.e., they are responsible for the operation of the data processing center and the management of the systems analysts, programmers, and so on). The IT organization may actually develop parts of the SDLC or may provide technical support to the project manager and sponsor/user during the system’s life cycle.
Quality assurance (QA) specialist. The operations-level QA staff is responsible for assuring the sponsor/user that an application system is developed in accordance with the system’s stated objectives, contains the needed internal controls and security to produce consistently reliable results, and operates in conformance with requirements and data processing procedures. Quality assurance, as defined in the SDLC matrix, is the function that establishes the responsibilities and methods used to ensure quality in data processing products. The QA specialist may or may not be personally involved in establishing these responsibilities and methods.
The QA charter should allow for independent reviews. QA staff should actively participate in reviewing the development of new systems or applications and the significant modification of existing systems. (Coordination with security/audit and VV&T participants is essential to avoid duplication of effort.) In addition, the QA staff should ensure data integrity of systems. The presence and effective functioning of the QA staff will determine the nature and extent of audit involvement, in that they commonly perform similar functions.
Software projects require that requirements be well defined, both for the system’s analysts and the users. The objective of this section is to help develop a requirements definition that meets the needs of the software project. This section explains how to identify what is a requirement, as well as what is not a requirement.
The following are working definitions of requirements as used in a software project:
Requirement. An attribute or characteristic to be possessed by a product or service
Requirements definition. A process by which customers and producers reach an understanding of requirements
Statement of requirements. A specification deliverable produced as a result of a requirements definition
These definitions identify three unique aspects of requirements. The requirements definition is the process, a predefined series of tasks that leads to a predefined deliverable, which the IT analyst and the customer go through to identify the requirements. These requirements are then documented in the deliverable known as the statement of requirements.
The following four requirements categories provide a structure for discussing requirements. These categories do not represent rigid boundaries, and most people will have legitimate concerns in all four while predominantly concentrating their efforts in one.
Business requirements. The customers for the business requirements are the business users. Because we’re talking about what the system “must do,” it’s the business users who are uniquely positioned to define these requirements. The IT staff, to the extent that they have learned or are learning the business, can assist in clarifying and defining these requirements; but only active business users can own them. The list of requirements includes all of the needs and wants of the user community. Although the definition of these requirements is generally presented in a “best-case” scenario, the priority of these requirements is sorted so that needs are addressed before nice-to-haves.
Implementation requirements. The customers for the implementation requirements are IT’s management because they are responsible for providing future production services to the business community. These are the things that the system “must be,” including day-to-day production service levels as well as training, enhancement, and maintenance activity; and disaster recovery and capacity planning. Although the business requirements present the best-case scenario, implementation requirements generally identify the “target-case” scenario. Business users often express direct concerns about these implementation specifics. The strength of this concern is usually directly proportional to the level of implementation problems associated with other application systems already in use.
Constraint requirements. Constraints are generally the requirements of life placed on the project from the outside. The customers for these requirements are management—executive, line business, and information systems—who are allocating resources to the project effort. Typical constraint requirement examples include budgets, schedules, and operating environments. These realities generally force the “worst-case” scenario.
Enterprise-wide requirements. Requirements applicable to all software systems, such as security.
This subsection discusses requirement attributes from three different perspectives: a systems analyst perspective, a tester perspective, and international industry standards.
Attributes of a requirement do not define what something “should do.” Attributes define what something “should be.” Desired attributes of a requirement define the positive properties that a requirement should have, such as the following:
Clear
Complete
Consistent
Correct
Modifiable
Pure
Relevant
Testable
Traceable
Usable
Later, we discuss project-verification strategies that focus on ensuring that requirements reasonably embody these attributes.
Verifying and validating requirements is ultimately a subjective assessment based on the agreement of all stakeholders. By providing measurement scales that focus on different aspects of requirements quality, the subjective assessments can be made more objective. Each requirements measure isolates a single aspect of the requirements so that quantitative assessments can be collected and prioritized.
Criticality. The requirement can be tied directly to an organizational goal or critical success factor. It’s not just a nice-to-have. Failure to conform to the requirement would hurt the business.
Measurability. It’s possible to tell whether the requirement is being satisfied. Reasonable people would not disagree about the level of conformance.
Controllability. It’s possible to differentiate systemic from random variation in the processes surrounding the requirement. This means that we can identify future nonconformance and categorize them according to frequency and severity.
Completeness. All items needed for the specification of the requirements of the solution to the problem have been identified.
Correctness. Each item in the requirements specification is free from error.
Clearness. Each item in the requirements specification is exact and not vague. There is a single interpretation of each item in the requirements specification; the meaning of each item in the requirements specification is understood; the specification is easy to read.
Consistency. No item in the requirements specification conflicts with another item in the specification.
Relevance. Each item in the requirements specification is pertinent to the problem and its solution.
Testability. During program development and acceptance testing, it will be possible to determine whether the item in the requirements specification has been satisfied.
Traceability. Each item in the requirements specification can be traced to its origin in the problem environment.
Pureness. The requirements specification is a statement of the requirements that must be satisfied by the problem solution, and is not obscured by proposed solutions to the problem.
Usability. Each item can be understood by its user and contains the information needed by the user.
Modifiability. The requirements specification is expressed in such a way that each item can be changed without dramatically affecting other items.
The quality of a requirement can be measured on a continuum. A standard that dictates that every requirement must be defined to a level that supports statistical process control would be unreasonable. However, we should strive to define requirements that support the most rigorous review possible.
Several internationally recognized organizations are currently developing and publishing standards for the requirements definition process. One standard described is the Institute of Electrical and Electronics Engineers (IEEE) IEEE1233: Guide for Developing System Requirements Specification.
The IEEE has several process standards that relate to requirements. Three standards are explained here:
IEEE 1233 (4.2): Properties of System Requirements Specifications (SRS)
Unique Set. Each requirement is stated only once.
Normalized. Requirements should not overlap; that is, they shall not refer to other requirements or capabilities of other requirements.
Linked Set. Explicit relationships should be defined among individual requirements to show how the requirements are related to form a complete system.
Complete. An SRS should include all the requirements identified by the customer, as well as those needed for definition of the system.
Consistent. The SRS content should be consistent and noncontradictory in the level of detail, style of statements, and presentation of material.
Bounded. The boundaries, scope, and context for the set of requirements should be identified.
Configurable. Versions should be maintained across time and instances of the SRS.
IEEE 1233 (4.3.1): Organizing Requirements
Identify requirements that are derived from other requirements.
Organize requirements of different levels of detail into their appropriate levels.
Verify the completeness of the set of requirements.
Identify inconsistencies among requirements.
Clearly identify the capabilities, conditions, and constraints for each requirement.
Develop a common understanding with the customer of the purpose and objectives of the set of requirements.
Identify requirements that will complete the SRS.
IEEE 1233 (4.4): Intended Use
During systems design, requirements are allocated to subsystems, hardware, software, operations, and other major components of the system.
During the system build, the SRS is used to write appropriate system verification plans for all components.
During implementation, test procedures are defined using the SRS.
Many organizations have developed classification schemes to evaluate the maturity of a software development methodology. One of the most successful was developed by the Software Engineering Institute (SEI) under a grant from the U.S. Department of Defense. The SEI methodology has itself matured over time and is now referred to as the CMMI (Capability Maturity Model Integrated).
The SEI methodology has four defined levels of maturity and one undefined level. For the defined levels, SEI has described the “focus” for achieving that particular level of maturity.
Note that the undefined level of maturity is level 1, referred to as Initial. There are no requirements for level 1; everyone who develops software begins at this level. The SEI defines what must be achieved as levels 2, 3, 4, and 5. The focus of levels 2, 3, 4, and 5 follows:
Level 2: Managed. Provide more consistency and less variability in software systems by initiating good project-management processes.
Level 3: Defined. At this level, the organization’s attempts to minimize process variability and maximize the effectiveness and efficiency of developed software. At level 3, the processes are standardized, and a high level of compliance to process is required.
Level 4: Quantitatively managed. At this level, the processes are mature enough that they can produce reliable and quantitative data. Therefore, management can rely on the quantitative data produced by the developmental processes, and therefore can take action on the quantitative data.
Level 5: Optimizing. At this level, the IT organization can build software at the cost it says it can build it for, and implement it by the scheduled implementation date. Having this level of processing sophistication enables the organization to innovate new and unique ways to build software more effectively and efficiently.
Table 14-1 shows the process areas for each level that need to be developed and deployed before that level can be considered mature. For example, level 2 requires configuration management, level 3 risk management, level 4 quantitative project management, and level 5 causal analysis and resolution. Note that it is not necessary to have all process areas fully proficient to be evaluated as being at a particular maturity level.
Table 14-1. CMMI SS Version 1.1 Staged Representation: Process Areas by Maturity Level
LEVEL | FOCUS | PROCESS AREA |
|---|---|---|
5 Optimizing | Continuous Process Improvement | Organizational Innovation and Deployment Causal Analysis and Resolution |
4 Quantitatively Managed | Quantitative Management | Organizational Process Performance Quantitative Project Management |
3 Defined | Process Standardization | Requirements Development Technical Solution Product Integration Verification Validation Organizational Process Focus Organizational Process Definition Organizational Training Integrated Project Management Risk Management Integrated Teaming Integrated Supplier Management Decision Analysis and Resolution Organizational Environment for Integration |
2 Managed | Basic Project Management | Requirements Management Project Planning Project Monitoring and Control Supplier Agreement Management Measurement and Analysis Process and Product Quality Assurance Configuration Management |
1 Initial | No Requirements; No Specific Process Areas |
Note
The SEI model is continually maturing; for an updated version, visit www.sei.cmu.edu.
The methodology also is not meant to imply that the process areas only begin at the level at which they are defined. For example, validation or dynamic testing would be performed at levels 1 and 2. It is listed at level 3 to indicate that a process area such as validation itself is a mature process that can be relied on. Validation at levels 1 and 2 would not be expected to be nearly as effective as validation at level 3.
Another example is quantitative project management occurring at level 4. This does not mean that project managers do not use quantitative data until level 4. Exactly the opposite is true. At level 1, project managers use quantitative data for budgeting and scheduling. What it does mean is that the processes are not mature enough to produce reliable and valid data for use in managing projects. If project managers rely exclusively on the quantitative data produced at levels 1 and 2, they probably will make many erroneous project decisions.
Project management is the application of skills, tools, and techniques to meet successfully or exceed the objectives of a project in the specified time frame at a specified cost. It aims to meet the needs of the client. Another view is that software development refers to the definition, planning, control, and conclusion of a project.
To understand competencies needed to effectively develop software, examine the skills associated with successful software project managers as defined by the Software Quality Institute. The Quality Assurance Institute has also defined skills specifically needed by software developers and skills specifically needed by software testers. This section has divided all these skills into the following five categories:
Process-Selection Skills
Assessing processes
Awareness of process standards
Defining the product
Evaluating alternative processes
Managing requirements
Managing subcontractors
Performing the initial assessment
Selecting methods and tools
Tailoring processes
Tracking product quality
Understanding development activities
Project-Management Skills
Building a work breakdown structure
Documenting plans
Estimating cost
Estimating effort
Managing risks
Monitoring development
Scheduling
Selecting metrics
Selecting project-management tools
Tracking processes
Tracking project progress
People-Management Skills
Appraising performance
Handling intellectual property
Holding effective meetings
Interaction and communication
Leadership
Managing change
Negotiating successfully
Planning careers
Presenting effectively
Recruiting
Selecting a team
Team building
Building Software Skills
Understanding development methodologies
Initiating software projects
Defining software requirements
Designing systems
Building systems
Developing user documentation and training
Installing software systems
Changing software systems
Testing Software Systems
Developing software testing principles and concepts
Building the test environment
Managing the test project
Test planning
Executing the test plan
Monitoring test status, analysis, and reporting
Testing user acceptance
Testing software developed by outside organizations
Testing software controls and the adequacy of security procedures
Testing new technologies
The project staff is defined as the stakeholders who have a vested interest in the success of the project. Primarily this includes the software development staff and the user staff.
A good process used by an untrained individual may not work; and if it does work, it might not be nearly as efficient or effective as that process when used by someone with experience. For example, if you were trained in how to change a tire on your car, you could perform that process effectively and efficiently. However, if you’ve not been trained, you may have difficulty finding the jack, the tire, and other tools necessary for the tire-change process. An experienced tire changer may be able to change a tire in 10 minutes; the inexperienced tire changer may take an hour, even though the process is mature and fully defined in the owner’s manual.
Experienced user staff should be knowledgeable in the software development methodology and their role and responsibilities in the building of software systems. Development project staff members need to be experienced in the business of the users and the processes and tools used to build the software.
No single method can fully assess experience. Some criteria to evaluate experience are as follows:
Number of software systems built
Training courses attended
Like work performed through many development cycles
Years of experience
Professional certifications
Configuration management (CM) is one of the components of software project management that differentiates software project management from general project management. There are similarities in controlling change, but CM involves much more. It includes maintaining an inventory of the items included in the total project configuration.
Project managers must implement an internal CM system for the control of all configuration documentation, physical media, and physical parts representing or comprising the product. For software, the system must address the evolving developmental configuration and support environments (engineering, implementation, and test) used to generate and test the product. The product manager’s CM system must consist of the following elements:
Configuration identification
Configuration control
Configuration-status accounting
Configuration audits
Configuration identification (CI) includes the following:
Selection of CIs
Determination of the types of configuration documentation required for each CI
Issuance of numbers and other identifiers affixed to the CIs
Technical documentation that comprises the CIs’ configuration documentation
As a part of the configuration identification process, the project manager must determine the documentation that will be used to establish the configuration baseline(s) required by the contract.
The project manager must also identify the developmental configuration for each CI.
The project manager must apply internal configuration-control measures to the configuration documentation for each CI. The project manager must apply configuration-control measures to each baselined CI, and its configuration documentation, in accordance with this standard. The configuration process must:
Ensure effective control of all CIs and their approved configuration documentation
Provide effective means, as applicable, for the following:
Proposing changes to CIs.
Requesting deviations or waivers pertaining to such items.
A waiver is a written authorization to accept an item, which during manufacture, or after having been submitted for inspection or acceptance, is found to depart from specified requirements, but nevertheless is considered suitable for use “as is” or after repair by an approved method.
Preparing notices of revision.
Preparing specification-change notices.
Ensure implementation of approved changes
The project manager must implement a configuration-status accounting (CSA) system. As a minimum, the CSA system must:
Identify the current approved configuration documentation and identification number associated with each CI.
Record and report the status of proposed engineering changes from initiation to final approval or contractual implementation.
Record and report the results of configuration audits to include the status and final disposition of identified discrepancies.
Record and report the status of all critical and major requests for deviations and waivers that affect the configuration of a CI.
Record and report implementation status of authorized changes.
Provide the traceability of all changes from the original base-lined configuration documentation of each CI.
Report the affectivity and installation status of configuration changes to all CIs at all locations.
The project manager must plan a CM process in accordance with the requirements of this standard, tailored appropriately for the particular CI(s), their scope and complexity, and the contracted phase(s) of the life cycle. Planning must be consistent with the objectives of a continuous improvement process, which includes the analysis of necessary means to prevent reoccurrence. The project manager’s CM planning must include the following:
The objectives of the CM process and of each applicable CM element
The CM organization and organizational relationships
Responsibilities and authority of CM managers
CM resources, such as tools, techniques, and methodologies
Coordination with internal and external organizations.
CM policies, processes, procedures, methods, records, reports, and forms
The project manager must assign distribution codes in accordance with organizational standards. Access to data must be limited in accordance with the applicable distribution codes and by data rights, security requirements, and data status level.
The following subsections cover the administrative activities performed to fulfill the CM responsibilities.
The project leader’s CM plan must be in accordance with the IT standards and describe the processes, methods, and procedures to be used to manage the functional and physical characteristics of the assigned CI(s). The project leader must:
Develop the project leader’s CM plan
Submit the plan and changes to the IT organization CM board
Implement the activities required by this standard in accordance with the approved plan
The project leader must ensure traceability of CIs to the Work Breakdown Structure (WBS) elements.
The project leader must ensure that the CM representatives participate in all technical reviews conducted in accordance with IT standards. The role of CM in the technical review process requires that you do the following:
Evaluate the adequacy of the type and content of the configuration documentation.
Ascertain that the configuration documentation is under internal configuration control.
Determine whether problems and action items identified at the review will be addressed by the project manager.
The following tasks are those configuration-identification activities performed to fulfill the CM responsibilities.
The purpose of configuration identification is to incrementally establish and maintain a definitive basis for control and status accounting for a CI throughout its life cycle. To accomplish configuration identification, the project leader must, for both hardware and software:
Select CIs.
Select configuration documentation to be used to define configuration baselines for each CI.
Establish a release system for configuration documentation.
Define and document interfaces.
Enter each item of configuration documentation and computer software source code into a controlled developmental configuration.
Establish the functional, allocated, and product baselines at the appropriate points in the system or CI life cycle, upon contractual implementation of the applicable configuration documentation, and in accordance with contract requirements.
Assign identifiers to CIs and their component parts and associated configuration documentation, including revision and version numbers where appropriate. Assign serial and lot numbers, as necessary, to establish the CI for each configuration of each item of hardware and software.
Ensure that the marking and labeling of items and documentation with their applicable identifiers enables correlation between the item, configuration documentation, and other associated data.
Ensure that applicable identifiers are embedded in the source and object code.
The project leader must select CIs. Any item requiring logistics support or designated for separate procurement is a CI. However, all CIs associated with any given development project are not necessarily designated as CIs at the same point in time. Computer hardware will be treated as CIs. Computer software will be treated as CIs throughout the life of the process regardless of how the software will be stored.
The project leader must establish a documentation library and implement procedures for controlling the documents residing within the documentation library.
The project leader shall establish a software development library and implement procedures for controlling the software residing within the software development library.
CM establishes baselines to enable project leaders to measure change(s) from the progressive definition and documentation of the requirements, and design information describing the various CIs designation for a system.
Configuration documentation is first released to the involved stakeholders for their review and use. The initial release includes the incorporation of related information into the configuration-status accounting information system.
The marking and labeling of software will be as follows:
Software identifier and version (and computer program identification number where applicable) will be embedded in the source code header.
Each software medium used must be marked and labeled. For example, for magnetic disk media containing copies of tested software entities, each medium must be marked with a label that lists cross-references to applicable software identifiers of the entities it contains.
Media copy numbers must distinguish each copy of the software media from its identical copies. Each time a new version of the software is issued, new copy numbers, starting from 1, must be assigned.
Configuration control is the systematic proposal, justification, evaluation, coordination, and approval or disapproval of proposed changes. The implementation of all approved changes must be controlled. Each approved and implemented change will result in a new baseline being established.
The project leader must implement a configuration-control function that ensures the following:
Regulation of the flow of proposed changes
Documentation of the complete impact of the proposed changes
Release only of approved configuration changes into configuration identifications and their related configuration documentation
Configuration control begins with the establishment of a functional baseline and continues as further configuration baselines are established. Configuration control continues throughout the life cycle of the configuration identification.
It is important for testers to measure the impact that using the software development methodology has on the resources and time needed to test effectively. If testers believe that a proposed test schedule and budget is inadequate, that might not result in increased resources or time. However, when testers know that inadequate time or resources have been allocated to testing, they can redirect the test effort to the high-risk components of the software.
Work Paper 14-1 is an assessment worksheet for evaluating the six components of software development that affect software testing. Be aware, however, that there are more than six components that affect testing. However, statistics show that if you pick a limited number of the “right” components, they will pull other components in the same direction. What this means is that the assessment questions for staff competency do not cover all the required competencies. However, if these are the right components of staff competency, the staff should also have the other related components at the same time.
Table 14-1. Self-Assessment of the Components of Software Development That Impact Testing
YES | NO | ||
Type of Development Process | |||
1. | Has a process been developed that identifies the criteria that will be used to select the most appropriate type of software development methodology for the software project being developed? | ||
2. | Does the developmental methodology selected have quality-control processes integrated into the development methodology? | ||
3. | Does the development methodology have both entrance and exit criteria? | ||
4. | Will management require compliance to the developmental methodology selected? | ||
5. | Does the developmental methodology selected have the appropriate management checkpoints so that go/no go decisions can be made at those checkpoints? | ||
Specifying Requirements | |||
1. | Is there a standard for requirements that definitively defines the attributes of a requirement? | ||
2. | If so, is that standard consistent with good practices and industry standards for requirement definition? | ||
3. | Are there enterprise-wide requirements, such as security, privacy, and control, that will be incorporated into all software projects? | ||
4. | Is there a process that will trace requirements from the requirements phase through implementation of the software project? | ||
5. | Is there a process in place that states that the requirements-definition phase of software development will not be complete until someone attests that the requirements are testable? | ||
Maturity of the IT Processes | |||
1. | Does your organization have all of the processes specified for CMMI level 2? | ||
2. | Does your organization have all of the processes specified for CMMI level 3? | ||
3. | Does your organization have all of the processes specified for CMMI level 4? | ||
4. | Does your organization have all of the processes specified for CMMI level 5? | ||
5. | Does your organization have a process in place that will continuously improve the processes specified in the CMMI maturity methodology? | ||
Competency of the Project Staff | |||
1. | Is the project staff competent in selecting the software development methodology used for building a specific software system? | ||
2. | Is the project staff competent in software testing? | ||
3. | Is the project staff competent in the procedures to be followed in developing software? | ||
4. | Is the software project staff competent in managing people? | ||
5. | Is the software project staff competent in managing projects? | ||
Experience of the Project Staff | |||
1. | Is the project staff experienced and knowledgeable in the business of the user? | ||
2. | Is the user associated with the project competent in the methodology used by the IT organization to develop software? | ||
3. | Will the users be involved throughout the entire software development methodology as needed, and will they be involved when needed? | ||
4. | Is the project staff experienced in using the selected software development methodology? | ||
5. | Have one or more members of the project staff been recognized for their experience and competency by being awarded a professional certification? | ||
Configuration-Management Controls | |||
1. | Does the configuration management consist of these four elements: Configuration identification Configuration control Configuration-status accounting Configuration audits | ||
2. | Are there internal configuration-control measures to control each configuration item? | ||
3. | Has a configuration-management plan been developed (or will one be) for the software project being developed? | ||
4. | Does the configuration-management system include a version control? | ||
5. | Does the configuration-management system restrict access to authorized individuals to protect data rights, security requirements, and data-status level? | ||
When answering the items on Work Paper 14-1, respond with either Yes or No. A Yes response means that the item is there and has shown some positive results. For example, an item in the competency of project staff asks whether the staff has competency in following and using the testing process. To answer Yes, a testing process must be in place. One or more individuals on the project staff must be competent in using that testing process, and experience in using the process has demonstrated that it is effective.
For each of the five components that affect the software development methodology, there are five questions. At this point, complete Work Paper 14-1 and answer the 30 questions on that work paper.
Upon completion of the self-assessment on Work Paper 14-1, post the results to Work Paper 14-2 (“Analysis Footprint of the Impact of the Software Development Methodology on Testing”). Total the number of Yes responses for each component. Put a dot on the line (for each of the six components) that represents the total number of Yes responses for that component. For example, if you answered Yes three times for the type of software development processes, place a dot on line number 3 for the type of process.
After you have posted all six scores to Work Paper 14-2, connect the six dots. You now have the footprint of the impact of the software development methodology on testing.
The footprint shows which of the components have the most impact on software testing. Impact is represented by the items that could not be answered Yes. In the example of posting the type of process, three Yes answers were presumed. The impact would be the two No responses.
Ideally, the footprint would be the number 5 line. Any time the footprint is not pushed out to the 5 ring, there is a negative impact from the components of the software development methodology.
Noting a negative impact, the software testing group has two options. One is to eliminate the item with the “No” response; the second is to minimize the negative impact. For example, if the project team does not have anyone competent in software testing, that can be addressed by adding someone to the team who has testing competency. The second way to address a weakness is to increase the amount of testing resources or extend the schedule.
This chapter had two major objectives: to explain to testers how attributes of the software development methodology can impact the resources and schedule needed for effective and efficient software testing and to provide an instrument to measure that impact so testers could make necessary adjustments to resources and schedules. Note that after the self-assessment is has been completed, an experienced tester must analyze the results of that self-assessment and make realistic adjustments to the budgeted testing resources and schedule.