The added value of testing lies mainly in the detection and reporting of a system’s deviations from its requirements. To be able to correct a deviation, development requires as much detailed information as possible about symptoms, reproduction, and consequences of abnormal system behavior. The test process must therefore place particular focus on accurate documentation and tracking of deviations. Part of this process is a risk estimation of the consequences of defects remaining in the system. This chapter explains the requirements of the documentation as well as categorization techniques and the prioritization and status tracking of deviations.
Generally speaking, all software contains defects. A defect is a widely used synonym for a software system’s deviation or divergence of actual from expected behavior. However, it is advisable to establish a more accurate distinction between the different stages in the development chain of deviations (see also [Spillner 07]):
An error is an erroneous act of a person or group causing a defect in the system.
Error, defect, failure
A defect is a fault in the system that, with a certain probability, will lead to a failure.
A failure is the visible deviation of a system’s operational behavior from the requirement specifications.
According to these definitions, testers do not detect defects, they find failures. Finding out which defect caused a particular observed failure must be considered a separate and subsequent activity in which test and development must cooperate. In this context, people typically speak of debugging.
Different failures may well be traced back to the same defect, while it is also possible that defects may remain in the system and not lead to a failure.
Consistent treatment of defects and change proposals
“It’s not a bug—it’s a feature.” Probably every programmer and tester knows this phrase expressing the differences of opinion that may well exist between development, test, and users as to whether or not a particular system behavior is to be seen as a failure, a tester’s wrong interpretation of the requirement documents, or an (justified or unjustified) enhancement or change request. A neutral, generic term for all these types is the term “deviation” (of a system’s behavior from the user’s or tester’s expectations; common synonyms are “anomaly”, “incident”).
Document deviations in testware, too.
Incident reports are mostly based on differences between the specification of a particular test case’s expected result and the actual behavior of the tested system. The reasons for such differences can lie in abnormal system behavior but also in an inaccurate test case specification. The correct answer often becomes clear only during the ensuing resolution activities. Many projects use a common management system to deal with incident reports written against the tested system and incident reports written against the testware; i.e. test plan, test specifications, and test automation.
The people who detect anomalies are normally not the same as those who analyze and correct them (see section 8.3.1). Because of this situation, detected anomalies require careful documentation and clear communication.
First, when communicating an incident report it makes no real difference whether it is documented and communicated via email, spreadsheet, or a specialized database (deviation management system). However, in order to support efficient treatment, the documentation of an incident must meet certain requirements independent of the medium used:
The description of the incident must allow simple and accurate reproduction of the failure:
Incident documentation requirements
• The test run and required input parameters are to be succinctly described so that the test run can be reproduced. Ideally, the incident report refers directly to a test case specification and/or an automated test procedure based on which development can reproduce the failure.
• The report shall address only one single, clearly defined problem and not a combination of several anomalies.
• The environment in which the test was executed must be clearly documented—for example, the version of the test object, platform, tested component, necessary reference data, etc.
It must be possible to prioritize a number of anomalies relative to each other in order to resolve them according to their assigned priority. To do so, the presumed failure potential must be evaluated from different perspectives, such as, for instance, from both the end user’s and the tester’s point of view.
Since typically more than one person is involved in the resolution of an incident, efficient control must be supported. This can be done by assigning a unique status to the report and by assigning people responsible for debugging, defect resolution, and retesting (see section 8.3).
Date and time the incident was detected is documented. For one thing, time may well be a relevant aspect regarding the cause of a failure; on the other hand, it allows some helpful analyses based on the total number of reported failures.
These requirements necessitate the use of a fixed format for communicating anomalies. Depending on the reporting method used, this may be in form of an e-mail, document template, or database scheme. Completing his report, the author must take sufficient care; he can do this best by putting himself into the position of those who have to read and evaluate his report later on.
Some basic rules regarding incident reports
It should not be forgotten that each incident report carries a certain political and emotional potential as in most cases it (implicitly) criticizes someone else’s work (namely, that of the affected software developer).
Dogmatism, malice, and displaced humor are inappropriate. Chapter 10 provides further advice on how appropriate forms of communication can help to avoid unnecessary disputes or putting up mental barriers.
Table 8-1 illustrates an incident report template that, due to its different attributes, satisfies the above mentioned requirements (see also [Spillner 07], section 6.6.2).
Table 8–1 Sample reporting template
Identification |
|
Number |
Incremental, unique number of report |
Test object |
Identifier of the test object (system or impacted component or components) |
Version |
Exact version identifier (tested build) of the test object |
Platform |
Identification of the hardware/software platform or test environment where the problem occurred |
Reporting person ID |
Unique identification of the person who detected the incident |
Responsible corrector ID (developer) |
Unique identification of the person responsible for correcting the defect |
Date of occurrence |
Date and time the incident was first observed |
Classification1 |
|
Status |
Current processing status of the report2, possibly historical Information allowing tracking of status changes with corresponding change dates |
Class |
Severity (e.g., crash, malfunction, deficiency, nice to have, change request) |
Priority |
Urgency of defect removal |
Requirement(s) |
Reference(s) to the requirement(s) violated or not fulfilled by the incident |
Source |
Phase of assumed defect injection: coding, documentation, design, etc. |
Problem description |
|
Test case |
Test case reference (identification) or direct description of the steps necessary for the reproduction of the failure |
Problem/symptom |
Accurate description of the incident stating the differences between target and actual behavior |
Comments |
Comments by involved staff on the report; possibly historical information |
Correction |
Details concerning correction measures |
References |
References to other, related incident reports; if required, additional references to background information, etc. |
1. For a more detailed classification scheme, see section 8.4.
2. See section 8.3.2.
In principle, each incident goes through the following life cycle: the anomaly is detected, documented, analyzed and corrected. Alternatively, the incident report is discarded or rejected because it is thought to be unsubstantiated.
The incident life cycle
Different people with different roles participate in the correction process. In order to coordinate these roles and to ensure orderly communication, it is necessary to define a deviation management process.
The most important components of such a management process are a suitable incident status model, an adequate workflow for the execution of status changes, and a definition of the roles and responsibilities pertaining to incidents within the status model.
The following roles and functions contribute to incident resolution. These roles have partly overlapping and partly contradictory interests in the resolution of incidents:
The tester reports incidents, documenting them following a well-defined pattern of input data necessary for analysis and management. He performs retesting after defect correction.
Interests of tester and test manager
The test manager leads the test group and organizes the deviation management, tracking the incident occurrence and the correction measures and using this as a basis for the planning and control of his test activities.
Both tester and test manager are interested in quick defect resolution, particularly of those potential “showstoppers” that might jeopardize or even stop the test activities altogether.
The developer analyzes the incident reports, perhaps jointly with the tester, and resolves defects. (Since the cause of a defect does not always lie in the system’s implementation phase but can also lie in the requirements or design phases, the term developer applies to programmers, designers, and requirements specifiers / requirements modelers.) The developer also implements approved change requests.
Interests of developers and development leader
The development leader is in charge of the development project and leads the development team; he is also responsible for the success of the defect resolution process. He estimates and is accountable for the effort required to perform the necessary resolution activities.
In order to reduce the stack, developers and the development leader are interested in correcting failures first whose resolution requires the least effort.
The product manager is responsible for product developed in the project. He approves change requests and is responsible for the prioritization of the correction activities.
It is in the product manager’s interest to achieve the highest possible customer satisfaction with the least possible cost. For this reason, it may be that in some cases change requests are given priority over defect resolution.
In some project situations, especially where individual software is being developed, the customer may be closely involved in the development process, evaluating new incidents with regard to their relevance for the operational use of the system. He may even report incidents himself that in many cases turn out to be change requests.
The interests of the customer
The customer expects a high-quality product, i.e., one that shows as little behavioral deviation from the expected quality profile as possible.
In order to balance these different and partly contradictory interests, the following board is established in a deviation management process:
The Change Control Board (CCB) regularly tracks the progress of deviation detection and resolution and decides on prioritization of defect resolution and change requests. Typically, this body is made up of the development leader, the test manager, and the product manager. The (sub)project leader, customer representative, or end user may also be part of it.
Change Control Board
The better and more qualified the information is on reported incidents, the easier the task is for the CCB. [IEEE 1044] (section 8.4) provides a lot of classifiable information. How information is used in concrete projects is up to the organization. The test manager in charge or the CCB must make a reasonable decision.
It also makes sense to introduce suitable defect metrics to be able to support CCB decisions with quantitative data. Chapter 11 provides some examples.
Figure 8-1 ([Spillner 07, section 6.6.4]) shows a model including incident statuses, events that lead to status changes, and the roles allowed to perform status changes:
Figure 8–1 Incident status model
The status model needs to be adjusted to satisfy organizational requirements, for instance by introducing the following:
A different number of final states—e.g., rejected, deferred to next “minor” release, deferred to next “major” release, deferred to release xy
A different number of “analyses states”, which help development show different workflow phases from reception up to the provision of a correction
Different roles and customized rights for individual roles—for instance, additional rights for the test manager to change the report status to “closed”
The transfer of activities of individual roles to the CCB, together with rules stating criteria based on which the CCB can make decisions regarding status changes
Example of an incident report with history data
Table 8.2 illustrates an incident report from VSR system test. Based on data recorded in the standard incident report template (see section 8.2), developers and tester process the report, documenting the different processing steps:
Table 8–2 Sample incident report
Identification |
||||
Number |
VSR-Systest 03347 |
|||
Test object |
VSR Component DreamCar |
|||
Version |
Build 02.013–04.27.2006 |
|||
Platform |
Windows XP–Standard-Client-Configuration |
|||
Reporting person |
G. Myers |
|||
Responsible corrector (Developer) |
K. Beck |
|||
Date of occurrence/reporting |
04.30.2006 10:17 |
|||
Classification |
||||
Status |
Status |
Date |
Set by |
Comment |
closed |
05.14.2006 |
G. Myers |
successful re-test |
|
test |
05.12.2006 |
K. Beck |
Recorrected check |
|
failure |
05.06.2006 |
G. Myers |
Discount = 100% still possible |
|
test |
05.04.2006 |
K. Beck |
Checking function corrected |
|
correction |
05.03.2006 |
K. Beck |
Correction in callback function |
|
analysis |
05.02.2006 |
K. Beck |
source code review |
|
analysis |
04.30.2006 |
CCB |
class 3->2 |
|
new |
04.30.2006 |
G. Myers |
|
|
Class |
2 (critical malfunction) |
|||
Priority |
2 (correction prior to release) |
|||
Requirements |
CC-Sys-CalcPrice-01/037 rebates |
|||
Source |
Validation procedures for rebates entry |
|||
Problem description |
||||
Test case |
TC-Price Calculation/0313 rebated purchase price |
|||
Problem / symptom |
Irrespective of the selected model, special edition, and accessories, it is possible to enter a discount of ≥ 100%. As a result, a negative purchase price is displayed. |
|||
Comments |
CCB, 04.30.2006: high classification as wrong calculation will leave an extremely bad impression with the customer |
Reporting template, storage, and management process are best implemented in form of a database—i.e., a so-called deviation management system (or a deviation or defect database1)—to obtain additional efficiency gains and benefits over simple “pen to paper” or spreadsheet solutions:
Deviation management systems
Users and user-assigned roles can be granted defined read and write access rights to ensure observance of the status model. Especially in chaotic projects, this will help control the workflow.
In case of status changes, the system can automatically inform affected people (e.g., via e-mail). This way, for instance, the tester will be automatically informed if an incident that he reported earlier is up for retest, thus saving him effort and time because he does not have to look for it in the reporting list.
It is possible to connect the deviation management system to other tools used in the software development process via suitable interfaces, which will enable traceability and provide considerable process support:
• An interface to a test management system (link between test cases and anomalies detected by them) will make typical test management tasks a lot easier for the test manager—for example, allowing him during retest planning to filter for all test cases linked to incidents that are reported closed.
• An interface to the configuration management system (linking deviations to changes in documents and source files that are necessary for the resolution of the problem) supports traceability during a development project’s consolidation phase. It may, for example, be required that in case of newly checked-in source code, the configuration management system must refer to a deviation addressed by the code changes.
• An interface to a requirements management system (relating deviations to requirements, often implemented by linking requirements to test cases in the test management system and linking the same test cases to deviations) allows a direct evaluation of the current product quality based on the requirements. The number of requirements in the system with status “untested,” “successfully tested,” or “tested with deviations” can thus be determined.
Reporting and statistics functionality allow consolidation and visualization of data, facilitating the evaluation of product quality and the indication of risk and improvement potentials.
Sample statistics and their application
The sample statistics in figures 8-2 and 8-3 are derived from the VSR system test deviation management system, showing the current state of the database at the end of the third system test cycle.
Figure 8–3 Reporting according to correction priority
Test manager and development leader discuss the statistics generated out of the deviation management system. Based on the above diagrams, in the final report of the third system test cycle, they make the following observations:
The diagram relating to the status of the reports (see figure 8-2) shows that a large number of open reports have been newly recorded but not yet processed. The test manager voices his concern that this carries a high risk and asks the development leader why so many reports have been left unattended by the development team.
The development leader sees in the diagram on correction priority (see figure 8-3) that his team needs to cope with a large number of urgent defect correction activities. More than half of all reports must be processed prior to the next system test cycle (priorities 1 and 2) so testing won’t be jeopardized. This hardly leaves any room for the analysis of newly registered reports.
Both deliberate on how to improve the situation. The development leader could provide the team with some additional staff to remove the bottleneck but requires some more information on where to put them to be most efficient. To answer this question, the test manager creates another statistic, filtering the database for the newly created reports in the third test cycle and categorizing the reports according to the impacted VSR components (see figure 8-4).
Figure 8–4 Reporting according to impacted component
The evaluation by components indicates that the majority of the new defects are localized in the two components DreamCar and VSR interface.
Consequently, the development leader decides to provide additional developers for the two components.
The test manager, on the other hand, updates his test planning for the fourth system test cycle, intensifying testing of these two components following the rule of thumb “If you’ve already found lots of failures, expect to have missed a lot more”.
As the example illustrates, a database-supported deviation management system is a very useful instrument for controlling a project. In order to establish such a system, a well-organized approach is needed that may well take a long time to realize (see chapter 12). In practice, we can see that with regard to deviation management, organizations often adopt an evolutionary improvement approach that goes hand in hand with other process improvement activities and has a sustained and positive impact on the efficiency of the test processes and on the achievable product quality:
Step by step from bits of paper toward a deviation management system
People often start reporting incidents or anomalies rather informally via e-mail. There is no control over the content and over processing the report. As a result, analysis, prioritization, and retesting are time-consuming and inaccurate.
As a next step, most organizations introduce a reporting template for their e-mail system. A project agreement is drawn up saying that in the future, copies of new reports are to be sent to the project or (if there is one) test manager. This will considerably improve traceability. The project or test manager can start with prioritizing, ensure that reports get actioned, and gain an overview of project progress based on simple statistics.
In the course of these activities, most managers start organizing storage of the reports—for example, in a directory structure in the file system or in a central table in a spreadsheet program. The latter makes report-related information machine readable so that it can be analyzed or evaluated in the form of status reports and statistics. This is the preliminary stage to a proper deviation management system.
The last step in this process is transferring all the information held in tables into a proprietary or commercial database system. Actioning of reports is now controlled by the status and rights system incorporated in the deviation management system. Sophisticated reporting and statistics functions, in combination with interfaces to the requirements and configuration management systems, open up the way to develop even further potentials.
Often the deviation management process is the first or the only (halfway) formally defined process in an organization and the tools used are the only available process-related tools. Gradual, step-by-step improvements in this area are often seen as a precedent for other improvements in the overall project, leading to medium-term improvements in other areas such as requirements or project management.
The [IEEE 1044] standard describes a classification of anomalies, the documentation of the attributes of an anomaly, and the associated process. This standard serves as a good basis and is briefly explained in the following section.
Standard techniques for the classification of anomalies (incidents)
For classification, IEEE 1044 defines a sequence of four steps:
1. Recognition
2. Investigation
3. Action
4. Disposition
Essentially, each, of these four steps consists of the execution of three parallel activities:
Recording
Classifying
Identifying impacts
Applied on each step, the three activities classify and document the features of the incident from different perspectives.
The focus of this workflow is clearly on report classification. The standard does not describe a complete management process and does not, for instance, consider how to deal with rejected reports or ineffective corrections. A reasonable management process is based on a status model similar to the one illustrated in figure 8.1 above, integrating the classification steps according to IEEE 1044 (see figure 8-5).
It is important to classify in each step; i.e., to classify not only the severity and impact of an anomaly defined during analysis but also the circumstances that have led to its detection, the necessary resolution actions, and the report’s disposition after closure.
Figure 8–5 Classification according to IEEE 1044
Classification is done using a selection of predefined values, so-called “classifications”, and attributes of assigned “categories”. The standard considers some of these attributes mandatory and others optional. Each classification and each category is clearly defined to ensure uniform recording and processing of anomalies even in large projects. Classifications and categories are combined in “classification schemes” and applied to each of the classification process steps.
The category recognition classification scheme – project activity serves as a good example. It is used to determine the activity (e.g., review or test) by which an incident is detected (see figure 8-6).
Figure 8–6 Example: Recognition classification scheme-project activity
RR100 implies that this scheme is to be applied to the recognition step of the anomaly. “Mandatory” in each line indicates that in order for the classification process to comply with the standard, the category as a whole and each of the possible activities in the associated classification table must be applied.
Based on this scheme, an anomaly found during system test is assigned code “RR160”, whereas an anomaly detected during review has “RR120” assigned to it.
Besides categories, the standard also defines so-called “supporting data items”, which can be applied in the description of the anomaly and the environment in which it occurred. This can be exemplified by the “investigation supporting data items” in figure 8-7, which support the anomaly analysis step.
Figure 8–7 Example: Investigation supporting data item
Acknowledgement |
Verification |
Date received |
Source of incident |
Report number assigned |
Data from recognition process |
Investigator |
|
Name |
|
Code or functional area |
|
E-mail address |
|
Phone number |
|
Estimated start date of investigation |
|
Estimated complete date of investigation |
|
Actual start date of investigation |
|
Actual complete date of investigation |
|
Person hours |
|
Date receipt acknowledgement |
|
Documents used in investigation |
|
Name |
|
ID number |
|
Revision |
|
During the analysis step this table is used to enter organizational pieces of information. They can serve a variety of purposes:
To support subsequent classification steps as well as the actual treatment of the anomaly. Stating the “source of incident” provides direct help to the person assigned to resolve the anomaly.
To help in analyzing data in different ways, keeping process improvement in mind. For example, if estimated and actual times used for the analysis step regularly show substantial deviations, the issue may be addressed by introducing improved estimation models or more highly qualified staff for the job.
To have better traceability of the different processing steps. For example, should one of the people in the processing chain drop out for some reason, it will be a lot easier for someone else to continue processing a particular step. In the example in figure 8.7, the information “documents used in investigation” provides an overview of the documents that have already been checked and those that still need to be analyzed.
Each project member can report anomalies. The following activities of the recognition step are performed as soon as the anomaly is encountered:
Recognition
Documentation by collecting recognition supporting data items. These items comprise all data related to the environment in which the anomaly was detected. Data such as hardware and software environment and test supporting software (e.g., test automation tools) and also the contact ID of the reporting person are recorded here, together with the time of occurrence and the documentation used during the operation of the system. Note that the standard considers the attributes list it provides as optional and incomplete.
Classification via selection of values out of the recognition classification scheme. It consists of six categories (three of which are mandatory) that are supposed to help provide accurate reporting—for example, project phase, symptom, and system status after the anomaly has occurred.
Identifying the impact by means of the impact classification scheme and the impact supporting data items. This data is to be used by the person who encountered the anomaly to report his observations (or assumptions) on the impacts of the incident: urgency and severity of the anomaly, effects on product quality, customer value, safety, project cost, risk, and project schedule.
Each reported anomaly must be investigated. The aim of the investigation is to be able to evaluate the anomaly. Usually this step is carried out by development, but important information may also come from testers or other project members.
Analysis (investigation)
Documentation: The investigation supporting data items are collected here to confirm the existence and reproducibility of the anomaly and to identify possible workarounds and correction measures. The standard considers this list, too, as optional and incomplete.
Classification: Selection of suitable data out of the investigation classification scheme to describe the actual cause, the impacted documents, and the nature of the anomaly. In addition, classifications made in the first step are checked and, if necessary, corrected.
Identification of the impact: The assumptions made on the impact of the anomaly documented in the first step are checked and, if necessary, corrected. The tables used in the first step are used as a basis.
Based on the analysis results, resolution actions are planned. This step may comprise both the direct resolution of the anomaly itself and process improvement actions designed to avoid further, similar incidents. In this step, development is primarily involved, but other roles may also be involved, such as the product manager or the quality assurance department.
Resolution (action)
Documentation: To begin with, action supporting data items are collected; for example, planned resolution date and product delivery status, a description of the resolution activities, and the names or functions of people responsible for correction or retesting.
Classification: This is done using the action classification scheme, describing the type of the action (e.g., code or documentation changes) and their priority (ranging from “immediate” to “no resolution”). Optionally, the scheme also describes additional countermeasures of a more strategic nature, such as process improvement and research activities.
Identifying the impact: Again, impact categories documented in the previous steps are reviewed and, if necessary, updated.
In the disposition phase—i.e., after all resolution activities have been completed or at least after long-term correction actions have been initiated—the removal of the anomaly is to be documented. The main actors involved in this step are usually testers, in some cases support staff, and the product manager.
Disposition
Documentation: The disposition supporting data items, for instance, document that the customer has been informed about the resolution. They are also used to document verification (e.g., retest) results.
Classification: The disposition classification scheme documents the final version of the anomaly report (e.g., “resolution completed” or “duplicate problem”).
Identifying the impact: A concluding consideration is made and previously documented impacts are reviewed and updated, if necessary—again based on the same reference tables as in the previous steps.
The sample tables and the descriptions of the individual categories show that the standard is very comprehensive, covering the entire software life cycle. In practice, however, it will not be applied in its entirety but rather tailored and reduced to the concrete needs in the project in the following manner:
By means of omission of redundant attributes or categories
By integrating the steps of the classification processes into one’s own deviation management process
Generally, the reasons for this so-called “tailoring” are found during a cost-benefit analysis—more detailed data collected during incident treatment means more data is available in subsequent process steps; however, it also amounts to more effort required for collection and maintenance. The following list includes some of the factors that may possibly influence the standard’s scope of use:
Influencing factors
The intended area of influence: Is the process to be applied across the different test levels, projects, or even the entire organization? Is it to be restricted to development and test or is it also to be applied in product support?
The evaluation of collected data for process improvement purposes: Accumulation of particular incident categories is an indicator that can be used for focusing on and removal of weaknesses in the development and test process.
The applicability of categories in the concrete project environment. Supporting data items such as databases and firmware only make sense in particular settings.
Documentation and obligation of accountability required by customers and/or licensing authorities. These may possibly require more detailed data than what is needed in the development department.
Supplement IEEE 1044.1 to the standard provides valuable support for the tailoring and customization of standards to suit one’s own requirements, providing the following:
Tailoring support
Support for the decision if, and within which scope, the standard can and should be applied in one’s own organizational setting
Detailed explanations and examples of categories and classifications
Additional supporting data items not mentioned in the standard
Guidelines for correct tailoring; i.e., particulars and examples of standard-compliant and noncompliant adaptations and omissions of categories and classifications
A sample status model for anomalies (similar to the one in section 8.3.2)
Methods to map the standard into commercial and in-house developed deviation management systems
Examples of incident metrics, their analysis, and application to project control, product evaluation, and process improvement
It is easy to draw a line from the contents of this supplement right back the beginning of this chapter. Both the sample incident reporting template (section 8.2) and the status model (section 8.3.2) were derived from, though not necessarily in compliance with, the IEEE 1044/1044.1 standard to be applied in software development and testing during development at the project level. Both have been successfully tried and tested in many projects.
These examples may serve as a starting point to readers establishing their own deviation management system. Nevertheless, we would recommend that the standard be used to critically scrutinize the examples for applicability.
On the one hand, anomalies require consistent and complete documentation to provide maximum support for their evaluation and resolution; on the other hand, they require a management process that reflects the life cycle of the incident reports and coordinates the roles involved in the processing of the deviation.
In practice, a large variety of different systems is used for incident reporting, ranging from simple management techniques to sophisticated deviation management tools, from the simple exchange of e-mails to deviation management systems spanning an entire organization with comprehensive databases that can be used for a wide range of different statistics. If such systems are linked to other corporate configuration, test, and requirements management systems, they can considerably increase efficiency in the development process.
To provide for a common basis for this historically grown heterogeneity, the IEEE defined a standard for the classification of anomalies (IEEE 1044), containing mandatory and optional attributes of anomalies as well as a description of a generic processing workflow.
IEEE 1044 arranges the documentation, monitoring, and classification of deviations into four process steps:
• Recognition
• Investigation
• Action
• Disposition
The IEEE 1044 standard is very comprehensive and allows application in test not only during development but also throughout a system’s entire life cycle. For this reason, however, the standard in most cases requires customization to suit the concrete requirements in a particular environment. A supplementary document (IEEE 1044.1) provides valuable information on this issue.