Static investigations like reviews and tool-supported analysis of code and documents can be used very successfully for improving quality. This chapter presents the possibilities and techniques.
An often-underestimated examination method is the so-called static test, often named static analysis. Opposite to dynamic testing (see chapter 5), the test object is not provided with test data and executed but rather analyzed. This can be done using one or more persons for an intensive investigation or through the use of tools. Such an investigation can be used for all documents relevant for software development and maintenance. Tool-supported static analysis is only possible for documents with a formal structure.
The goal of examination is to find defects and deviations from the existing specifications, standards to comply with, or even the project plan. An additional benefit of the results of these examinations is optimizing the development process. The basic idea is defect prevention: defects and deviations should be recognized as early as possible before they have any effect in the further development process where they would result in expensive rework.
Systematic use of the human ability to think and analyze
Reviews apply the human analytical capabilities to check and evaluate complex issues. Intensive reading and trying to understand the examined documents is the key.
There are different techniques for checking documents. They differ regarding the intensity, formality, necessary resources (staff and time), and goals.
In the following sections, the different techniques are explained in more detail. Unfortunately, there is no uniform terminology concerning static analysis techniques. The terms used in this chapter are similar to the terms in the ISTQB syllabus and [IEEE 1028] (see the glossary in the appendix). Detailed descriptions can be found in [Freedman 90] and [Gilb 96].
Review is a common generic term for all the different static analysis techniques people perform as well as the term for a specific document examination technique.
Another term, often used with the same meaning, is →inspection. However, inspection is usually defined as a special, formal review using data collection and special rules [Fagan 76], [IEEE 1028], [Gilb 96]. All documents can be subjected to a review or an inspection, such as, for example, contracts, requirements definitions, design specifications, program code, test plans, and manuals. Often, reviews provide the only possibility to check the semantics of a document. Reviews rely on the colleagues of the author to provide mutual feedback. Because of this, they are also called peer reviews.
A means for quality assurance
Reviews are an efficient means to assure the quality of the examined documents. Ideally, they should be performed as soon as possible after a document is completed to find mistakes and inconsistencies early. The verifying examinations at the end of a phase in the general V-model normally use reviews (so-called phase exit reviews). Eliminating defects and inconsistencies leads to improved document quality and positively influences the whole development process because development is continued with documents that have fewer or even no defects.
Positive effects
In addition to defect reduction, reviews have further positive effects:
Potential problem
The following problem can arise: In a badly moderated review session, the author may get into a psychologically difficult situation, feeling that he as a person and not the document is subject to critical scrutiny. Motivation to subject documents to a review will thus be destroyed. Concretely expressing the review objective, which is improving the document, may be helpful. One book [Freedman 90] extensively discusses how to solve problems with reviews.
Reviews costs and savings
The costs caused by reviews are estimated to be 10–15% of the development budget [Gilb 96, pg. 27]. The costs include the activities of the review process itself, analyzing the review results, and the effort put toward implementing them for process improvement. Savings are estimated to be about 14–25% [Bush 90]. The extra effort for the reviews themselves is included in this calculation.
If reviews are systematically used and efficiently run, more than 70% of the defects in a document can be found and repaired before they are unknowingly inherited by the next work steps [Gilb 96]. Considering that the costs for defect removal substantially increase in later development steps, it is plausible that defect cost in development is reduced by 75% and more.
The following factors are decisive for success when using reviews (as suggested by [IEEE 1028]):
The term review describes a whole group of static examinations. Section 4.1.5 describes the different types of reviews. The process underlying all reviews is briefly described here in accordance with the IEEE Standard for Software Reviews [IEEE 1028].
A review requires six work steps: planning, kick-off, individual preparation, review meeting, rework, and follow-up.
Reviews need planning
Early, during overall planning, management must decide which documents in the software development process shall be subject to which review technique. The estimated effort must be included in the project plans. Several analyses show optimal checking time for reviewing documents and code [Gilb 96]. During planning of the individual review, the review leader selects technically competent staff and assembles a review team. In cooperation with the author of the document to be reviewed, she makes sure that the document is in a reviewable state, i.e., it is complete enough and reasonably finished. In formal reviews, entry criteria (and the corresponding exit criteria) may be set. A review should continue only after any available entry criteria has been checked.
Different perspectives increase the effect
A review is, in most cases, more successful when the examined document is read from different viewpoints or when each person checks only particular aspects. The viewpoints or aspects to be used should be determined during review planning. A review might not involve the whole document. Parts of the document in which defects constitute a high risk could be selected. A document may also be sampled only to make a conclusion about the general quality of the document.
If a kick-off meeting is necessary, the place and time must be agreed upon.
The kick-off (or overview) serves to provide those involved in the review with all of the necessary information. This can happen through a written invitation or a meeting when the review team is organized. The purpose is sharing information about the document to be reviewed (the review object) and the significance and the objective of the planned review. If the people involved are not familiar with the domain or application area of the review object, then a short introduction to the material may be arranged, and a description of how it fits into the application or environment may be provided.
Higher-level documents are necessary
In addition to the review object, those involved must have access to other documents. These include the documents that help to decide if a particular statement is wrong or correct. The review is done against these documents (e.g., requirements specification, design, guidelines, or standards). Such documents are also called base documents or baselines. Furthermore, review criteria (for example, checklists) are very useful for supporting a structured process.
For more formal reviews, the entry criteria might be checked. If entry criteria are not met, the review should be canceled, saving the organization time that would otherwise be wasted reviewing material that may be “immature,” i.e., not good enough.
Intensive study of the review object
The members of the review team must prepare individually for the review meeting. A successful review meeting is only possible with adequate preparation.
The reviewers intensively study the review object and check it against the documents given as a basis for it as well as against their own experience. They note deficiencies (even any potential defects), questions, or comments.
A review leader or →moderator leads the review meeting. Moderator and participants should behave diplomatically (not be aggressive with each other) and contribute to the review in the best possible way.
The review leader must ensure that all experts will be able to express their opinion knowing that the product will be evaluated and not the author. Conflicts should be prevented. If this is not possible, a solution for the situation should be found.
Usually, the review meeting has a fixed time limit. The objective is to decide if the review object has met the requirements and complies with the standards and to find defects. The result is a recommendation to accept, repair, or rewrite the document. All the reviewers should agree upon the findings and the overall result.
Rules for review meetings
Here are some general rules for a review meeting:1
Protocol and summary of results
The protocol contains a list of the issues/findings that were discussed during the meeting. An additional review summary report should collect all important data about the review itself, i.e., the review object, the people involved, their roles (see section 4.1.4), a short summary of the most important issues, and the result of the review with the recommendation of the reviewers. In a more formal review, the fulfillment of formal exit criteria may be documented. If there was no physical meeting and, for example, electronic communication was used instead, there should definitely be a protocol.
The manager decides whether to follow the recommendation or do something else. A different decision is, however, the sole responsibility of the manager. Usually, the author will eliminate the defects on the basis of the review results and rework the document. More formal reviews additionally require updating the defect status of every single found defect.
The proper correction of defects must be followed up, usually by the manager, moderator, or someone especially assigned this responsibility.
Second review
If the result of the first review was not acceptable, another review should be scheduled. The process described here can be rerun, but usually it is done in an abbreviated manner, checking only changed areas.
The review meetings and their results should then be thoroughly evaluated to improve the review process, to adapt the used guidelines and checklists to the specific conditions, and to keep them up-to-date. To achieve this, it is necessary to collect and evaluate measurement data.
Find and fix deficiencies in the software development process
Recurring, or frequently occurring, defect types point to deficiencies in the software development process or lack of technical knowledge of the people involved. Necessary improvements of the development process should be planned and implemented. Such defect types should be included in the checklists. Training must compensate for lack of technical knowledge.
For more formal reviews, the final activity is checking the exit criteria. If they are met, the review is finished. Otherwise, it must be determined whether rework can be done or if the whole review was unsuccessful.
The description of the general approach included some information on roles and responsibilities. This section presents the people involved and their tasks.
Manager
The manager selects the objects to be reviewed, assigns the necessary resources, and selects the review team.
Representatives of the management level should not participate in review meetings because management might evaluate the qualifications of the author and not the document. This would inhibit a free discussion among the review participants. Another reason is that the manager often lacks the necessary detailed understanding of technical documents. In a review, the technical content is checked, and thus the manager would not be able to add valuable comments. Management reviews of project plans and the like are a different thing. In this case, knowledge of management principles is necessary.
Moderator
The moderator is responsible for executing the review. Planning, preparation, execution, rework, and follow-up should be done in such a way that the review objectives are achieved.
The moderator is responsible for collecting review data and issuing the review report.
This role is crucial for the success of the review. First and foremost, a moderator must be a good meeting leader, leading the meeting efficiently and in a diplomatic way. A moderator must be able to stop unnecessary discussions without offending the participants, to mediate when there are conflicting points of view, and be able to see “between the lines.” A moderator must be neutral and must not state his own opinion about the review object.
The author is the creator of the document that is the subject of a review. If several people have been involved in the creation, one person should be appointed to be responsible; this person assumes the role of the author. The author is responsible for the review object meeting its review entry criteria (i.e., that the document is reviewable) and for performing any rework required for meeting the review exit criteria.
It is important that the author does not interpret the issues raised on the document as personal criticism. The author must understand that a review is done only to help improve the quality of the product.
Reviewer
The reviewers, sometimes also called inspectors, are several (usually a maximum of five) technical experts that participate in the review meeting after necessary individual preparation.
They identify and describe problems in the review object. They should represent different viewpoints (for example, sponsor, requirements, design, code, safety, test). Only those viewpoints pertinent to the review of the product should be considered.
Some reviewers should be assigned specific review topics to ensure effective coverage. For example, one reviewer might focus on conformance with a specific standard, another on syntax. The manager should assign these roles when planning the review.
The reviewers should also label the good parts in the document. Insufficient or deficient parts of the review object must be labeled accordingly, and the deficiencies must be documented for the author in such a way that they can be corrected.
Recorder
The recorder (or scribe) shall document the issues (problems, action items, decisions, and recommendations) found by the review team.
The recorder must be able to record in a short and precise way, correctly capturing the essence of the discussion. This may not be easy because contributions are often not clearly or well expressed. Pragmatic reasons may make it meaningful to let the author be recorder. The author knows exactly how precisely and how detailed the contributions of the reviewers need to be recorded in order to have enough information for rework.
Reasons for less successful reviews
Reviews may fail to achieve their objectives due to several causes:
Detailed advice for solving these problems is described in [Freedman 90].
Two main groups of reviews can be distinguished depending on the review object to be examined:
Reviews in the second group are called →management reviews3 [IEEE 1028] or project reviews. Their objective is to analyze the project itself or the development process. For example, such a review determines if plans and rules are followed, if the necessary work tasks are executed, or the effectiveness of process improvements or changes.
The project as a whole and determining its current state are the objects of such a review. The state of the project is evaluated with respect to technical, economic, time, and management aspects.
Management reviews are often performed when reaching a milestone in the project, when completing a main phase in the software development process, or as a “postmortem” analysis to learn from the finished project.
In the following sections, the first group of reviews is described in more detail. We can distinguish between the following review types: →walkthrough, inspection, →technical review, and →informal review. In the descriptions, the focus is on the main differences between the particular review type and the basic review process (see section 4.1.3).
A walkthrough4 is a manual, informal review method with the purpose of finding defects, ambiguities, and problems in written documents. The author presents the document to the reviewers in a review meeting.
Educating an audience regarding a software product is mentioned in [IEEE 1028] as a further purpose of walkthroughs. Further objectives of walkthroughs are to improve the product, to discuss alternative implementations, and to evaluate conformance to standards and specifications.
The main emphasis of a walkthrough is the review meeting (without a time limit). There is less focus on preparation compared to the other types of reviews; it can even be omitted sometimes.5
Discussion of typical usage situations
In most cases, typical usage situations, also called scenarios, will be discussed. Test cases may even be “played through.” The reviewers try to find possible errors and defects by spontaneously asking questions.
Suitable for small development teams
The technique is useful for small teams of up to five persons. It does not require a lot of resources because preparation and follow-up are minor or sometimes not even required. The walkthrough is useful for checking “noncritical” documents.
The author chairs the meeting and therefore has a great amount of influence. This can have a detrimental effect on the result if the author impedes an intensive discussion of the critical parts of the review object.
The author is responsible for follow-up; there is no more checking.
Before the meeting the reviewers prepare, the results are written in a protocol, and someone other than the author records the findings. In practice there is a wide variation from informal to formal walkthroughs.
Objectives
The main objectives of a walkthrough are mutual learning, development of an understanding of the review object, and error detection.
Formal process
The inspection is the most formal review. It follows a formal, prescribed process. Every person involved, usually people who work directly with the author, has a defined role. Rules define the process. The reviewers use checklists containing criteria for checking the different aspects.
The goals are finding unclear items and possible defects, measuring review object quality, and improving the quality of the inspection process and the development process. The concrete objectives of each individual inspection are determined during planning. The inspectors (reviewers) prepare for only a specific number of aspects that will be examined. Before the inspection begins, the inspection object is formally checked with respect to entry criteria and reviewability. The inspectors prepare themselves using procedures or standards and checklists.
Traditionally, this method of reviewing has been called design inspection or code or software inspection. The name points to the documents that are subject to the inspection (see [Fagan 76]). However, inspections can be used for any document in which formal evaluation criteria exist.
Inspection meeting
A moderator leads the meeting. The inspection meeting follows this agenda:
Additional assessment of the development and inspection process
In an inspection, data are also collected for general quality assessment of the development process and the inspection process. Therefore, an inspection also serves to optimize the development process, in addition to assessing the inspected documents. The collected data are analyzed in order to find causes for weaknesses in the development process. After process improvement, comparing the collected data before the change to the current data checks the improvement effect.
Objective
The main objective of inspection is defect detection or, more precisely, the detection of defects causes and defects.
Does the review object fulfill its purpose?
In a technical review, the focus is compliance of the document with the specification, fitness for its intended purpose, and compliance to standards. During preparation, the reviewers check the review object with respect to the specified review criteria.
Technical experts as reviewers
The reviewers must be technically qualified. Some of them should not be project participants in order to avoid “project blindness.” Management does not participate. Basis for the review is only the “official” specification and the specified criteria for the review. The reviewers write down their comments and pass them to the moderator before the review meeting.7 The moderator (who ideally is properly trained) sorts these findings based on their presumed importance. During the review meeting, only selected remarks are discussed.
High preparation effort
Most of the effort is in preparation. The author does not normally attend the meeting. During the meeting, the recorder notes all the issues and prepares the final documentation of the results.
The review result must be approved unanimously by all involved reviewers and signed by everyone. Disagreement should be noted in the protocol. It is not the job of the review participants to decide on the consequences of the result; that is the responsibility of management. If the review is highly formalized, entry and exit criteria of the individual review steps may also be defined.
In practice, very different versions of the technical review are found, from a very informal to a strictly defined, formal process.
Objective
Discussion is expressly requested during a technical review. Alternative approaches should be considered and decisions made. The specialists may solve the technical issues. The conformity of the review object with its specifications and applicable standards can be assessed. Technical reviews can, of course, reveal errors and defects.
The informal review is a light version of a review. However, it more or less follows the general procedure for reviews (see section 4.1.3) in a simplified way. In most cases, the author initiates an informal review. Planning is restricted to choosing reviewers and asking them to deliver their remarks at a certain point in time. Often, there is no meeting or exchange of the findings. In such cases, the review is just a simple author-reader-cycle. The informal review is a kind of cross reading by one or more colleagues. The results need not be explicitly documented; a list of remarks or the revised document is in most cases enough. Pair programming, buddy testing, code swapping, and the like are types of informal review. The informal review is very common and highly accepted due to the minimal effort required.
Objective
An informal review involves relatively little effort and low costs. Discussion and exchange of information among colleagues are welcome “side effects” of the process.
Selecting the type of review
The type of review that should be used depends very much on how thorough the review needs to be and the effort that can be spent. It also depends on the project environment; we cannot give specific recommendations. The decision about what type of review is appropriate must be made on a case-by-case basis. Here are some questions and criteria that should help:
Testers as reviewers
It makes sense to use testers as reviewers. The reviewed documents are usually used as the test basis to design test cases. Testers know the documents early and they can design test cases early. By looking at documents from a testing point of view, testers may check new quality aspects, such as testability.
As we said in the beginning of the chapter, there are no uniform descriptions of the individual types of review. There is no clear boundary between the different review types, and the same terms are used with different meanings.
Company-specific reviews
Generally, it can be said that the types of reviews are very much determined by the organization that uses them. Reviews are tailored to the specific needs and requirements of a project. This has a positive influence on their efficiency.
A cooperative collaboration between the people involved in software development can be considered beneficial to quality. If people examine each other’s work results, defects and ambiguities can be revealed. From this point of view, pair programming, as suggested in →Extreme Programming, can be regarded as a permanent “two-person-review” [Beck 00].
With distributed project teams, it might be hard to organize review meetings. These days, reviews can be in the form of structured discussion by Internet, videoconferencing, telephone conference calls, etc.
The following factors are crucial for review success and must be considered:
Analysis without executing the program
The objective of static analysis is, as with reviews, to reveal defects or defect-prone parts in a document. However, in static analysis, tools do the analysis. For example, even spell checkers can be regarded as a form of →static analyzers because they find mistakes in documents and therefore contribute to quality improvement.
The term static analysis points to the fact that this form of checking does not involve an execution of the checked objects (of a program). An additional objective is to derive measurements, or metrics, in order to measure and prove the quality of the object.
Formal documents
The document to be analyzed must follow a certain formal structure in order to be checked by a tool. Static analysis makes sense only with the support of tools. Formal documents can note, for example, the technical requirements, the software architecture, or the software design. An example is the modeling of class diagrams in UML.8 Generated outputs in HTML9 or XML10 can also be subjected to tool-supported static analysis. Formal models developed during the design phases can also be analyzed and inconsistencies can be detected. Unfortunately, in practice, the program code is often the one and only formal document in software development that can be subjected to static analysis.
Developers typically use static analysis tools before or during component or integration testing to check if guidelines or programming conventions are adhered to. During integration testing, adherence to interface guidelines is analyzed.
Analysis tools often produce a long list of warnings and comments. In order to effectively and efficiently use the tools, the mass of generated information must be handled intelligently; for example, by configuring the tool. Otherwise, the tools might be avoided.
Static analysis and reviews
Static analysis and reviews are closely related. If a static analysis is performed before the review, a number of defects and deviations can be found and the number of the aspects to be checked in the review clearly decreases. Due to the fact that static analysis is tool supported, there is much less effort involved than in a review.
Not all defects can be found using static testing, though. Some defects become apparent only when the program is executed (that means at runtime) and cannot be recognized before. For example, if the value of the denominator in a division is stored in a variable, that variable can be assigned the value zero. This leads to a failure at runtime. In static analysis, this defect cannot easily be found, except for when the variable is assigned the value zero by a constant having zero as its value.
All possible paths through the operations may be analyzed, and the operation can be flagged as potentially dangerous. On the other hand, some inconsistencies and defect-prone areas in a program are difficult to find by dynamic testing. Detecting violation of programming standards or use of forbidden error-prone program constructs is possible only with static analysis (or reviews).
The compiler is an analysis tool
All compilers carry out a static analysis of the program text by checking that the correct syntax of the programming language is used. Most compilers provide additional information, which can be derived by static analysis (see section 4.2.1). In addition to compilers, there are other tools that are so-called analyzers. These are used for performing special analyses or groups of analyses.
The following defects and dangerous constructions can be detected by static analysis:
Finding security problems
Static analysis can be used to detect security problems. Many security holes occur because certain error-prone program constructs are used or necessary checks are not done. Examples are lack of buffer overflow protection and failing to check that input data may be out of bounds. Tools can find such deficiencies because they often search and analyze certain patterns.
Violation of the programming language syntax is detected by static analysis and reported as a fault or warning. Many compilers also generate further information and perform other checks:
The information is usually provided in the form of lists. A result reported as “suspicious” by the tool is not always a fault. Therefore, further investigation is necessary.
Compliance to conventions and standards can also be checked with tools. For example, tools can be used to check if a program follows programming regulations and standards. This way of checking takes little time and almost no personnel resources. In any case, only guidelines that can be verified by tools should be accepted in a project. Other regulations usually prove to be bureaucratic waste anyway. Furthermore, there often is an additional advantage: if the programmers know that the program code is checked for compliance to the programming guidelines, their willingness to work according to the guidelines is much higher than without an automatic test.
Checking the use of data
→Data flow analysis is another means to reveal defects. Here, the usage of data on →paths through the program code is checked. It is not always possible to decide if an issue is a defect. Instead, we speak of →anomalies, or data flow anomalies. An anomaly is an inconsistency that can lead to failure but does not necessarily do so. An anomaly may be flagged as a risk.
An example of a data flow anomaly is code that reads (uses) variables without previous initialization or code that doesn’t use the value of a variable at all. The analysis checks the usage of every single variable. The following three types of usage or states of variables are distinguished:
Data flow anomalies
We can distinguish three types of data flow anomalies:
Data flow anomalies are usually not that obvious
In this example, the anomalies are obvious. But it must be considered that between the particular statements that cause these anomalies there could be an arbitrary number of other statements. The anomalies would not be as obvious anymore and could easily be missed by a manual check such as, for example, a review. A tool for analyzing data flow can, however, detect the anomalies.
Not every anomaly leads directly to an incorrect behavior. For example, a du-anomaly does not always have direct effects; the program could still run properly. The question arises why this particular assignment is at this position in the program, just before the end of the block where the variable is valid. Usually, an exact examination of the program parts where trouble is indicated is worthwhile and further inconsistencies can be discovered.
Control flow graph
In figure 4-1, a program structure is represented as a control flow graph. In this directed graph, the statements of the program are represented with nodes. Sequences of statements are also represented with a single node because inside the sequence there can be no change in the course of program execution. If the first statement of the sequence is executed, the others are also executed.
Changes in the course of program execution are made by decisions, such as, for example, in IF statements. If the calculated value of the condition is true, then the program continues in the part that begins with THEN. If the condition is false, then the ELSE part is executed. Loops lead to previous statements, resulting in repeated execution of a part of the graph.
Control flow anomalies
Due to the clarity of the control flow graph, the sequences through the program can easily be understood and possible anomalies can be detected. These anomalies could be jumps out of a loop body or a program structure that has several exits. They may not necessarily lead to failure, but they are not in accordance with the principles of structured programming. It is assumed that the graph is not generated manually but that it is generated by a tool that guarantees an exact mapping of the program text to the graph.
If parts of the graph or the whole graph are very complex and the relations, as well as the course of events, are not understandable, then the program text should be revised, because complex sequence structures often bear a great risk of being wrong.
In addition to graphs, a tool can generate predecessor-successor tables that show how every statement is related to the other statements. If a statement does not have a predecessor, then this statement is unreachable (so-called dead code). Thus a defect or at least an anomaly is detected. The only exceptions are the first and last statements of a program: They can legally have no predecessor or successor. For programs with several entrance and/or exit points, the same applies.
Measuring of quality characteristics
In addition to the previously mentioned analyses, static analysis tools provide measurement values. Quality characteristics can be measured with measurement values, or metrics. The measured values must be checked, though, to see if they meet the specified requirements [ISO 9126]. An overview of currently used metrics can be found in [Fenton 91].
The definition of metrics for certain characteristics of software is based on the intent to gain a quantitative measure of software whose nature is abstract. Therefore, a metric can only provide statements concerning the one aspect that is examined, and the measurement values that are calculated are only interesting in comparison to numbers from other programs or program parts that are examined.
Cyclomatic number
In the following, we’ll take a closer look at a certain metric: the →cyclomatic number (McCabe number [McCabe 76]). The cyclomatic number measures the structural complexity of program code. The basis of this calculation is the control flow graph.
For a control flow graph (G) of a program or a program part, the cyclomatic number can be computed like this:11
v(G) = e - n + 2
v(G) |
= cyclomatic number of the graph G |
e |
= number of edges of the control flow graph |
n |
= number of nodes of the control flow graph |
The cyclomatic number gives information about the testing effort
The cyclomatic number can be used to estimate the testability and the maintainability of a particular program part. The cyclomatic number specifies the number of independent paths in the program part.12 If 100% branch coverage (see section 5.2.2) is intended, then all these independent paths of the control flow graph have to be executed at least once. Therefore, the cyclomatic number provides important information concerning the volume of the test.
Understanding a program is essential for its maintenance.
The higher the value of the cyclomatic number, the more difficult it is to understand the flow in a certain program part.
The cyclomatic number has been very much discussed since its publication. One of its drawbacks is that the complexity of the conditions, which lead to the selection of the control flow, is not taken into account. It does not matter for the calculation of the cyclomatic number whether a condition consists of several partial atomic conditions with logical operators or is a single condition. Many extensions and adaptations have been published concerning this.