11. Reviewing an Architecture Document

With David Emery and Rich Hilliard

The prologue presented seven rules for sound documentation. The rules concluded with this prescription:

Review documentation for fitness of purpose. Only the intended users of a document will be able to tell you whether it contains the right information presented in the right way. Enlist their aid. Before a document is released, have it reviewed by representatives of the community or communities for which it was written.

This chapter describes a procedure for doing just that. Like all prescriptions in this book, you should use just as much of it as you think will be beneficial, given the realities and circumstances of your organization and project. For example, Scrum projects often require a complete product (including requirements, design, code, and test results) every 30-day sprint, with a planning session at the beginning and an evaluation at the end. The question sets given in this chapter could serve as a quick checklist for evaluating the documentation products along the way.

To be clear, we are not discussing how to evaluate an architecture, there are several existing methods for that already. Rather, we are evaluating the documentation of an architecture (one purpose of which may be to support an architecture evaluation exercise).

11.1 Steps of the Procedure

This is a six-step procedure. The first step establishes the “why, when, and who” of the review. Subsequent steps provide the “what” and “how.”

Step 1: Establish the purpose of the review. An architecture document (AD) review establishes whether the AD is fit for some specific purpose by a set of identified stakeholders. Stating that purpose will focus the review participants and direct the review. The questions you’ll ask about the document will be different depending on the purpose you have in mind. The sidebar “Why Review an Architecture Document?” provides some examples of why the AD might be reviewed.

It is likely that any AD will need to be fit for more than one purpose, and hence the review will be multi-faceted. The alternative is several smaller reviews, each with a single purpose.

Knowing the “why” will help you identify the “who.” As part of establishing the purpose, identify the stakeholders of the AD who should be represented in the review.

Knowing the “why” will also tell you the “when.” No matter what life-cycle process you’re using, various review purposes will align with certain project stages or milestones. To give an idea of this, Table 11.1 shows a loosely defined set of broadly applicable project phases, the typical activities in each phase, and what you might wish to review the AD for in each case. Of course, the particular life-cycle model your project uses will lead to different phases, activities, and reviews. Carry out the review with enough spare time to allow the AD to be modified after the review to serve its purpose.

Table 11.1 Typical life-cycle phases and the AD reviews that are appropriate for each

Step 2: Establish the subject of the review. This step involves identifying the types of artifacts, the versions of the artifacts, their sources, and the degree of completeness of the artifacts necessary to conduct the review. Obviously, the AD needs to be available. Use the purpose(s) laid out in step 1 to establish the artifact collection required and then gather them for the review. For example, if the AD is being reviewed for conformance to a standard or to a framework, the normative requirements of the standards/framework should also be available. In all cases, make sure that all reviewers are working from the same version(s) of the artifact(s).

Step 3: Build or adapt the appropriate question set(s). This step involves identifying the questions that your review will put to the AD. If you already have a set of questions that meets the purpose of your review, you can use it (perhaps with some modification). If not, you will have to construct it. Organizing questions as question sets allows them to be reused by providing contextual information about the purpose and stakeholder concerns that need to be addressed, as well as guidance for obtaining and interpreting the results. Later in this chapter we present a number of example question sets, each one designed to serve a review purpose. If you choose to use existing question sets, they need to be tailored for the purposes of the review. Questions that are not relevant can be omitted. General questions can be made more specific according to the technology of the project (for example, references to data persistence may be replaced by references to an Oracle database). The question set(s) that you pick will suggest a particular approach, and the questions need to be formulated appropriately. For example, will you use the active design review technique, a questionnaire or checklist given to stakeholders, some sort of automated or measurement-based analysis, or some other approach?

Step 4: Plan the details of the review. Planning involves setting a date for the review, as well as deciding on the time frame and the basic format of the review. The time frame might allow as much time as needed to answer questions or only a limited amount of time, in which case the questions need to be prioritized. Time and resources will affect the format and “weight” of the review. How the results will be communicated needs to be determined and could affect the format and weight of the required answers.

This step also involves identifying the actual review participants (not just abstract stakeholder roles) and securing their participation. An initial assignment of questions to the reviewers responsible for asking them and the stakeholders responsible for supplying the answers can be made at this time. As the review is conducted, the initial priorities and stakeholder assignments may change as a deeper understanding of the documentation is gained and the reviewers probe further into applicable areas.

This step also involves handling the logistics for the review: time and place of meeting(s), paying for everyone’s time, providing read-ahead materials, and so on.

Step 5: Perform the review. Performing the review involves posing the questions to the stakeholders involved in the review and gathering their answers. Depending on the specific approach chosen, this might involve an individual objective review, where stakeholders also play the role of the reviewer and pose questions to themselves; or an inspection, where a separate review team poses questions to the stakeholders. Inspections could take the form of an all-hands gathering, a number of one-on-one meetings, or something in between; the meetings could be face to face, or distributed and remote, using (for example) online virtual meetings. After the results are gathered, the evaluation considerations and criteria are applied, as defined by the chosen question set(s). Although the reviewers can make some preparations, not all the important issues can be known beforehand. These issues need to be determined in the initial part of the review and will influence the questions and artifacts used as the reviewers dig deeper in these areas.

Step 6: Analyze and summarize the results. The intent of this step is to aggregate the answers to the questions and then make a qualitative determination of the overall impact of the AD against the stakeholders and concerns. Results are not likely to be a simple pass/fail but rather a more nuanced conclusion concerning specific problems in specific parts of the AD.

11.2 Sample Question Sets for Reviewing the Architecture Document

Posing and answering questions in a review is, of course, the heart of the matter. This section discusses what is involved in the formation of question sets—groups of questions that, together, address a narrowly focused purpose for an AD review. Besides the questions themselves, a question set must also contain information to allow a user to make sure the question set is appropriate and use it effectively, as shown below:

1. Question Set Name. As an artifact to be reused, give the question set a name by which it can be referred.

2. Purpose. What review purpose does the question set address?

3. Stakeholders and Concerns. Who are the stakeholders, and which of their concerns are being addressed by the questions? Making stakeholders and concerns a first-class dimension of an AD review effectively elaborates the purpose of the question set and informs the formulation of the questions. (While we can’t expect all of an architecture’s stakeholders to participate in a review, we want to make sure that all of the important stakeholder roles are represented.)

4. Questions. This section contains the questions that constitute the question set. For each question, give the following information:

a. Respondents. To whom should each question be posed? The questions might be addressed to the person speaking for the AD. Usually this will be the architect. The questions might be addressed to reviewers checking the understandability of the AD by using it to answer questions about the architecture it describes. For instance, if the AD should support project planning (a purpose) and is being reviewed for such (using a “project planning” question set), the respondents would include those concerned with project planning—technical managers. If the AD should support development and is now being reviewed for that, the respondents will certainly include key developers. Questions about the AD itself can be answered by examining the AD or analyzing it with a tool (for example, automatically checking to make sure that every cross-reference is defined).

The person(s) to whom a question is posed may or may not be the same as the stakeholder(s) whose concern the question addresses. Review participants may be proxies for stakeholders.

b. Expected Answers. What answer(s) are we looking for? A question set will also involve formulating a set of considerations and criteria to help the reviewers evaluate the AD based on the answers they receive. For example, they might wish to understand not just the answers given by the reviewers but also how much difficulty the reviewers had coming up with those answers. They might wish to understand the criteria the stakeholders used for why they answered “Yes, we’re happy” or “No, we’re not happy.”

The respondents should not be shown the expected answers, to avoid biasing their answers.

c. Criticality. How critical is each question? The “wrong” answer to some questions might halt a project until it’s resolved, whereas the “wrong” answer to other questions might merely be something to watch over time. The questions should come with guidance (perhaps a weighting) to help establish their importance.

5. Advice. Provide additional useful information on how and when the review should be conducted. You might relate experience gained through using the question set in a prior review.

Figure 11.1 provides a sample template that can be used when constructing a question set.

Figure 11.1 Template for a question set

Following are a few example question sets to serve specific AD review purposes. (Some questions might apply to more than one question set.) They are written in different styles to illustrate the ways a question set may be used. For example, the example question set for capturing the right stakeholders is written in the active design review style, and the questions are really directions to stakeholders to use the AD for some purpose. The other example question sets are written as if an interviewer is questioning a stakeholder. These could be adapted to an active design review style or for the purposes of an individual objective review. Some questions that can be answered yes or no are serving as filters, and when the answer is yes, it is appropriate to ask follow-up questions of the form, “How do you know?”

11.2.1 Example Question Set for Capturing the Right Stakeholders and Concerns

The Views and Beyond approach to architecture documentation uses the explicit identification of stakeholders and their concerns to determine which views to include in the AD. Explicitly identifying stakeholders and concerns is also a requirement of ISO/IEC 42010:2007. Therefore, a useful review of the AD examines its choice of stakeholders and concerns to ensure that the important ones are accounted for. Such a review could be usefully carried out quite early, when the stakeholders and concerns are documented but before the rest of the AD is created.

The questions in the example question set below are formulated using the active design review technique.

11.2.2 Example Question Set for Supporting Evaluation

When an architecture is subjected to a comprehensive evaluation, the AD is the vehicle for communicating the architecture to the reviewers, or at least substantiating the architect’s presentation of the architecture. Therefore, it is useful to review the AD before an architecture evaluation takes place to see if it contains the necessary information to allow the evaluation to go forward. By extension, such a review determines whether the architecture is ready (complete enough) to be evaluated.

11.2.3 Example Question Set for Supporting Development

Architecture has value by driving a conforming implementation—that is, that the developers can follow the specifications and constraints of the architecture. The purpose of a review for supporting development is to determine whether there is enough information in the architecture for the development stakeholders to do their jobs. A closely related task is to determine if the AD is sufficient to determine whether a system’s implementation actually conforms to the architecture described in the AD. The emphasis there is on the ability of the AD to identify conformance points for the implemented system, with the expectation that a subsequent review or audit will actually determine conformance of the system to the architecture (described by the AD).

11.2.4 Example Question Set for Reviewing for Conformance to ISO/IEC 42010

This review assesses whether the AD conforms to the requirements of ISO/IEC 42010, Systems and Software Engineering—Architecture Description.

11.3 An Example of Constructing and Conducting a Review

This section shows an example of constructing and carrying out an AD review. The review was conducted to see if a project’s AD was sufficient to support an architecture evaluation.

• Step 1: Establish the purpose of the review. The purpose was to evaluate an AD to see if it was sufficiently complete and consistent to support a formal evaluation of the architecture. The chosen architecture evaluation method was the Architecture Tradeoff Analysis Method (ATAM), which uses a trained evaluation team to assess the consequence of architecture decisions in light of quality attribute requirements and business goals. The evaluation team interacts with the project’s architect and senior designers, as well as important architecture stakeholders. The purpose of the AD review is to ensure that those analysis artifacts (architectural decisions, quality attributes, and business goals) are well documented.

The “why” establishes the “who,” and in this case the architecture evaluation team became the architecture documentation review team.

The “why” also establishes the “when,” and in this case, we conducted the review in time for everyone to present their results at the evaluation kick-off meeting. This meeting is a standard part of the ATAM, in which the evaluation team meets, discusses the architecture, agrees on team roles, and makes the go/no-go decision.

• Step 2: Establish the subject of the review. The ATAM requires the client to provide a presentation of the architecture as well as the architecture documentation before the evaluation exercise commences. In fact, these are used to make a go/no-go decision: If the architecture is not sufficiently mature, it cannot be reliably evaluated. In this case, the client provided the ATAM team leader with copies of both the presentation and the architecture document (in this case called a “software design document”) one month before the scheduled beginning of the evaluation.

• Step 3: Build or adapt the appropriate question set(s). The ATAM go/no-go criteria led us to select the question sets for (1) capturing the right stakeholders and concerns and (2) supporting evaluation (see Section 11.2). If a framework such as TOGAF had been used, the question set for reviewing the choice of the framework and associated viewpoints would have been included as well. We did not involve the business manager or the architect in our review, but we did have available a viewgraph presentation from each of them that described the business drivers and architecture, respectively.

• Step 4: Plan the details of the review. The evaluation team leader did double-duty as the review team leader. That involved making sure all team members had the appropriate artifacts: the architecture documentation and presentation and the right question sets. Since our evaluation team was geographically distributed, with members in five different U.S. cities, we arranged the review process so that members could work independently, at their own pace and schedule. Everyone was asked to report their findings at the kick-off meeting.

• Step 5: Perform the review. The reviewers checked the AD to make sure that the following are documented: a list of the stakeholders’ roles and concerns, the criteria the architect used to produce that list, and how the architecture satisfies the concerns. Each reviewer applied the questions against the AD and recorded their answers. They were e-mailed to the review leader before the kick-off meeting, so that he could get a sense of the findings and make a preliminary judgment as to the suitability of the AD.

• Step 6: Analyze and summarize the results. Each member of the evaluation team provided answers to the team leader. It took each person anywhere from one to four hours to complete the question set. The evaluation leader examined the answers, and in less than an hour was able to glean a team consensus that the AD, while not perfect from the perspective of the question set, was sufficient to support an evaluation. This impression was confirmed during a subsequent telecon. The team leader, satisfied that the AD was sufficiently developed to support an evaluation, decided to proceed. During the evaluation itself, 12 scenarios were analyzed. In every case, the architect was able to use architecture information from the AD (in the form of a viewgraph presentation) to walk through the scenarios and explain how the architecture did or did not support them. In two cases, the evaluation team asked where a particular piece of key information was documented, and the architect was able to show its location in the AD. In all cases, the AD supported the analysis, suggesting that the team’s conclusion as to its suitability was well founded.

11.4 Summary Checklist

• Review architecture documentation to ensure that the architecture is effectively captured in a form that allows stakeholders to understand and use the architecture in the way it was intended.

• Choose questions based on the purpose of the review; three examples of why the AD might be reviewed include: conformance to some normative specification, suitability to support use of the architecture for its intended purpose, and suitability to support architecture evaluation or analysis.

• Organize questions as question sets so they can be reused by providing contextual information about the purpose and stakeholder concerns that need to be addressed, as well as guidance for obtaining and interpreting the results.

11.5 Discussion Questions

1. Suppose that you have been asked to review architecture documentation for conformance to an architecture framework such as DoDAF or TOGAF. Given this purpose, whom among the stakeholders would you invite and when in the life cycle would you hold the review? What questions or question sets (if any) from those in this chapter would you reuse? What additional questions would you ask?

2. Discuss the advantages and disadvantages of conducting an AD review as a separate method or as a procedure that is part of an existing method. For a project you have in mind, which would you choose and why?

3. Knowing that you intend to review the architecture documentation, how might this influence your choosing the views and building the documentation package? What criteria would help you decide whether to incorporate reviews as part of the documenting process or to conduct a separate review activity upon completion of the documentation?

11.6 For Further Reading

An active design review (Parnas and Weiss 1985) is a technique for carrying out guided documentation-based reviews. Some of the example question sets provided in this report use the active design review approach.

SEI Active Reviews for Intermediate Designs (ARID) (Clements, Kazman, and Klein 2002) is a method for performing a scenario-based stakeholder-centric review of a portion of architecture. The review is focused on whether the design is sufficient for the software developers who will use it. ARID is based on active design reviews and the ATAM. The elements of the ARID method could be focused on documentation to create a method to review documentation in line with the approach described in this chapter. The question set for supporting development is especially relevant. Active design reviews are a most promising starting point. For example, active design reviews call for recruiting different kinds of reviewers for different kinds of reviews. Support staff is often used, for instance, to review for document consistency and completeness and for conformance to a template. Active design reviews naturally go with the idea of a spectrum of review purposes, either as separate reviews or as multiple purposes of a single review.

Architecture-centered software project planning (ACSPP) (Paulish 2002) is another approach (like ARID) where a portion of the architecture documentation is given to the developers who are asked to use it. In this case, they are asked to take four hours to sketch an initial design of the subsystem they are tasked with developing and to fill out a sheet of metrics documenting the time and resources needed for the development effort. The question set for supporting development would be relevant for that part of the effort that involves understanding the architecture.

The SARA report (SARA 2002) presents a useful generic model for evaluating software architectures, and it is a good starting point for reading on this subject. Particular methods, such as the SEI’s ATAM, can be thought of as special cases of the SARA model.