Goals and Objectives.

Before making decisions, one needs to ask: what are the goals and objectives of the stakeholders? These will probably be different at different levels of the organization. The goals of a line supervisor will be different than a program manager. Which holds the higher priority? And what are the goals of management above the decision maker? The decision should be made to satisfy (as far as possible) the goals and objectives of the important stakeholders.

Decision Type.

The decision maker needs to understand the type of decision required. Many bad decisions stem from a misunderstanding about the type required. Is the decision binary? Maybe the decision is concerned with a permission of some sort. In these cases, a simple yes/no decision is required. Other binary decisions may not be yes or no, but a choice between two alternatives, make or buy being a classic example. More complex decisions typically involve one or more choices among a set of alternatives. Lastly, the decision maker needs to understand who and what will be affected. Is the decision purely technical, or is there a personal element? Providing the wrong type of decision will certainly lead to significantly negative consequences.

In the same vein, understanding who needs to be included in the decision is vital. Is this decision to be made by an individual? Or is a consensus among a group required? Who needs to approve the decision before it is implemented? The answers to these questions influences when, and how, decisions will be made.

Decision Context.

Understanding the scope of the decision is also essential to making a proper decision. A global (or enterprise-wide) decision will be much different than a system component decision. The consequences of a wrong decision will be far-reaching if the decision affects the enterprise, for example. Context involves understanding the problem or issue that led to a decision point. This will be difficult since context has many dimensions, leading to different goals and objectives for your decision maker:

• technical, involving physical entities, such as subsystem decisions;
• financial, involving investment instruments and quantities;
• personnel, involving people;
• process, involving business and technical procedures, methods, and techniques;
• programmatic, involving resource allocations (including time, space, and funding);
• temporal, meaning the time frame in which a decision is needed (this may be dynamic); and
• legacy, involving past decisions.

Stakeholders.

Stakeholders can be defined as anyone (people or organizations) who will be affected by the results of the decision. Understanding who the stakeholders are with respect to a decision needs to be established before a decision is made. Many times, this does not occur—stakeholders are not recognized before a decision is made. Yet, once the decision is announced or implemented, we can be sure that all who are affected will make their opinion heard.

Legacy Decisions.

Understanding what relevant decisions have been made in the past helps with both the context (described above) and the environment in which the current decision must be made. Consequences and stakeholders can be identified more readily if the decision maker has knowledge of the past.

Supporting Data.

Finally, necessary supporting data for the decision need to be provided in a timely fashion. A coherent and timely data collection plan is needed to ensure proper information can be gathered to support the decision. Accuracy in data collected is dependent on the decision type and context. Many times, decisions are delayed unnecessarily because greater accuracy than needed was demanded before the decision maker would act.

Structured.

These types of decisions tend to be routine, in that the context is well understood and the decision scope is known. Supporting information is usually available, and minimal organization or processing is necessary to make a good decision. In many cases, standards are available, either globally or within an organization, to provide solution methods. Structured decisions have typically been made in the past; thus, a decision maker has a historical record of similar or exact decisions made like the one he is facing.

Semistructured.

These types of decisions fall outside of “routine.” Although similar decisions may have been made, circumstances are different enough that past decisions are not a clear indicator of the right decision choice. Typically, guidance is available though, even when specific methods are not. Many systems engineering decisions fall within the category.

Unstructured.

Unstructured decisions represent complex problems that are unique and typically one-time. Decisions regarding new technologies tend to fall into this category due to the lack of experience or knowledge of the situation. First-time decisions fall into this category. As experience grows and decisions are tested, they may transition from an unstructured decision to the semistructured category.

In addition to the type, the scope of control is important to recognize. Decisions within each scope are structured differently, have different stakeholders, and require different technologies to support.

Operational.

This is the lowest scope of control that systems engineering is concerned about. Operational control is at the practitioner level—the engineers, analysts, architects, testers, and so on, who are performing the work. Many decisions at this scope of control involve structured or semistructured decisions. Heuristics, procedures, and algorithms are typically available to either describe in detail when and how decisions should be made or at least to provide guidelines to decision making. In rare cases, when new technologies are implemented, or a new field is explored, unstructured decisions may rise.

Managerial.

This scope of control defines the primary level of systems engineering decision making—that of the chief engineer, the program manager, and of course, the systems engineer. This scope of control defines the management, mentoring, or coaching level of decisions. Typically, for semistructured decisions, policies, heuristics, and logical relationships are available to guide the systems engineer in these decisions.

Strategic Planning.

This level of control represents an executive- or enterprise-level control. Semistructured decisions usually rely on causality concepts to guide decisions making. Additionally, investment decisions and decisions under uncertainty are typically made at this scope of control level.

Cartoons.

While not typically a systems engineering tool, cartoons are a form of pictorial model that illustrates some of the modeled object’s distinguishing characteristics. First, it is a simplified depiction of the subject, often to an extreme degree. Second, it emphasizes and accentuates selected features, usually by exaggeration, to convey a particular idea. Figure 2.2, “The ideal missile design from the viewpoint of various specialists,” makes a visual statement concerning the need for systems engineering better than words alone can convey. An illustration of a system concept of operations may well contain a cartoon of an operational scenario.

Architectural Models.

A familiar example of the use of modeling in the design of a complex product is that employed by an architect for the construction of a home. Given a customer who intends to build a house to his or her own requirements, an architect is usually hired to translate the customer’s desires into plans and specifications that will instruct the builder exactly what to build and, to a large extent, how. In this instance, the architect serves as the “home systems engineer,” with the responsibility to design a home that balances the desires of the homeowner for utility and aesthetics with the constraints of affordability, schedule, and local building codes.

The architect begins with several sketches based on conversations with the customer, during which the architect seeks to explore and solidify the latter’s general expectations of size and shape. These are pictorial models focused mainly on exterior appearance and orientation on the site. At the same time, the architect sketches a number of alternative floor plans to help the customer decide on the total size and approximate room arrangements. If the customer desires to visualize what the house would more nearly look like, the architect may have a scale model made from wood or cardboard. This would be classified as a physical model, resembling the shape of the proposed house. For homes with complex rooflines or unusual shapes, such a model may be a good investment.

The above models are used to communicate design information between the customer and the architect, using the form (pictorial) most understandable to the customer. The actual construction of the house is done by a number of specialists, as is the building of any complex system. There are carpenters, plumbers, electricians, masons, and so on, who must work from a much more specific and detailed information that they can understand and implement with appropriate building materials. This information is contained in drawings and specifications, such as wiring layouts, air conditioning routing, plumbing fixtures, and the like. The drawings are models, drawn to scale and dimensioned, using special industrial standard symbols for electrical, plumbing, and other fixtures. This type of model represents physical features, as do the pictorials of the house, but is more abstract in the use of symbols in place of pictures of components. The models serve to communicate detailed design information to the builders.

System Block Diagrams.

Systems are, of course, far more complex than conventional structures. They also typically perform a number of functions in reacting to changes in their environment. Consequently, a variety of different types of models are required to describe and communicate their structure and behavior.

One of the most simple models is the “block diagram.” Hierarchical block diagrams have the form of a tree, with its branch structure representing the relationship between components at successive layers of the system. The top level consists of a single block representing the system; the second level consists of blocks representing the subsystems; the third decomposes each subsystem into the components, and so on. At each level, lines connect the blocks to their parent block. Figure 9.2 shows a generic system block diagram of a system composed of three subsystems and eight components.

The block diagram is seen to be a very abstract model, focusing solely on the units of the system structure and their physical relationships. The simple rectangular blocks are strictly symbolic, with no attempt to depict the physical form of the system elements. However, the diagram does communicate very clearly an important type of relationship among the system elements, as well as identify the system’s organizing principle. More complex interactions across the subsystems and components are left to more detailed diagrams and descriptions. The interactions among blocks may be represented by labeling the connecting lines.

System Context Diagrams.

Another useful model in system design is the context diagram, which represents all external entities that may interact with a system, either directly or indirectly. We have already seen the context diagram in Figure 3.2. Such a diagram pictures the system at the center, with no details of its interior structure, surrounded by all its interacting systems, environments, and activities. The objective of a system context diagram is to focus attention on external factors and events that should be considered in developing a complete set of system requirements and constraints. In so doing, it is necessary to visualize not only the operational environment but also the stages leading up to operations, such as installation, integration, and operational evaluation.

Figure 9.3 shows a context diagram for the case of a passenger airliner. The model represents the external relationships between the airliner and various external entities. The system context diagram is a useful starting point for describing and defining the system’s mission and operational environment, showing the interaction of a system with all external entities that may be relevant to its operation. It also provides a basis for formulating system operational scenarios that represent the different conditions under which it must be designed to operate. In commercial systems, the “enterprise diagram” also shows all the system’s external inputs and outputs but also usually includes a representation of the related external entities.

Functional Flow Block Diagrams (FFBDs).

The models discussed previously deal primarily with static relationships within the system’s physical structure. The more significant characteristics of systems and their components are related to how they behave in response to changes in the environment. Such behavior results from the functions that a system performs in response to certain environmental inputs and constraints. Hence, to model system behavior, it is necessary to model its principal functions, how they are derived, and how they are related to one another. The most common form of functional model is called the FFBD.

An example of an FFBD is shown in Figure 9.4. The figure shows the functional flow through an air defense system at the top-level functions of detect, control, and engage, and at the second-level functions that make up each of the above. Note the numbering system of the functional blocks that ties them together. Note also that the names in the blocks represent functions, not physical entities, and thus, all begin with a verb instead of a noun. The arrowheads on the lines between blocks in an FFBD indicate the flow of control and, in this case, also the flow of information. Keep in mind that flow of control does not necessarily equate with flow of information in all cases. The identity of the functions flowing between the blocks may be denoted on the FFBD as an optional feature but is not expected to be complete as it would be in a software DFD.

In the above example, the physical implementation of the functional blocks is not represented and may be subject to considerable variation. From the nature of the functions, however, it may be inferred that a radar installation may be involved in the detection function, along with very considerable software; that the control function is mostly software with operator displays; and that the engage function is largely hardware, such as guns, missiles, or aircraft.

A valuable application of functional flow diagrams was developed by the then Radio Corporation of America, Moorestown Division. Named the functional flow diagrams and descriptions (F²D²), the method is used to diagram several functional levels of the system hierarchy, from the system level down to subcomponents. The diagrams use distinctive symbols to identify hardware, software, and people functions, and show the data that flow between system elements. An important use of F²D² diagrams is in a “war room” or storyboard arrangement, where diagrams for all subsystems are arranged on the walls of a conference room and linked to create a diagram of the entire system. Such a display makes an excellent communication and management tool during the system design process.

DFDs.

DFDs are used in the software structural analysis methodology to model the interactions among the functional elements of a computer program. DFDs have also been used to represent the data flow among physical entities in systems consisting of both hardware and software components. In either case, the labels represent data flow and are labeled with a description of the data traversing the interface.

Integrated Definition Language 0 (IDEF0) Diagrams.

IDEF0 is a standard representation of system activity models, similar to software DFDs, and was described in Chapter 8. Figure 8.3 depicts the rules for depicting an activity. IDEF0 is widely used in the modeling of complex information systems. As in FFBD and F²D² diagrams, the functional blocks are rectangular and the sides of the activity boxes have a unique function. Processing inputs always enter from the left, controls from the top, and mechanisms or resources from the bottom; outputs exit on the right. The name of each block starts with a vowel and carries a label identifying its hierarchical location.

Functional Flow Process Diagrams (FFPD).

The functional flow diagrams described earlier model the functional behavior of a system or a system product. Such diagrams are equally useful in modeling processes, including those involved in systems engineering. Examples of FFPDs are found in every chapter. The system life cycle model is a prime example of a process FFPD. In Chapter 4, Figures 4.1, 4.3, and 4.4 define the flow of system development through the defined stages and phases of the system life cycle. In Chapters 5–8, the first figures show the functional inputs and outputs between the corresponding life cycle phase and those immediately adjoining.

The systems engineering method is modeled in Chapter 4, Figure 4.10, and in greater detail in Figure 4.11. The functional blocks in this case are the principal processes that constitute the systems engineering method. Inside each block is a functional flow diagram that represents the functions performed by the block. The inputs coming from outside the blocks represent the external factors that contribute to the respective processes. Chapters 5–8 contain similar functional flow diagrams to illustrate the processes that take place during each phase of system development.

FFPDs are especially useful as training aids for production workers by resolving complex processes into their elementary components in terms readily understandable by the trainees. All process diagrams have a common basic structure, which consists of three elements: input → processing → output.

Trigonal System Models.

In attempting to understand the functioning of complex systems, it is useful to resolve them into subsystems and components that individually are more simple to understand. A general method that works well in most cases is to resolve the system and each of its subsystems into three basic components:

1. 1. sensing or inputting signals, data, or other media that the system element operates on;
2. 2. processing the inputs to deduce an appropriate reaction to the inputs; and
3. 3. acting on the basis of the instructions from the processing element to implement the system element’s response to the input.

In an example of a system simulation described in the previous subsection, an air defense system was shown to be composed of three functions, namely, detect, control, and engage (see Fig. 9.4). The detect function is seen to correspond to the input portion, control (or analyze and control response) to the processing portion, and engage to the response action portion.

The input–processing–output segmentation can then be applied to each of the subsystems themselves. Thus, in the air defense system example, the detect function can be further resolved into the radar, which senses the reflection from the enemy airplane or missile, the radar signal processor, which resolves the target reflection from interfering clutter and jamming, and the automatic detection and track software, which correlates the signal with previous scans to form a track and calculate its coordinates and velocity vector for transmission to the control subsystem. The other two subsystems may be similarly resolved.

In many systems, there is more than a single input. For example, the automobile is powered by fuel but is steered by the driver. The input–processing–output analysis will produce two or more functional flows: tracing the fuel input will involve the fuel tank and fuel pump, which deliver the fuel, the engine, which converts (processes) the fuel into torque, and the wheels, which produce traction on the road surface to propel the car. A second set of components are associated with steering the car, in which the sensing and decision is accomplished by the driver, with the automobile executing the actual turn in response to steering wheel rotation.

Modeling Languages.

The schematic models described above together were developed relatively independently. Thus, although they have been in use for several decades, they are used according to the experience of the engineer. However, these models do have certain attributes in common. They are, by and large, activity focused. They communicate functionality of systems, whether that is of the form of activities, control, or data. Even block diagrams representing physical entities include interfaces among the entities showing flow of materials, energy, or data. Because of their age (basic block diagrams have been around for over 100 years), we tend to categorize these models as “functional” or “traditional.”

When software engineering emerged as a significant discipline within system development, a new perspective was presented to the engineering community: object-oriented analysis (OOA). Rather than activity based, OOA presented concepts and models that were object based, where object is defined in very broad terms. Theoretically, anything can be an object. As described in Chapter 8, Unified Modeling Language (UML) is now a widely used modeling language for support of systems engineering and architecting.

Approximate Calculations.

Chapter 1 contains a section entitled The Power of Systems Engineering, which cites the critical importance of the use of approximate (“back of the envelope”) calculations to the practice of systems engineering. The ability to perform “sanity checks” on the results of complex calculations or experiments is of inestimable value in avoiding costly mistakes in system development.

Approximate calculations represent the use of mathematical models, which are abstract representations of selected functional characteristics of the system element being studied. Such models capture the dominant variables that determine the main features of the outcome, omitting higher-order effects that would unduly complicate the mathematics. Thus, they facilitate the understanding of the primary functionality of the system element.

As with any model, the results of approximate calculations must be interpreted with full knowledge of their limitations due to the omission of variables that may be significant. If the sanity check deviates significantly from the result being checked, the approximations and other assumptions should be examined before questioning the original result.

In developing the skill to use approximate calculations, the systems engineer must make the judgment as to how far to go into the technical fundamentals in each specific case. One alternative is to be satisfied with an interrogation of the designers who made the original analysis. Another is to ask an expert in the discipline to make an independent check. A third is to apply the systems engineer’s own knowledge, to augment it by reference to a handbook or text, and to carry out the approximate calculation personally.

The appropriate choice among these alternatives is, of course, situation dependent. However, it is advisable that in selected critical technical areas, the systems engineer becomes sufficiently familiar with the fundamentals to feel comfortable in making independent judgments. Developing such skills is part of the systems engineer’s special role of integrating multidisciplinary efforts, assessing system risks, and deciding the areas that require analysis, development, or experimentation.

Elementary Relationships.

In every field of engineering and physics, there are some elementary relationships with which the systems engineer should be aware, or familiar. Newton’s laws are applicable in all vehicular systems. In the case of structural elements under stress, it is often useful to refer to relationships involving strength and elastic properties of beams, cylinders, and other simple structures. With electronic components, the systems engineer should be familiar with the elementary properties of electronic circuits. There are “rules of thumb” in most technical fields, which are usually based on elementary mathematical relationships.

Statistical Distributions.

Every engineer is familiar with the Gaussian (normal) distribution function characteristic of random noise and other simple natural effects. Some other distribution functions that are of interest include the Rayleigh distribution, which is valuable in analyzing signals returned from radar clutter, the Poisson distribution, the exponential distribution, and the binomial distribution; all of these obey simple mathematical equations.

Graphs.

Models representing empirical relationships that do not correspond to explicit mathematical equations are usually depicted by graphs. Figure 2.1a in Chapter 2 is a graph illustrating the typical relationship between performance and the cost to develop it. Such models are mainly used to communicate qualitative concepts, although test data plotted in the form of a graph can show a quantitative relationship. Bar charts, such as one showing the variations in production by month, or the cost of alternative products, are also models that serve to communicate relationships in a more effective manner than by a list of numbers.

Scale Models.

These are (usually) small-scale versions of a building, vehicle, or other system, often used to represent the external appearance of a product. An example of the engineering use of scale models is the testing of a model of an air vehicle in a wind tunnel or of a submersible in a water tunnel or tow tank.

Mock-Ups.

Full-scale versions of vehicles, parts of a building, or other structures are used in later stages of development of systems containing accommodation for operators and other personnel. These provide realistic representations of human–system interfaces to validate or possibly to modify their design prior to a detailed design of the interfaces.

Prototypes.

Previous chapters have discussed the construction and testing of development, engineering, and product prototypes, as appropriate to the system in hand. These also represent physical models of the system, although they possess most of the properties of the operational system. However, strictly speaking, they are still models.

Computer-based tools are being increasingly used in place of physical models such as mock-ups and even prototypes. Such tools can detect physical interferences and permit many engineering tasks formerly done with physical models to be accomplished with computer models.

Examples: Aircraft, Automobiles, and Space Vehicles.

Few technical problems are as complicated as the design of high-speed aircraft. The aerodynamic forces are quite nonlinear and change drastically in going between subsonic and supersonic regimes. The stresses on airplane structures can be extremely high, resulting in flexure of wings and control surfaces. There are flow interference effects between the wings and tail structure that depend sharply on altitude, speed, and flight attitude. Simulation permits all of these forces and effects to be realistically represented in six-degree-of-freedom models (three position and three rotation coordinates).

The basic motions of an automobile are, of course, far simpler than those of an aircraft. However, modern automobiles possess features that call on very sophisticated dynamic analysis. The control dynamics of antilock brakes are complex and critical, as are those of traction control devices. The action of airbag deployment devices is even more critical and sensitive. Being intimately associated with passenger safety, these devices must be reliable under all expected conditions. Here again, simulation is an essential tool.

Without modern simulation, there would be no space program as we know it. The task of building a spacecraft and booster assembly that can execute several burns to put the spacecraft into orbit, that can survive launch, deploy solar panels, and antennae, control its attitude for reasons of illumination, observation, or communication, and perform a series of experiments in space would simply be impossible without a variety of simulations. The international space station program achieved remarkable sustainability as each mission was simulated and rehearsed to near perfection.

Mechanical Stress Testing.

System or system elements that are designed to survive harsh environments during their operating life, such as missiles, aircraft systems, spacecraft, and so on, need to be subjected to stresses simulating such conditions. This is customarily done with mechanical shake tables, vibrators, and shock testing.

Crash Testing.

To meet safety standards, automobile manufacturers subject their products to crash tests, where the automobile body is sacrificed to obtain data on the extent to which its structural features lessen the injury suffered by the occupant. This is done by use of simulated human occupants, equipped with extensive instrumentation that measures the severity of the blow resulting from the impact. The entire test and test analysis are usually computer driven.

Wind Tunnel Testing.

In the development of air vehicles, an indispensable tool is an aerodynamic wind tunnel. Even though modern computer programs can model the forces of fluid flow on flying bodies, the complexity of the behavior, especially near the velocity of sound, and interactions between different body surfaces often require extensive testing in facilities that produce controlled airflow conditions impinging on models of aerodynamic bodies or components. In such facilities, the aerodynamic model is mounted on a fixture that measures forces along all components and is computer controlled to vary the model angle of attack, control surface deflection, and other parameters, and to record all data for subsequent analysis.

As noted in the discussion of scale models, analogous simulations are used in the development of the hulls and steering controls of surface vessels and submersibles, using water tunnels and tow tanks.

Spatial Simulations.

A spatial virtual reality simulation is often useful when it is important to visualize the interior of enclosed spaces and the connecting exits and entries of those spaces. Computer programs exist that permit the rapid design of these spaces and the interior furnishings. A virtual reality feature makes it possible for an observer to “walk” through the spaces in any direction. This type of model can be useful for the preliminary designs of houses, buildings, control centers, storage spaces, parts of ships, and even factory layouts. An auxiliary feature of this type of computer model is the ability to print out depictions in either two- or three-dimensional forms, including labels and dimensions.

Spatial virtual simulations require the input to the computer of a detailed three-dimensional description of the space and its contents. Also, the viewing position is input into the simulation either from sensors in the observer’s headset or directed with a joystick, mouse, or other input device. The virtual image is computed in real time and projected either to the observer’s headset or on a display screen. Figure 9.7 illustrates the relationship between the coordinates of two sides of a room with a bookcase on one wall, a window on the other, and a chair in the corner, and a computer-generated image of how an observer facing the corner would see it.

Video Games.

Commercial video games present the player with a dynamic scenario with moving figures and scenery that responds to the player’s commands. In many games, the display is fashioned in such a way that the player has the feeling of being inside the scene of the action rather than of being a spectator.

Battlefield Simulation.

A soldier on a battlefield usually has an extremely restricted vision of the surroundings, enemy positions, other forces, and so on. Military departments are actively seeking ways to extend the soldier’s view and knowledge by integrating the local picture with situation information received from other sources through communication links. Virtual reality techniques are expected to be one of the key methods of achieving these objectives of situational awareness.

Defining the Objective (Requirements Analysis).

The trade-off process must start by defining the objectives for the trade study itself. This is carried out by identifying the requirements that the solution (i.e., the result of the decision) must fulfill. The requirements are best expressed in terms of measures of effectiveness (MOE), as quantitatively as practicable, to characterize the merits of a candidate solution.

Identification of Alternatives (Concept Exploration).

To provide a set of alternative candidates, an effort must be made to identify as many potential courses of action as will include all promising candidate alternatives. Any that fail to comply with an essential requirement should be rejected.

Comparing the Alternatives (Concept Definition).

To determine the relative merits of the alternatives, the candidate solutions should be compared with one another with respect to each of their MOEs. The relative order of merit is judged by the cumulative rating of all the MOEs, including a satisfactory balance among the different MOEs.

Sensitivity Analysis (Concept Validation).

The results of the process should be validated by examining their sensitivity to the assumptions. MOE prioritization and candidate ratings are varied within limits reflecting the accuracy of the data. Candidates rated low in only one or two MOEs should be reexamined to determine whether this result could be changed by a relatively straightforward modification. Unless a single candidate is clearly superior, and the result is stable to such variations, further study should be conducted.

Step 1: Definition of the Objectives.

To introduce the trade study, the objectives must be clearly defined. These should include the principal requirements and should identify the mandatory ones that all candidates must meet. The issues that will be involved in selecting the preferred solution should also be included. The objectives should be commensurate with the phase of system development. The operational context and the relationships to other trade studies should be identified at this time. Trade studies conducted early in the system development cycle are typically conducted at the system level and higher. Detailed component-level trade studies are conducted later, during engineering and implementation phases.

Step 2: Identification of Viable Alternatives.

As stated previously, before embarking on a comparative evaluation, an effort should be made to define several candidates to ensure that a potentially valuable one is not overlooked. A useful strategy for finding candidate alternatives is to consider those that maximize a particularly important characteristic. Such a strategy is illustrated in the section on concept selection in Chapter 8, in which it is suggested to consider candidates based on the following:

• the predecessor system as a baseline,
• technological advances,
• innovative concepts, and
• candidates suggested by interested parties.

In selecting alternatives, no candidate should be included that does not meet the mandatory requirements, unless it can be modified to qualify. However, keep the set of mandatory requirements small. Sometimes, an alternative concept that does not quite meet a mandatory requirement but is superior in other categories, or results in significant cost savings, is rejected because it does not reach a certain threshold. Ensure that all mandatory requirements truly are mandatory—and not simply someone’s guess or wish.

The factors to consider in developing the set of alternatives are the following:

• There is never a single possible solution. Complex problems can be solved in a variety of ways and by a variety of implementations. In our experience, we have never encountered a problem with one and only one solution.
• Finding the optimal solution is rarely worth the effort. In simple terms, systems engineering can be thought of as the art and science of finding the “good enough” solution. Finding the mathematical optimum is expensive and many times near impossible.
• Understand the discriminators among alternatives. Although the selection criteria are not chosen at this step (this is the subject of the next step), the systems engineer should have an understanding of what discriminates alternatives. Some discriminators are obvious and exist regardless of the type of system you are developing: cost, technical risk, reliability, safety, and quality. Even of some of these cannot be quantified, yet a basic notion of how alternatives discriminate within these basic categories will enable the culling of alternatives to a reasonable quantity.
• Remain open to additional solutions surfacing during the trade study. This step is not forgotten once an initial set of alternatives has been identified. Many times, even near the end of the formal trade study, additional options may emerge that hold promise. Typically, a new option arises that combines the best features of two or more original alternatives. Many times, identifying these alternatives is not possible, or at least difficult, early in the process.

Staged Process.

This step tends to occur in discrete stages. Initially, a large number and variety of alternatives should be considered. Brainstorming is one effective method of capturing a variety of alternatives, without evaluating their merits. Challenge participants to think “out of the box” to ensure that no option is overlooked. And while some ridiculous ideas are offered, this tends to stimulate thinking on other, plausible options. In our experience, 40–50 alternatives can be identified initially. This set is not our final set of alternatives, of course. It needs to be reduced.

As long as there are more than three to five potential alternatives, it is suggested that the staged approach be continued, culling the set down to a manageable set. The process of reducing alternatives generally follows a rank-ordering process, rather than quantitative weighing and scoring, to weed out less desirable candidates. Options can be dismissed due to a variety of reasons: cost, technological feasibility, safety, manufacturability, operational risk, and so on. This process may also uncover criteria that are not useful differentiators. Follow-on stages would focus on a few candidates that include likely candidates. These would be subjected to a much more thorough analysis as described below.

Remember to document the choices and reasoning behind the decision. Include the specifications for the alternatives to make the trade-off as quantitative as possible. The result of this multistage process is a reasonable set of alternatives that can be evaluated formally and comprehensively.

Step 3: Definition of Selection Criteria.

The basis of differentiating between alternative solutions is a set of selection criteria to be chosen from and referenced to the requirements that define the solution. Each criterion must be an essential attribute of the product, expressed as a MOE, related to one or more of its requirements. It is desirable that it be quantifiable so that its value for each alternative may be derived objectively. Cost is almost always a key criterion. Reliability and maintainability are also usually important characteristics, but they must be quantified. In the case of large systems, size, weight, and power requirements can be important criteria. In software products, ease of use and supportability are usually important differentiators.

Characteristics that are possessed by all candidates to a comparable degree do not serve to distinguish among them and hence should not be used, because their inclusion only tends to obscure the significant discriminators. Also, two closely interdependent characteristics do not contribute more discrimination than can be obtained by one of them with appropriate weighting. The number of criteria used in a particular formal trade study can vary widely but usually ranges between 6 and 10. Fewer criteria may not appear convincing of a thorough study. More criteria tend to make the process unwieldy without adding value.

Step 4: Assignment of Weighting Factors to Selection Criteria.

In a given set of criteria, not all of them are equally important in determining the overall value of an alternative. Such differences in importance are taken into account by assigning each criterion a “weighting factor” that magnifies the contribution of the most critical criteria, that is, those to which the total value is the most sensitive, in comparison to the less critical. This procedure often turns out to be troublesome to carry out because many, if not most, of the criteria are incommensurable, such as cost versus risk, or accuracy versus weight. Also, judgments of relative criticality tend to be subjective and often depend on the particular scenario used for the comparison.

Several alternative weighting schemes are available. All of them should engage domain experts to help with the decisions. Perhaps the simplest is to assign weights from 1 to n (with n having the greatest contribution). Although subjective, the criteria are measured relative to each other (as opposed to an absolute measure). A disadvantage with using the typical 1 to n scheme is that people tend to group around the median, in this case, (1 + n)/2. For example, using a 1–5 scale may really be using a 1–3 scale since many will simply not use 1 and 5 often. Other times, people tend to rate all criteria high, either a 4 or 5—resulting in the equivalent of using 1–2.

Adding some objectivity requires a trade-off decision in and of itself when assigning weights. For instance, we could still use the 1–5 scale, but use a maximum number of weighting points; that is, the sum of all of the weights must not exceed a maximum value. A good starting maximum sum might be to take the sum of all average weights,

where

Thus, this scheme holds the average weight as a constant. If the engineer (or stakeholders, depending on who is weighting the criteria) wants to weight a criterion higher, then she must reduce the weight of another criterion. Keep in mind, however, with any subjective weighting scheme (any scheme that uses “1 to n”), you are making assumptions about the relative importance. A “5” is five times as relevant as a “1.” These numbers are used in the calculations to compare alternatives. Make sure the scheme is appropriate.

If more mathematical accuracy is desired, the weights could be constrained to sum to 1.0. Thus, each weighting would be a number between 0 and 1.0. This scheme has some mathematical advantages that will be described later in this chapter. One logical advantage is that weightings are not constrained to integers. If one alternative is 50% more important than another, this scheme can represent that relationship; integers cannot. When using spreadsheets for the calculations, be sure not to allow too many significant figures! The credibility of the engineering judgment would fall quickly.

To summarize, deciding on a weighting scheme is important. Careful thinking about the types of relative importance of alternatives is required. Otherwise, the engineer can inadvertently bias the results without knowing.

Step 5: Assignment of Value Ratings for Alternatives.

This step can be confusing to many people. You may ask, why can we not simply measure the criteria values for each alternative at this point and use those values in our comparison? Of course, we could, but it becomes hard to compare the alternatives without integrating the criteria in some manner. Each criterion may use different units; so how does the systems engineer integrate multiple criteria together to gain an understanding of an overall value assessment for each alternative? We cannot combine measures of area (square foot) with velocity (foot per second), for example. And what if a criterion is impossible to measure? Does that mean subjective criteria are simply not used? In fact, subjective criteria are used in system development frequently (though usually in combination with objective criteria). Thus, we need a method to combine criterion together without trying to integrate units that are different. Basically, we need an additional step beyond measuring criterion values for each alternative. We need to assign an effectiveness value.

There are several methods of assigning a value for each criterion to each alternative. Each has its own set of advantages and attributes. And the method ultimately used may not be a choice for the systems engineer, depending on what data can be collected. Three basic options are available: (1) the subjective value method, (2) the step function method, and (3) the utility function method.

The first method relies on the systems engineer’s subjective assessment of the alternative relative to each criterion. The latter two methods use actual measurements and translate the measurement to a value. For example, if volume is a criterion with cubic feet as the unit, then each alternative would be measured directly—what is the volume that each alternative fills, in cubic feet? Combinations of the three methods are also frequently used.

Subjective Value Method.

When this method is chosen, the procedure begins with a judgment of the relative utility of each criterion on a scale analogous to student grading, say 1–5. Thus, 1 = poor, 2 = fair, 3 = satisfactory, 4 = good, and 5 = superior. (A candidate that fails a criterion may be given a zero, or even a negative score if the scores are to be summed, to ensure that the candidate will be rejected despite high scores on other criteria.) This is the effectiveness value for each criterion/alternative pair. The score assigned to the contribution of a given criterion to a specific candidate is the product of the weight assigned to the criterion and the assigned effectiveness value of the candidate in meeting the criterion.

Table 9.3 depicts a generic example that could be constructed for each alter­native, for four selection criteria (they are not described, just numbered one through four).

In this method, the value v_i would be an integer between 1 and 5 (using our subjective effectiveness rating above), and would be assigned by the systems engineer.

Actual Measurement Method.

If a more objective effectiveness rating is desired (more than “poor/fair/satisfactory/good/superior”), and alternatives could be measured for each criterion, then a simple mathematical step function could be constructed that translates an actual measurement into an effectiveness value. The systems engineer still needs to define this function and what value will be assigned to what range of measurements. Using our example of volume as a criterion, we could define a step function that assigns an effectiveness value to certain levels of volume. Assuming lesser volume is better effectiveness,

If an alternative fills 3.47 ft³ of volume, it would be given an effectiveness value of 3. Keep this concept in mind as we will use something similar with our next method.

Table 9.4 illustrates this method. In this case, the alternative is actually measured for each criterion; the result is m_i. The step function is then used to translate the measurement to an effectiveness value, v_i. The final score for that criterion is the product of the measurement and value, m_iv_i. Once the measurements are converted to values, the actual measurements, m_i, are no longer used.

Utility Function Method.

A refinement of the second approach is to develop a utility function for each criterion, which relates its measurable performance to a number between zero and one. Each criterion is measured, just as in the second method. But instead of allocating subjective values, a utility function is used to map each measurement to a value between zero and one.

Advantages to this method over the second are mathematical. As in using a utility function for weights (i.e., summing the weights to one), using utility functions places all criteria on an equal basis—the effectiveness of each criterion is constrained to a number between zero and one. Furthermore, if utility functions are used, mathematical properties of utility functions can be utilized. These are described in the next section.

Figure 9.8 illustrates some examples of utility functions. A utility function can be either continuous or discrete, linear or nonlinear.

If utility functions are used, calculating a total score for each criterion is similar to the second method. The score is simply the product of the weight and the utility. Table 9.5 depicts these relationships.

Step 6: Calculating Comparative Scores.

The conventional method for combining the scores for the several alternatives is to calculate the sum of the weighted scores for each criterion to produce a total score. The candidate with the greatest summed value is judged to be the best candidate given the selection criteria and weightings, provided the score of the next highest alternative is statistically lower:

This process is simple to implement, but lumping together the scores of the individual criteria tends to obscure factors that may be more important than initially supposed. For example, a candidate may receive a very low score on an essential MOE and high scores on several others. This lack of balance should not be obscured. It is strongly recommended that in addition to presenting the total scores, a graph of the criteria profile for each candidate be also included. Figure 9.9 presents a notional example of a criteria profile for three alternatives.

Deciding which alternative among the three is best is difficult since Alt-1 scores very low on criterion D but very high on criteria A, B, and C. Is this significant? If only the weighted sums are used, then Alt-1 would be the best candidate (with a sum of 5 + 5 + 4 + 1 = 15). In its purest form, Alt-1 is selected due to its greatest weighted sum, but as always, numbers do not tell the whole story; we need analysis.

Step 7: Analyzing the Results.

Because of the necessary reliance on qualitative judgments and the incommensurable nature of many of the criteria, the results of a trade study should be subjected to critical scrutiny. This process is especially important when the two or three top scores are close together and do not produce a decisive winner.

An essential step in analyzing the results is to examine the individual candidate profiles (scores for each criterion). Candidates that score poorly on one or more criteria may be less desirable than those with satisfactory scores in all categories. Cost is another factor that needs to be considered separately.

The conventional method of summing the individual scores is simple to use but has the unfortunate characteristic of underemphasizing low scores. A technique that does not suffer from this defect is to derive the composite score for a candidate by calculating the product (or geometric mean), rather than the sum, of the individual scores for the several criteria. If a candidate scores a zero on any criterion, the product function will also be zero, rejecting the alternative. An equivalent variant with the same property is to sum the logarithms of the individual scores.

A conventional approach to testing the robustness of trade study results is called “sensitivity analysis.” Sensitivity analysis tests the invariance of the results to small changes in the individual weighting factors and scores. Because of uncertainties in the assignment of weighting and scores, substantial variations (20–30%) should be considered. A preferred approach is to sequentially set each criterion equal to zero and to recalculate the study. When such variations do not change the initial top choice, the procedure builds confidence in the result of the analysis.

An additional sensitivity test is to consider if there are important criteria that have not been included in the evaluation. Examples may be risk, growth potential, availability of support services, maturity of the product or of its supplier, ease of use, and so on. One of the alternatives may be considerably more attractive in regard to several of such additional issues.

Trade-Off Analysis Report.

The results of a formal trade study represent an important milestone in the development of a system or other important operation and will contribute to decisions that will determine the future course. As such, they have to be communicated to all principal participants, who may include customers, managers, technical leaders, and others closely associated with the subject at issue. Such communication takes two forms: presentations and written reports.

Both oral and written reports must contain sufficient material to fully explain the method used and the rationale leading to the conclusions. They should include

• a statement of the issue and requirements on the solution;
• a discussion of assumptions and relationships to other components and subsystems;
• a setting of mission or operational considerations;
• a listing of relevant and critical system or subsystem requirements;
• a description of each alternative selected and the key features that led to its selection;
• an explanation of how the evaluation criteria were selected and the rationale for their prioritization (weighting);
• a rationale for assigning specific scores to each alternative for each criterion;
• a summary of the resulting comparison;
• a description of the sensitivity analysis and its results;
• the final conclusion of the analysis and an evaluation of its validity;
• recommendation for adoption of the study results or further analysis; and
• references to technical, quantitative material.

The presentation has the objective of presenting valuable information to program decision makers in order to make informed decisions. It requires a careful balance between sufficient substance to be clear and too much detail to be confusing. To this end, it should consist mainly of graphical displays, for which the subject is well suited, with a minimum of word charts. On the other hand, it is essential that the rationale for selection weighting and scoring is clear, logical, and persuasive. A copy of the comparison spreadsheet may be useful as a handout.

The purpose of the written trade study report is not only to provide a historical record of the basis for program decisions but also, more importantly, to provide a reference for reviewing the subject if problems arise later in the program. It represents the documented record of the analysis and its results. Its scope affords the opportunity for a detailed account of the steps of the study. For example, it may contain drawings, functional diagrams, performance analysis results, experimental data, and other materials that support the trade-off study.

Scoring.

On a scale of 0–5, the maximum weight of 5 was assigned to accuracy—for obvious reasons. The next highest, 4, was assigned to speed, versatility, and reliability, all of which have a direct impact on the utility of the tool. User interface and support were assigned a medium weight of 3 because, while they are important, they are not quite as critical as the others to the successful use of the tool.

Cost was considered separately to enable the consideration of cost/effectiveness as a separate evaluation factor.

The subjective value method was used to determine raw scores. The raw scores for each of the candidates were assigned on a scale of 5 = superior, 4 = good, 3 = satisfactory, 2 = weak, 1 = poor, and 0 = unacceptable. The row below the criteria lists the summed total of the weighted scores. The cost for each candidate tool and the ratio of the total score to the cost are listed in the last two rows.

Analysis.

Comparing the summed scores in Table 9.6 shows that HPA and Zenco score significantly higher than the others. It is worth noting, however, that CodeView scored “satisfactory” on all criteria and is the least expensive by a substantial margin. Videx, CodeView, and HPA are essentially equal in cost/effectiveness.

Sensitivity analysis by varying criteria weightings does not resolve the difference between HPA and Zenco. However, examining the profiles of the candidates’ raw scores highlights the weak performance of Zenco with respect to accuracy. This, coupled with its very high price, would disqualify this candidate. The profile test also highlights the weak reliability and poor user support of PeopleSoft, and the weak accuracy and high price of Videx. In contrast, HPA scores satisfactory or above in all categories and superior in half of them.

The above detailed analysis should result in a recommendation to select HPA as the best tool, with an option of accepting CodeView if cost is a determining factor.

MAUT

This form of mathematics (which falls under operations research) is used quite extensively in all types of engineering, due to its simplicity. It can easily be implemented via a spreadsheet.

As described above, the basic concept involves identifying a set of evaluation criteria with which to select among a set of alternative candidates. We would like to combine the effectiveness values for these criteria into a single metric. However, these criteria do not have similar meanings that allow their integration. For example, suppose we had three selection criteria: reliability, volume, and weight. How do we evaluate the three together? Moreover, we typically need to trade off one attribute for another. So, how much reliability is worth x volume and y weight? In addition, criteria typically have different units. Reliability has no units as it is a probability; volume may use cubic meter and weight may use kilogram. How do we combine these three criteria into a single measure?

MAUT’s answer to this dilemma is to use the concept of utility and utility functions. A utility function, U(m_i), translates the selection criterion, m_i, to a unitless measure of utility. This function may be subjective or objective, depending on the data that are available. Typically, utility is measured using a scalar between zero and one, but any range of values will do.

Combining weighted utilities can be accomplished in a number of ways. Three were mentioned above: weighted sum, weighted product, and sum of the logarithms of the weighted utility. Typically, the weighted sum is used, at least as a start. During sensitivity analysis, other methods of combining terms are attempted.

Equal Cost–Variable Effectiveness.

This type constrains the alternatives to a single cost level or a maximum cost threshold. If all of the alternatives are constrained to similar or the same costs, then the results offer an observable difference in effectiveness—enabling a simple ranking of alternatives. In essence, cost is taken out of the equation in comparing alternative systems.

The disadvantages of this CBA include the difficulty in constraining the alternatives to the same, or a maximum cost. Examples include selecting a system within a cost range, such as selecting a new car or purchasing equipment. Of course, one could argue that decisions such as these do not need detailed analysis—a straightforward trade-off study would be sufficient! More detailed examples include a new strike weapon system for the military. A maximum cost level is typically included in a new system’s requirements, including its key performance parameter (KPP). All alternatives are required to be less than the cost threshold. Only effectiveness of these system alternatives varies.

Variable Cost–Equal Effectiveness.

This type constrains the alternatives to a single effectiveness level or a minimum effectiveness threshold. If all of the alternatives are constrained to similar or the same effectiveness levels, then the results offer an observable difference in cost, enabling a simple ranking of alternatives. In essence, effectiveness is taken out of the equation in comparing alternative systems.

The disadvantages of this CBA include the difficulty in constraining the alternatives to the same or minimum effectiveness level. Examples include selecting a power plant to provide a selected amount of energy. In this case, the energy level, or amount of electricity, would be the minimum effectiveness threshold. Options would then be judged largely on cost.

Variable Cost–Variable Effectiveness.

This type constrains the alternatives to both a maximum cost level and a minimum effectiveness level. However, beyond the limits, the alternatives can be any combination of cost and effectiveness. In some cases, no limits are established, and the alternatives are “free” to be at any cost and effectiveness levels. This is rare for government CBAs, but there can be advantages to this form of analysis. Out-of-the-box alternatives can be explored when cost and effectiveness constraints are removed. In some cases, a possible alternative that provides effectiveness that is just under the minimum (say, 5% under the threshold) may cost 50% less than any other alternative. Would not a decision maker at least want to be informed of that possibility? By and large, however, minimum and maximum levels are established to keep the number of alternatives manageable, with the exceptional case being handled separately.

The disadvantages of this CBA type include the risk that no alternative is clearly “better” than the rest. Each alternative offers effectiveness that is commensurate with its cost. Of course, this is not necessarily bad; the decision maker then must decide which alternative he wants. In these cases, calculating the effectiveness per unit cost is an additional measure that can shed light on the decision.

Most systems fall into this category: a new vehicle design, a new spaceship or satellite, a new software system, a new energy system, and so on.

Of course, the examples and notional Figure 9.19 all address single-dimensional applications. Multidimensional costs and effectiveness increase the complexity but still fall into one of the three types of CBA. Two general methods for handling multidimensional CBA are (1) combining effectiveness and cost into a single metric, typically by employing MAUT, then applying one of the three methods described; or (2) using an effectiveness and cost profile vector, with mathematical constraints on the vector as opposed to a single scalar threshold.