1.1 Introduction

This book is about trade-off analyses in the life cycle of a system. It is written from the perspective of engineers, systems engineers, and other decision-makers involved in the life cycle of a system. In this book, we present the best practices for performing systems engineering trade-off analyses in a step-by-step, structured manner. Our intent is to make it an easy-to-understand and useful reference for students, practitioners, and researchers.

Systems are developed to create value for stakeholders by providing desired capabilities. Stakeholders include investors, government agencies, customers/acquirers, end users/operators, system developers/integrators, trainers, and system maintainers, among others. Decisions are ubiquitous across the system life cycle. System decision-makers (DMs) are those individuals who make important decisions pertaining to the technical and management compromises that shape the concept definition, system definition, system realization, deployment and use, and product and service life management (including maintenance, enhancement, and disposal).

When there are multiple stakeholders, there are often competing objectives and requirements. To achieve a certain attainment level on one objective, a sacrifice or trade-off may be required in the attainment level of other objectives. Similarly, complex system designs may offer multiple alternatives to achieve the system's objectives, and this, too, requires analysis to achieve the best balance among the trade-offs. The process that leads to a reasoned compromise in these situations is commonly referred to as a “trade-off analysis” or a “trade study.”

This book project began with a request by the International Council on Systems Engineering (INCOSE) (INCOSE Home Page, 2015) Corporate Advisory Board (CAB) to the INCOSE Decision Analysis Working Group. The CAB identified the lack of effective trade-off analysis methods as a key concern and requested help in documenting best practices. This book project was also motivated by the need to formalize systems engineering trade-off analysis to help make it an integral part of the systems engineering life cycle. It provides essential elaboration of the decision management process in ISO/IEC/IEEE 15288, Systems and Software Engineering – System Life Cycle Processes, the INCOSE Systems Engineering Handbook Version 4, and the Systems Engineering Body of Knowledge (SEBoK, 2015).

Decision-makers (DM), especially program managers and systems engineers, stand to benefit from a collaborative decision management process that engages all stakeholders (SH) who have a say in system design decisions. In particular, systems engineers can exploit trade-off studies to help define the problem/opportunity, characterize the solution space, identify sources of value, identify and evaluate alternatives, identify risks, acquire insights, and provide recommendations to system SHs and other DMs.

This book focuses on engineering trade-off analysis techniques for both systems and systems of systems (Madni and Sievers, 2014a,b; Ordoukhanian and Madni, 2015). We recommend that trade-off studies be consistent with SE standards (ISO/IEC/IEEE 15288:2015), based on a formal lexicon, have a sound mathematical foundation, and provide credible and timely data to DMs and other SHs. We provide such a lexicon and a formal foundation (Chapter 2) based on decision analysis for effective and efficient trade-off studies. Our approach supplements decision analysis, a central part of decision-based design (Hazelrigg, 1998), with Value-Focused Thinking (Keeney, 1992) within a model-based engineering framework (Madni & Sievers, 2015).

1.2 Trade-off Analyses Throughout the Life Cycle

New system development entails a number of interrelated decisions. Table 1.1 provides a partial list of decisions opportunities to improve the system value that are commonly encountered throughout a system's life cycle. Many of these decisions stand to benefit from a holistic perspective that combines the systems engineering discipline with a composite decision model that aggregates the data produced by engineering, performance, and cost models and translates them into terms that are relevant and meaningful to the various stakeholders, especially DMs. This holistic perspective is especially valuable in gate (go/no-go funding) decisions to ensure that affordable alternatives are available for the next life cycle stage.

1.3 Trade-off Analysis to Identify System Value

Systems provide value through the capabilities they provide or the products and services they enable (Madni, 2012). Decision analysis is an operations research technique that provides models to define value and a sound data-driven, objective, defensible, mathematical foundation for trade-off analyses. The graphic shown in Figure 1.1 helps visualize the importance of opportunity definition (Chapter 6) to value creation. For example, Chevron uses the “Eagle's Beak”, as shown in this figure, to convey the importance of project definition and project execution. The five phases shown in Figure 1.1 constitute the project life cycle used by Chevron (Lavingia, 2014). The process leads to value identification and value realization. At Chevron, decision analysis plays an important role in the three phases of value identification: identify opportunity; generate and select alternatives; and develop the preferred opportunity. The Chevron process employs stages and gates similar to those found in most system life cycles. Each phase consists of activities that produce information; clearly defined deliverables; and an explicit decision to proceed, exit, or recycle. Chevron employs project management in all five phases of the Chevron Project Development and Execution Process (Decision-Making in an Uncertain World: A Chevron Case Study, 2014). Similarly, for the system life cycle, value identification occurs during the concept definition and system definition phases, and decision analysis plays the same important role.

Figure 1.1 highlights five important points. First, the problem or opportunity definition (see Chapter 6) is an important first step in value identification. Second, the generation of good alternatives (see Chapter 8) is critical to identifying higher value. Third, the development, evaluation, and selection of preferred alternatives can significantly increase value. Fourth, good project execution is required to realize potential value. Fifth, project execution is performed in the face of uncertainties (see Chapter 3). In this book, we focus on the value of using trade-off studies to help in the identification of both value and risk, as the timely identification of risk can help implementers mitigate potential barriers to value realization.

1.4 Trade-off Analysis to Identify System Uncertainties and Risks

System risks can affect performance, schedule, and cost. Building on several frameworks, Table 1.2 provides a list of the sources of systems risk (Parnell, 2009). The first column in Table 1.2 lists the potential source of risk. The second column lists the major questions defining the risk. The third column lists some of the major potential uncertainties for this risk source. The major questions and the uncertainties are meant to be illustrative and not all inclusive. Many of these risks create uncertainties, which should be considered in trade-off analyses. Chapter 3 provides techniques for using probability to model these uncertainties in trade-off analyses. Later chapters explicitly consider these uncertainties in illustrative trade-off analyses.

1.5 Trade-off Analyses can Integrate Value and Risk Analysis

Program managers for the development of a new system must consider performance, cost, and schedule, as they are all interrelated. We know that performance problems can cause cost increases and schedule delays. Similarly, schedule changes can increase costs. Finally, cost estimate increases can result in reduced performance targets or schedule delays to make the system more affordable. Trade-off analysis, cost analysis, and risk analysis are frequently separate analyses performed by different analysts. Cost analysts typically perform a cost-risk analysis using Monte Carlo simulation. Many trade-off studies ignore uncertainty and risk.

A major theme of this book is that trade-off analyses should be used to identify both system value and system risks and that the analysis needs to be performed in a more integrated manner. In Chapter 9, we discuss and provide examples of how system value, system costs, and system risks can be integrated by identifying the system features that impact value, cost, and risk.

1.6 Trade-off Analysis in the Systems Engineering Decision Management Process

Successful systems engineering requires sound decision making. Many systems engineering decisions are difficult because they include multiple competing objectives, numerous stakeholders, substantial uncertainty, significant consequences, and high accountability. In these cases, sound decision making requires a formal decision management process. The purpose of the decision management process, as defined by ISO/IEC/IEEE 15288:2015, is “…to provide a structured, analytical framework for objectively identifying, characterizing and evaluating a set of alternatives for a decision at any point in the life cycle and select the most beneficial course of action.” The process presented in this book aligns with the structure and principles of the decision management process of ISO/IEC/IEEE 15288, the INCOSE Systems Engineering Handbook v4.0 (INCOSE, 2015), the Systems Engineering Body of Knowledge (SEBoK), and an INCOSE proceedings paper that elaborated this process (Cilli & Parnell, 2014). This process was designed to use best practices and to avoid the trade-off analysis mistakes discussed in the next section.

The INCOSE decision management process, introduced in Figure 1.2, is presented in more detail in Chapter 5. The purpose of the process is to “provide a structured, analytical framework for objectively identifying, characterizing, and evaluating a set of alternatives for a decision at any point in the life cycle and select the most beneficial course of action.” The white text within the outer green ring identifies elements of a systems engineering process while the 10 blue arrows represent the 10 steps of the decision management process. Interactions between the systems engineering process and the decision management process are represented by the small, dotted green (outer ring to inner ring) or blue arrows (inner ring to outer ring). (The reader is referred to the online version of this book for color indication.)

The steps in the decision management process are briefly described in Table 1.3, with references to the primary chapters that provide additional details about each step. Chapter 5 describes and illustrates the INCOSE decision management process.

1.7 Trade-off Analysis Mistakes of Omission and Commission

Using the INCOSE decision management process, we identify and discuss the most common trade-off study mistakes of omission and commission (Parnell et al., 2014). Mistakes of omission are errors made by not doing the right things, while mistakes of commission are errors made by doing the right things the wrong way. For each step in the decision process, Table 1.4 provides a list of trade-off mistakes, the type of mistake (omission or commission), and the potential impacts.

1.7.3 Impacts of the Trade-Off Analysis Mistakes

Earlier, we discussed mistakes of omission and commission as distinct errors. Any one of these errors can have significant consequences in system design and, ultimately, on the program and the system (Madni, 2010, 2011). Unfortunately, it is quite common to find multiple mistakes made in trade-off studies with some errors leading to, or cascading with, other errors. These cascading errors can lead to adverse impacts for the trade-off study team, decision-makers, stakeholders, and the system or program. In addition, repeating these trade-off mistakes can undermine the credibility of the SE organization/enterprise (Madni et al., 2005). In Figure 1.3, we show the dependencies among these errors. A dotted arrow represents correlation. A solid arrow represents high correlation. These dependencies show how the mistakes can cascade to the impacts described as follows.

1.7.3.1 Potential Selection of Poor Designs

The ultimate impact of mistakes made during the conduct of a system design trade-off study is the selection of poor designs. This includes missed opportunities to increase value and reduce risk. The selection of a poor design may then lead to significant impacts to cost, schedule, and technical performance. The primary sources of this shortcoming are poor decision frame, alternatives that fail to span the decision space, not taking the time to improve alternatives, not integrating trade-off studies with system/program risk assessments, failure to implement recommendations, and results not produced in timely fashion or imperfectly understood. The most obvious, but often neglected, cause of poor designs is poor alternatives. There are two primary causes of poor alternatives. The first is beginning with a limited set of alternatives that does not span the decision space and then not applying techniques to expand the alternative set (Madni et al., 1985; Madni, 2013). The second cause is not systematically trying to improve and expand the alternatives during the trade-off study. The identification and improvement of a good set of alternatives are sometimes impeded by the biases of the people involved. For example, there are often preconceptions of adequate solutions that limit the thought process from looking at the full decision space. The alternative generation table and decision analysis models provide excellent information for Value-Focused Thinking to improve the alternatives (Parnell et al., 2011).

1.7.3.2 Loss of Systems Engineering (SE) Credibility

The long-term organizational impact of trade-off study mistakes is the loss of SE credibility in the program, organization, or enterprise, and ultimately with the customer or acquisition organization. Nearly any of the trade-off study errors can result in a loss of SE credibility, since they impact the quality of the analysis and the resulting decision. Once the SE credibility is lost, it takes a significant effort to restore it.

1.7.3.3 No Trade-Off Studies Conducted or Trade-Off Studies Conducted on Wrong Issues

The impact of a poor problem definition (decision frame) can lead to no trade-off studies being conducted or trade-off studies being conducted on the wrong issues. Defining the problem(s) is difficult and time-consuming; however, it is critical. A great solution to the wrong problem may not be even a feasible solution to the real problem. If the error is not detected, it can lead to the selection of a poor design. The causes of poor problem definition are usually the lack of a clear problem statement and a poor framing of the decision. The typical cause for a poor decision frame is not having access to (or not listening to) decision-makers and stakeholders. While the decision frame can be improved late in the study, it will likely have schedule and cost impacts.

1.7.3.4 Loss of Trade-off Study Credibility

The loss of trade-off study credibility can usually be traced to multiple causes. The first is not having a sound mathematical foundation for the analysis. This typically results in not using normalized value functions and/or swing weights. The second is lack of confidence on the part of decision makers and/or stakeholders in the objectives and measures, which they might perceive as incomplete or not credible. Not showing the linkage of objectives to system functions can contribute to this problem. The third is the lack of good alternatives that span the decision space. A symptom of this problem is an advocacy trade-off analysis that advocates the team's preferred alternative and denigrates the others. The fourth is not identifying uncertainties and performing risk analysis. Trade-off studies can be improved or even reconducted late in the study but most likely not without adverse schedule and cost impacts.

1.7.3.5 Not Implementing Recommendations

Not having the trade-off study recommendations implemented can be a disheartening outcome. The reasons may lie with analysts, decision-makers, and/or program managers. The analyst may not have worked the right problem, may not have developed a credible model, may not have sensitivity/uncertainty/risk analysis to provide confidence in the decision, may not have communicated well, may not have provided an implementation plan, or may have delivered the study too late. The decision-maker may not have provided the analyst enough guidance to work on the right problem, may not have bothered to understand the rationale for the recommendation, or may have already made up their own mind about the decision before or during the study. The program manager may have failed to secure the requisite funds to implement the recommendations (Neches and Madni, 2013).

1.8 Overview of the Book

Figure 1.4 provides a graphical overview of the organization of this book. The first four chapters provide the foundations for understanding trade-off analyses. This chapter has described the need for trade-off analysis and identified errors of omission and commission to avoid. Chapter 2 provides the mathematical foundations for trade-off analyses; Chapter 3 provides a review of the probability theory, cognitive biases, and probability assessment necessary to understand and model uncertainty; and Chapter 4 provides techniques for conducting resource analyses that are essential to assess system affordability.

Chapter 5 presents the INCOSE decision management process for conducting a trade-off analysis, and provides a detailed illustrative example using some of the techniques illustrated in this book. Chapter 6 presents techniques for defining the opportunity space that frames the trade-off analysis. Chapter 7 describes how to identity and structure objectives and, then, how to develop value measures that quantify the attainment of these objectives. Chapter 8 provides commonly used techniques to generate and evaluate alternatives and compares the ability of these techniques to generate and evaluate alternatives. Chapter 9 provides an example of how we can perform an integrated model of value that can be used to perform value and risk trade-off analysis. The next five chapters discuss the life cycle stage context and provide examples of techniques that can be used to perform trade-off analyses in that stage. Since different types of information are available in the different life cycle stages, different trade-off analysis techniques are more appropriate. Chapter 10 discusses trade-off analysis that is conducted in the conceptual design phase. Chapter 11 discusses architecture trade-off analysis. Chapter 12 discusses design trade-off analysis. Chapter 13 discusses sustainment trade-off analyses. Chapter 14 discusses programmatic trade-off analyses (acceptance and retirement). Chapter 15 provides a summary and discusses future developments that may impact trade-off analyses.

1.9 Key Terms

1. Decision: A choice among alternatives that results in an allocation of resources.
2. Decision Management Process: “a structured, analytical framework for objectively identifying, characterizing, and evaluating a set of alternatives for a decision at any point in the life cycle and selecting the most beneficial course of action.” The decision management process uses the systems analysis process to perform the assessments (ISO/IEC/IEEE 15288, 2015).
3. Life Cycle Model: “An abstract functional model that represents the conceptualization of the need for the system, its realization, utilization, evolution, and disposal” (ISO/IEC/IEEE 15288, 2015).
4. Mistake of Omission: Mistakes of omission are errors made by not doing the right things.
5. Mistake of Commission: Mistakes of commission are errors made by doing the right things the wrong way.
6. Stakeholder: “An individual or organization having a right, share, claim or interest or in its possession of characteristics that meet their needs or expectations” (ISO/IEC/IEEE 15288, 2015).
7. System: “Systems are manmade, created and utilized to provide products or services in defined environments for the benefits of users and other stakeholders” (ISO/IEC/IEEE 15288, 2015).
8. Risk: “The effect of uncertainty on objectives” (ISO/IEC/IEEE 15288, 2015).
9. Uncertainty: Imperfect knowledge of the outcome of some future variable.
10. Trade-off: “Decision making actions that select from various requirements and alternative solutions on the basis of net benefit to the stakeholders” (ISO/IEC/IEEE 15288, 2015).
11. Trade-off Study: An engineering term for an analysis that provides insights to support system decision-making in a decision management process.
12. Value: The benefits provided by a product or service to the stakeholders (customers, consumers, operators, etc.).
13. Value Identification: The determination of the potential value of a new capability.
14. Value Realization: The delivery of the value of a new capability.

1.10 Exercises

1. 1.1 Trade-off analysis
   1. a. Provide a definition.
   2. b. Why are trade-offs needed?
   3. c. Who should participate in a trade-off analysis?
   4. d. Why is trade-off analysis different in different life cycle stages?
   5. e. Should trade-off analysis be performed to support life cycle gate decisions?
   6. f. Who should make the trade-off decisions?
2. 1.2 Decision management
   1. a. What is a decision management process?
   2. b. Why does an organization need a decision management process?
   3. c. What are the benefits of having a decision management process?
   4. d. What is the impact of not having a decision management process?
3. 1.3 Identify a system that is currently operational and answer the following questions:
   1. a. Identify the system stakeholders.
   2. b. Define value for the system.
   3. c. Describe the products and services of the system that provide value.
   4. d. Identify system risks.
   5. e. What is the anticipated life of the system?
   6. f. How difficult will it be to retire the system?
4. 1.4 Identify a system early in its life cycle and answer the following questions:
   1. a. Identify the system stakeholders.
   2. b. Define value for the system.
   3. c. Describe the products and services of the system that provide value.
   4. d. Identify potential system risks.
   5. e. What is the anticipated life of the system?
   6. f. List one major decision in each future life cycle stage that could require a trade-off analysis.
5. 1.5 Identify a system in development and answer the following questions:
   1. a. Describe the difference between value identification and value realization.
   2. b. What products and services will provide value?
   3. c. How are risks identified?
   4. d. What risks does the system have?
   5. e. What is the anticipated life of the system?
   6. f. List one major decision in each future life cycle stage that could require a trade-off analysis.
6. 1.6 Trade-off analysis mistakes
   1. a. Why are trade-off analysis mistakes made?
   2. b. Describe the difference between a mistake of omission and a mistake of commission.
   3. c. What is the reason that trade-off analysis mistakes cascade?
7. 1.7 Review a trade-off analysis paper in the proceedings of an engineering conference.
   1. a. How was value defined?
   2. b. What life cycle stage was considered?
   3. c. Were uncertainty and risk considered?
   4. d. Identify any mistakes of commission or omission in the paper.
   5. e. What trade-off analysis insights were provided?
   6. f. Was the trade-off analysis convincing?
   7. g. What decision was recommended or made?
8. 1.8 Review a published trade-off analysis paper in a refereed engineering journal.
   1. a. How was value defined?
   2. b. What life cycle stage was considered?
   3. c. Were uncertainty and risk considered?
   4. d. Identify any mistakes of commission or omission in the paper.
   5. e. What trade-off analysis insights were provided?
   6. f. What decision was recommended or made?
   7. g. Was the trade-off analysis convincing?

References

1. Buede, D.M. and Miller, W.D. (2009) The Engineering Design of Systems: Models and Methods, 2ndVol. Wiley Series in Systems Engineering edn, Wiley & Sons, Hoboken.
2. Cilli, M. and Parnell, G. (2014). Systems Engineering Tradeoff Study Process Framework. International Council on Systems Engineering (INCOSE) International Symposium. June 30–July 3. Las Vegas, NV.
3. Decision-making in an uncertain world: A Chevron Case Study. (2014). From Business Case Studies: http://businesscasestudies.co.uk/chevron/decision-making-in-an-uncertain-world/#axzz3NDPqSBNF (accessed 28 December 2014).
4. Hazelrigg, G.A. (1998) A framework for decision-based engineering design. Journal of Mechanical Design, 120 (4), 653–658.
5. INCOSE (2015) INCOSE systems engineering handbook v. 4.0. INCOSE, SE Handbook Working Group, INCOSE.
6. INCOSE Home Page. (2015). http://www.incose.org/: http://www.incose.org/ (accessed 06 June 2015).
7. ISO/IEC/IEEE (2015) Systems and Software Engineering - System Life Cycle Processes, International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC)/Institute of Electrical and Electronics Engineers (IEEE), Geneva, Switzerland.
8. Keeney, R.L. (1992) Value-Focused Thinking: A Path to Creative Decision Making, Harvard University Press, Cambridge, MA.
9. Keeney, R. and Raiffa, H. (1976) Decision with Multiple Objectives: Preference and Value Tradeoffs, Wiley & Sons, New York.
10. Kirkwood, C. (1997) Strategic Decision Making with Multiobjective Decision Analysis with Spreadsheets, Duxbury Press, Belmont, CA.
11. Lavingia, N. J. (2014). Business Success through Excellence in Project Management. Critical Facilities Roundtable: http://www.cfroundtable.org/ldc/040706/excellence.pdf (accessed 28 December 2014).
12. Madni, A.M. (2010) Integrating humans with software and systems: technical challenges and a research agenda. Systems Engineering, 13 (3), 232–245.
13. Madni, A.M. “Integrating humans with and within software and systems: challenges and opportunities,” (Invited Paper) CrossTalk, The Journal of Defense Software Engineering, May/June 2011, “People Solutions.”
14. Madni, A. (2012) Adaptable platform-based engineering: key enablers and outlook for the future. Systems Engineering, 15 (1), 95–107.
15. Madni, A.M. (2013) Generating novel options during systems architecting: psychological principles, systems thinking, and computer-based aiding. Systems Engineering, 16 (4), 1–9.
16. Madni, A.M. (2015) Expanding stakeholder participation in upfront system engineering through storytelling in virtual worlds. Systems Engineering, 18 (1), 16–27.
17. Madni, A.M., Brenner, M.A., Costea, I., MacGregor, D., and Meshkinpour, F. “Option Generation: Problems, Principles, and Computer-Based Aiding,” Proceedings of 1985 IEEE International Conference on Systems, Man, and Cybernetics, Tucson, Arizona, November, 1985, pp 757–760.
18. Madni, A.M., Sage, A., and Madni, C.C. “Infusion of Cognitive Engineering into Systems Engineering Processes and Practices,” Proceedings of the 2005 IEEE International Conference on Systems, Man, and Cybernetics, October 10–12, 2005, Hawaii.
19. Madni, A.M. and Sievers, M. (2014a) Systems integration: key perspectives, experiences, and challenges. Systems Engineering, 17.1, 37–51.
20. Madni, A.M. and Sievers, M. (2014b) System of systems integration: key considerations and challenges. Systems Engineering, 17.3, 330–347.
21. Madni, A. M. and Sievers, M. N. (2015). Model based systems engineering: motivation, current status and needed advances. Systems Engineering, Accepted for publication, 2016.
22. Madni, A.M. and Sievers, M. (2016) Model based systems engineering: motivation, current status and needed advances. Systems Engineering, Accepted for publication, 2016.
23. Neches, R. and Madni, A.M. (2013) Towards affordably adaptable and effective systems. Systems Engineering, 16 (2), 224–234.
24. Ordoukhanian, E. and Madni, A. M. (2015) System Trade-offs in Multi-UAV Networks. AIAA SPACE 2015 Conference and Exposition.
25. Parnell, G. (2009). Evaluation of risks in complex problems. in Making Essential Choices with Scant Information: Front-end Decision-Making in Major Projects (eds T. Williams, K. Sunnevåg, and K. Samset), Palgrave MacMillan, Basingstoke, UK, pp. 230–256.
26. Parnell, G.S., Bresnick, T.A., Tani, S.N., and Johnson, E.R. (2013) Handbook of Decision Analysis, Wiley & Sons.
27. Parnell, G., Cilli, M., and Buede, D. (2014). Tradeoff Study Cascading Mistakes of Omission and Commission. International Symposium. June 30–July 3. Las Vegas, NV: INCOSE.
28. Parnell, G.S., Driscoll, P.J., and Henderson, D.L. (eds) (2011) Decision Making for Systems Engineering and Management, 2nd edn, Wiley & Sons.
29. SEBoK. (2015). Systems Engineering Body of Knowledge (SEBoK) wiki page. SEBoK: http://www.sebokwiki.org (accessed 06 June 2016).
30. Tufte, E.R. (1983) The Visual Display of Quantitative Information, Graphics Press, Cheshire, CT.