Place of the Advanced Development Phase in the System Life Cycle

The advanced development phase marks the transition of the system development from the concept development to the engineering development stage. As seen in Figure 10.1, it follows the concept definition phase, from which it derives the inputs of system functional specifications and a defined system concept. Its outputs to the engineering design phase are system design specifications and a validated development model. It thus converts the requirements of what the system is to do and a conceptual approach of its configuration into a specification of generally how the required functions are to be implemented in hardware and software. Other required outputs, not shown in the figure, include an updated work breakdown structure (WBS), a revised systems engineering management plan (SEMP) or its equivalent, and related planning documents. Additionally, the system architecture is updated to reflect changes to date.

As noted above, this phase is especially critical in the development of complex systems that involve extensive use of advanced technology and/or novel unproven concepts. It may require several years of intensive development effort before the new features are sufficiently mature and well demonstrated to warrant initiating full-scale engineering. In addition, it may also be necessary to develop new manufacturing processes to support the proposed new technology. In such cases, the advanced development phase is frequently contracted separately from the follow-on engineering.

At the other end of the spectrum, those systems that do not involve major technological advances over similar previous systems, and hence require only a minor amount of development, may not have a separately defined and managed advanced development phase. Instead, the corresponding work may be included in the front end of the engineering design phase. However, the tasks embodied in the translation of the system functional requirements into a system implementation concept and system-level design specifications must still be accomplished prior to undertaking detailed engineering.

Design Materialization Status

Table 10.1 depicts the system materialization status during the advanced development phase. It is seen that the principal change in the system status is designated as “validation”—validation of the soundness of the selected concept, validation of its partitioning into components, and validation of the functional allocation to the component and subcomponent levels. The focus of development in this phase is thus the definition of how the components will be built to implement their assigned functions. The manner in which these tasks are accomplished is the subject of this chapter.

Systems Engineering Method in Advanced Development

The organization of this chapter is arranged according to the four steps of the systems engineering method (see Chapter 4) followed by a brief section that discusses risk reduction, a methodology used throughout system development, but is especially important in this phase. The principal activities during this phase in each of the four steps in the systems engineering method, as applied to those subsystems and components requiring development, are briefly summarized below and are illustrated in Figure 10.2.

System Functional Specifications

In defining the preferred system concept in the concept definition phase, the system functions were allocated to the principal subsystems, and these were further broken down into functional elements. These functional design concepts were then embodied in the system specifications document prepared as an input to the advanced development phase.

The analysis of these specifications should take into account the circumstances under which the concept definition phase took place. If, as is frequently the case, it was performed in the space of a few months and with limited funding, and especially if it was done in a competitive environment, then the results should be viewed as preliminary and subject to modification, and must be analyzed very thoroughly. Prior design decisions must be viewed with some skepticism until they are examined and demonstrated to be well founded. This does not mean that the selected technical approaches should necessarily be changed, only that they should not be accepted without understanding their derivation.

Requirements Derivation

The key to understanding the significance and sensitivity of system functional specifications is to trace them back to their derivation from the system performance requirements. Such an understanding is essential to making the design decisions required for the physical implementation of the functions in hardware and software.

The system life cycle support scenario should be revisited to identify functions necessary to sustain the different circumstances to which the system will be exposed during its preoperational as well as its operational life. In addition, the requirements for compatibility; reliability, maintainability, availability (RMA); and environmental susceptibility should be examined, as well as those for operational performance. At this time, specifications concerning human–system interface issues and safety are incorporated into subsystem and component specifications.

As stated previously, some requirements are frequently unstated, and others are immeasurable. For example, affordability and system growth potential are frequently not explicitly addressed. User interface requirements are often qualitative and are not susceptible to measurement. The relation of each of the above issues to the functional design needs to be understood and documented.

Relation to Operational Requirements

If some system specifications cannot be readily met, it is necessary to gain an even deeper understanding of their validity by tracing them back one step further, namely, to their relationship to the execution of the system’s mission, that is, to the system operational requirements. This relationship is often lost in the early phases of system definition and needs to be recaptured to provide the systems engineer with an informed rationale for dealing with problems that invariably arise during development.

One of the means for gaining such understanding, as well as for obtaining an appreciation of operational factors beyond those formally stated, is to develop contacts with prospective system users. Such contacts are not always available, but when they are, they can prove to be extremely valuable. Organizations that specialize in operational analyses and those that conduct system field evaluations are also valuable sources in many system areas. Involving the user as a team member during development should be considered where appropriate.

Relation to Predecessor Systems

If the new system has a predecessor that fulfills a similar function, as is usually the case, it is important to fully understand the areas of similarity and difference, and how and why the new requirements differ from the old. This includes the understanding of the perceived deficiencies of the predecessor system and how the new system is intended to eliminate them.

The degree of benefit to be gained from this comparison, of course, depends on the accessibility of key persons and records from the predecessor system development. However, at a minimum, the comparison should provide added confidence in the chosen approach or suggest alternatives to be explored. Where key participants in the development are accessible, their advice with regard to potential problems and lessons learned can be invaluable.

Identification of Components Requiring Development

The principal purpose of the advanced development phase has been stated as ensuring that all components of the system have been demonstrated to be ready for full-scale engineering. This means that component design is sound and capable of being implemented without significant risk of functional or physical deficiencies that would require different approaches to satisfy the requirements.

The above statement implies that all system components must be brought up to a level of maturity where all significant design issues have been resolved. The process that raises the level of maturity is called “development,” and therefore the advanced development phase consists largely of development effort focused on those system components that have not previously been brought to the necessary level of proven performance. This, in turn, means that all components that are determined to be insufficiently mature for full-scale engineering would be further developed and their design validated. Those components that are deemed to be sufficiently mature that they do not require development still need to be validated through analysis or test prior to their acceptance for engineering.

Example: Unproven Components.

Table 10.2 illustrates the above considerations by listing several representative examples of hypothetical unproven components that use new functional or physical design approaches or new production methods. The first column indicates the relative maturity of the functional, physical, and production characteristics of the design approach. The second column is a bar graph representing the maturity of these three characteristics (names abbreviated) by the relative heights of the three vertical bars. The third column shows the type of development that is usually appropriate to resolving the resulting issues of each of the new designs. The fourth column lists the particular characteristic to be validated. These examples are, of course, very much simplified compared to the factors that must be considered for an actual complex system, but they indicate the component-by-component analysis and planning that is associated with the advanced development process. The table illustrates that components in a new system may have a variety of different types of unproven features, each requiring a development approach tailored to its specific character. The decisions as to the choice of development strategy are the primary responsibility of systems engineering. The subsequent three sections describe the application of each of the remaining steps of the systems engineering method to the resolution of the above design issues.

Extended Functional Performance

The identification of system elements (components or subsystems) whose required performance may exceed demonstrated limits can be illustrated by reference to the set of functional system building blocks discussed in Chapter 3. Table 3.2 lists 23 basic functional elements grouped into four classes: signal, data, material, and energy. Each functional element has a number of key characteristics that define its functional capability. Most of these characteristics have limits established by the physical properties of their implementing technologies and often by the basic interdependence between functions (e.g., accuracy vs. speed). A functional requirement for a new system that poses demands on a system element beyond its previously demonstrated limits signals the potential need for either a component development effort or a reallocation of the requirement.

To illustrate this type of comparison, Table 10.3 lists the functional elements along with some of the characteristics that most often turn out to be critical in new systems. The table represents the application of the systems engineering approach to the analysis of system functional requirements and the identification of development objectives.

In using system building blocks to identify functional elements requiring development, the first step is to relate each system element to its functionally equivalent generic element and then to compare the required performance with that of corresponding physical components whose capabilities have been demonstrated as a part of existing systems.

Given an approximate correspondence, the next step is to see whether the differences between the required and existing elements can be compared quantitatively by established engineering relations so as to make a convincing case that the new element can be engineered with confidence, on the basis of proven performance and straightforward engineering practice. When such a case cannot be made, it is necessary either to reduce the specified performance requirement to a level where it can be so adapted or to plan a development and test program to obtain the necessary engineering data.

The process of identifying elements requiring development is often part of the process of “risk identification” or “risk assessment.” Risk assessment considers the likely effect of a given decision, in this case, the choice of a particular technical approach, on the success or failure of the overall objective. Thus, the utilization of unproven system components involves a degree of risk depending on the likelihood that the system will fail to meet its design goals. If the risk is considerable, as when the element is both unproven and critical to the overall system operation, then the element must be developed to a point where its performance may be demonstrated and validated (i.e., low risk). The subject of risk management is discussed in Chapter 5 and is encountered in all phases of the system life cycle.

Highly Complex Components

Consideration of the functional building blocks as system architectural components is also useful in identifying highly complex functions. Equally important is to identify complex interfaces and interactions because elements of even moderate complexity may interact with one another in complicated ways. Interfaces are especially important because complexities internal to elements are likely to be detected and resolved during design, while problems resulting from interface complexities may not reveal themselves until integration testing, at which time changes required to make them operate properly are likely to be very costly in time and effort. The existence of excessively complicated interfaces is a sign of inappropriate system partitioning and is the particular responsibility of the systems engineer to discover and to resolve. This concern is particularly important when several organizations are involved in the system development.

Ill-Defined System Environments

Poorly defined system environments and imprecise external interface requirements are also design issues that must be carefully examined and clarified. For example, a radar system designed to detect targets in the presence of clutter due to weather or surface returns is impossible to characterize in a well-defined fashion due to the great diversity of possible operational and environmental conditions and the limited understanding of the physics of radar scattering by clutter and of anomalous radar propagation. Similarly, space environments are difficult to understand and characterize due to the limited data available from past missions. The expense of placing systems into the space environment means testing and operational data are not as prevalent as atmospheric data.

The operation of user-interactive systems involves the human–machine interface, which is also inherently difficult to define. The parts of the system that display information to the user and that accept and respond to user inputs are often relatively uncomplicated physically but are very intricate logically. This complexity operates at several levels, sometimes beginning with the top-level objective of the system, as in the conceptual design of a medical information system where the needs of the physicians, nurses, clerical staff, and others that interact with the system tend to be not only ill-defined but also highly variable and subject to argument. At lower levels, the form of the display, the format of information access (menu, commands, speech, etc.), portability, and means of data entry may all constitute system design issues that are not likely to be settled without an extensive testing of alternatives.

The design of automobile air bags represents another type of component with a complex environmental interface that has required extensive development. In this case, the conditions for actuating the air bag had to be explored very thoroughly to establish a range between excessively frequent (and traumatic) false alarms and assured response to real collisions. The shape, size, and speed of inflation and subsequent deflation of the air bag had to provide maximum safety for the individual with minimum chance for injury by the force of inflation of the bag. This example is representative of system–environment interactions that can only be accurately defined experimentally. It also illustrates a system component whose operational and functional performance cannot be separated from its physical implementation.

Functional Design

Beyond identifying system elements requiring further development, the functional design and integration of the total system and all its functional elements must be completed during this phase. This is a necessary step to developing the system design specifications, which are a prerequisite to the start of the engineering design phase.

Use of Simulations

While many of the above problem areas require resolution by prototyping actual hardware and software, a number of others can be effectively explored by simulation. Some examples are the following:

• Dynamic Elements. Except for very high frequency dynamic effects, most system dynamics can be simulated with adequate fidelity. The six-degree-of-freedom dynamics of an aircraft or missile can be explored in great detail.
• Human–Machine Interfaces. User interfaces are control elements of most complex systems. Their proper design requires the active participation of potential users in the design of this system element. Such participation can best be obtained by providing a simulation of the interface early in the development and by enhancing it as experience accumulates.
• Operational Scenarios. Operational systems are usually exposed to a variety of scenarios that impact the system in different ways. A simulation with variable input conditions is valuable in modeling these different effects well before system prototypes or field tests can be conducted.

Potential Problem Areas

In looking for potential problems, it is essential to examine the entire system life cycle—engineering, production, storage, operational use, and operational maintenance. Special attention must be devoted to manufacturing processes, the “ilities” (RMA), logistic support, and the operational environment. The approach is that of risk assessment: what risks may be involved at each phase and where are the unknowns such as areas in which prior experience is scanty? For each potential risk, the likelihood and impact of a failure in that area must be determined.

As in the case of functional characteristics, the most likely areas where proposed component implementation may be significantly different from previous experience can be classified in four categories:

1. 1. components requiring unusually stringent physical performance, such as reliability, endurance, safety, or extremely tight manufacturing tolerances;
2. 2. components utilizing new materials or new manufacturing methods;
3. 3. components subjected to extreme or ill-defined environmental conditions; and
4. 4. component applications involving unusual or complex interfaces.

Examples of each of these categories are discussed below.

Component Design

The previous sections described a number of criteria that may be used to identify components that require development effort to bring their design to a level of maturity sufficient to qualify them for full-scale engineering. Such development effort involves some combination of analysis, simulation, design, and testing according to the specific nature of the proposed design approach and its departure from proven practice.

The extent of development required may, naturally, vary widely. At one extreme, the design may be taken only to the stage where its adequacy can be verified by inspection and analysis. This may be done for components whose departure from their predecessors is mainly related to size and fit rather than to performance or producibility. At the other extreme, components for which the validation of new materials, or the verification of stringent production tolerances (or other characteristics of the production article), are required may need to be designed, constructed, and extensively tested. Here again, the decisions involve systems engineering trade-offs between program risk, technical performance, cost, and schedule.

Design Testing

The process of component design is iterative, just as we have seen the system development process to be. This means that testing must be an integral part of design rather than just a step at the end to make sure it came out properly. This is especially true in the design of components with new functionality or those utilizing unproven implementation approaches. The appropriate process in such cases is “build a little, test a little,” providing design feedback at every step of the way. This may not sound very orderly but is often the fastest and most economical procedure. The objective is to validate the large majority of design elements at lower levels, where the results are more easily determined in less complex test configurations and errors corrected at the earliest time.

As stated earlier, the degree of completion to which the design of a given component is carried during this phase is very much a function of what is required to ensure a sound basis for its subsequent engineering. Thus, if a component’s design issues are largely functional, they may be resolved by comparative simulation to establish which will best fulfill the required functional needs of the system. However, if the design issues relate to physical characteristics, then the component usually needs to be designed and built in prototype form, which can then be tested in a physical environment simulating operational conditions. The design of such tests and of the corresponding test equipment will be discussed in the next section.

Rapid Prototyping

This is a term describing the process of expedited design and building of a test model of a component, a subsystem, and sometimes the total system to enable it to be tested at an early stage in a realistic environment. This process is employed most often when the user requirements cannot be sufficiently well defined without experimenting with an operating model of the system. This is particularly true of decision support systems, dynamic control systems, and those operating in unusual environments. Rapid prototyping can be thought of as a case of carrying development to a full-scale demonstration stage prior to committing the design to production engineering.

When engaging in rapid prototyping, the term “rapid” means that adherence to strict quality standards, normally a full part of system development, is suspended. The goal is to produce a prototype that features selected functionality of the system for demonstration as quickly as possible. The article that is produced is not intended to survive—once requirements are developed and validated using the prototype, the article itself should be discarded. At times, the prototype article is used as a basis for another iteration of rapid prototyping. The risk in this process is that eventually, the pressure to use the prototype article as the foundation for the production article becomes too great. Unfortunately, because the prototype was developed without the strict quality standards, it is not appropriate for production.

Examples abound where rapid prototyping was engaged and a prototype article was developed without quality controls (e.g., development standards, docu­mentation, and testing). Unfortunately, the customer deems the article sufficient and requires the developer to provide the article for production (after all, the customer paid for the prototype—he owns it!). Once production starts, the flaws in this process quickly become evident, and the system fails its development and operational testing. In the end, development and production cause slippages in schedule and overruns in cost.

Rapid prototyping was pioneered in software development and will be discussed further in the next chapter.

Development Facilities

A development facility or environmental test facility, as referred to here, is a physical site dedicated to simulating a particular environmental condition of a system or a part thereof in a realistic and quantitative manner. It is usually a fixed installation capable of use on a variety of physical and virtual models (or actual system components with embedded software) representing different systems or components. It can be used for either development or validation testing, depending on the maturity of the system/component subjected to the environment. Such facilities contain a set of instrumentation to control the simulated environment and to measure its effects on the system. They may be used in conjunction with a system simulation and usually have computing equipment to analyze and display the outputs.

A development facility usually represents a substantial investment; it is often enclosed in a dedicated building and/or requires a significant amount of real estate. A wind tunnel is an example of a facility used to obtain aerodynamic data. It contains a very substantial amount of equipment test chambers, air compressors, precise force measuring devices, and data reduction computers and plotters. Often the cost to build and operate a wind tunnel is so high that support is shared by a number of commercial and government users. When a wind tunnel is used to obtain data on a number of candidate aerodynamic bodies or control surfaces, it can be thought of as a development tool; when it is used to supply a source of high-speed airflow to check out a full-scale airplane control surface, it serves as a validation test facility.

Automobile manufacturers use test tracks to help design and test new model cars and to prove-in the final prototypes before production begins. Test tracks can simulate various wear conditions under accelerated aging, for example, by driving heavily loaded cars at high speed or over rough pavement. Other development facilities use electromagnetic radiation to test various electronic devices, for example, to measure antenna patterns, to test receiver sensitivity, to check for radio frequency (RF) interference, and so on.

Most development facilities use some form of models and simulations when conducting tests. It is common for some part of the system under test to be the actual article while other parts are simulated. An RF anechoic chamber that tests a tracking device in the presence of various RF interference signals is an example. In this case, the flight of the vehicle can also be simulated by a computer, which solves the equations of motion using appropriate aerodynamic and dynamic models of the system.

The engineering design of hardware components that are subjected to external stresses, high temperatures, and vacuum conditions in space requires the extensive use of stress testing, environmental chambers, and other special test facilities. The same facilities are also used in the development of these components. Thus, shake and shock facilities, vacuum chambers, hot and cold chambers, and many other engineering test facilities are as necessary in the development as in the engineering phases. The main difference is that development testing usually requires the acquisition of more performance data and more extensive analyses of the results.

Test and Test Analysis Plans

An essential but sometimes insufficiently emphasized step in the advanced development process is the development of a well-designed test plan for determining whether or not the system design is sufficiently mature to proceed to the engineering design phase.

Special Test Equipment and Test Facilities

It has been noted in previous chapters that the simulation of the system operational environment for purposes of system test and evaluation can be a task of major proportions, sometimes approaching the magnitude of the system design and engineering effort itself. In the advanced development phase, this aspect of system development is not only very important but frequently is also very expensive. Thus, the judgment as to the degree of realism and precision that is required of such simulation is an important systems engineering function. This and related subjects are also discussed in Chapter 13.

The magnitude of the effort to provide suitable test equipment and facilities naturally depends on the nature of the system and on whether the developer has had prior experience with similar systems. Thus, the development of a new spacecraft requires a host of equipment and facilities ranging from vacuum chambers and shake and vibration facilities that simulate the space and launch environment, and space communication facilities to send commands and receive data from the spacecraft, to clean rooms that prevent contamination during the building and testing of the spacecraft. Some of these facilities were described in the subsection on development facilities. Having a full complement of such equipment and facilities enables an established spacecraft developer to limit the cost and time for developing the necessary support for a new development. However, even if the bulk of such equipment may be available from previous system developments, every new program inevitably calls for different equipment combinations and configurations. The rate of technological change creates both new demands and new opportunities, and this is no less true in the area of system testing than in the area of system design.

Creating the Test Environment.

The design and construction of the test environment to validate a major component or subsystem requires equipment for the realistic generation of all the input functions and the measurement of the resulting outputs. It also requires the prediction and generation of a set of outputs representing what the system element should produce if it operates according to its requirements. The latter, in turn, requires the existence of mathematical or physical models designed to convert the test inputs into predicted system outputs for comparison with test results.

The above operations are represented by a functional flow diagram (Figure 10.3) that is an expansion of the test and evaluation block of Figure 8.2. The four functions on the left side of the figure show how the design of the test environment creates a predictive test model and a test scenario, which in turn activates a test stimulus generator. The test stimuli activate the system element (component or subsystem) under test and are also used by the mathematical or physical model of the system element to create a corresponding set of predicted outputs for comparison with the actual test outputs. The functions on the right side of Figure 8.3 represent the analysis and evaluation of test results, as further described below in a subsequent subsection bearing that name.

Demonstration and Validation Testing

The actual conduct of tests to demonstrate and validate the system design is often the most critical period in the development of a new system. The primary effort during advanced development has been seen to be concerned with the resolution of identified design issues—in other words, eliminating the known unknowns or “unks.” And, with luck, it will succeed in resolving the great majority of the initial uncertainties in the system design. But every new complex system inevitably also encounters unanticipated “unknown unknowns,” or “unk-unks.” Thus, it is also a major objective of the advanced development phase to discover such features before committing to full-scale engineering. To this end, the validation tests are designed to subject the system to a broad enough range of conditions to reveal hitherto undiscovered design deficiencies.

Analysis and Evaluation of Test Results

The operations involved in evaluating test results are illustrated in the right half of Figure 10.3. The outputs from the component or subsystem under test are either recorded for subsequent analysis or compared in real time with the predicted values from the simulated element model. The results must then be analyzed to disclose all significant discrepancies, to identify their source, and to assess whether or not remedial measures are called for, as derived with reference to a set of evaluation criteria. These criteria should be developed prior to the test on the basis of careful interpretation of system requirements and understanding of the critical design features of the system element.

It should be noted that one of the first places to look for as a cause of a test discrepancy is a defect in the test equipment or procedure. This is largely because there is usually less time and effort available to validate the test setup than has gone into the design of the system element under test.

The successful use of test results to either confirm the design approach or to identify specific design deficiencies is wholly dependent on the acquisition of high-quality data and its correct interpretation in terms of system requirements. An essential factor in effective test analysis is a versatile and experienced analysis team composed of analysts, test engineers, and systems engineers. The function of the analysts is to apply analytical tools and techniques to convert the raw test results to a measurement of the performance of specific system elements. The test engineers contribute their intimate knowledge of the test conditions, sensors, and other test variables to the systems analysis. The systems engineers apply the above knowledge to the interpretation of the tests in terms of system performance as related to requirements.

Tracing deficiencies in performance to the stated system requirements is especially important when remedying the deficiencies may require significant redesign. In such cases, the requirements must be critically reviewed to determine whether or not they may be relaxed without significant loss in system effectiveness, in preference to expending the time and cost required to effect the system changes required to meet them fully. In view of the potential impact of any deficiencies uncovered in the test analysis process, it is essential that the analysis be accomplished quickly and its results used to influence further testing, as well as to initiate such further design investigations as may be called for.

Evaluation of User Interfaces

A special problem in the validation of system design is posed by the interface and interaction between the user/controller and the system. This is especially true in decision support systems where the system response is critically dependent on the rapid and accurate interpretation of complex information inputs by a human operator aided by displays driven by computer-based logic. The air traffic controller function is a prime example of such an interface. However, even in much less information-intensive systems, the trends toward increased automation have made user interfaces more interactive and hence more complex. Even the basic interface between a personal computer and the user, while becoming more intuitive and powerful, has nevertheless become correspondingly more complicated and challenging to the nonexpert user.

The test and evaluation of user interface controls and displays poses difficult problems because interfaces are inherently incapable of objective quantitative measurement, except in their most primitive features (e.g., display luminosity). Large variations in the experience, visual and logical skills, and personal likes and dislikes of individual users also color their reactions to a given situation. Moreover, it is essential that members of the design team do not serve as sole subjects for the assessment of user interfaces. Rather, to the maximum extent possible, operators of similar systems should be employed for this purpose.

Nonetheless, the importance of an effective user interface to the performance of most systems makes it essential to plan and conduct the most substantive evaluation of this systems feature as may be practicable. This is especially relevant because of the inherent difficulty of establishing user requirements at the outset of the development. Thus, there are bound to be surprises when users are first confronted with the task of operating the system.

User interfaces are areas where rapid prototyping can be particularly effective. Before the full system or even the full human–computer interface is designed, prototypes can be developed and demonstrated with potential users to solicit early feedback on preferences of information representation.

The evaluation of the user interface may be considered in four parts:

1. 1. ease of learning to use the operational controls,
2. 2. clarity of visual situational displays,
3. 3. usefulness of information content to system operation, and
4. 4. online user assistance.

Of these, the first and last are not explicitly parts of the basic system operation, but their effectiveness can play a decisive role in the user’s performance. It is, therefore, important that sufficient attention be paid to user training and basic user help to ensure that these factors do not obscure the evaluation of the basic system design features.

Even more than most other design characteristics, user interfaces should be tested in anticipation of discovering and having to fix inadequacies. To this end, wherever practicable, users should be presented with design alternatives to choose from rather than having to register their level of satisfaction with a single design option. This may usually be accomplished in software rather than in hardware.

As in the case of other operational characteristics, such as reliability, producibility, and so on, the design related to the human–machine interface should have involved human factor experts as well as potential users. For the developer to obtain the participation of the latter, it may be necessary to obtain customer assistance. In these and other cases, customer participation in the development process can materially enhance the utility and acceptance of the final product.

The evaluation of the effectiveness of the user interface is not subject to quantitative engineering methods and extends the systems engineer into the field of human–machine interaction. The experts (usually psychologists) in certain aspects of such interactions are mostly specialists (e.g., in visual responses) and must be integrated into the evaluation process along with other specialists. It is a systems engineering responsibility to plan, lead, and interpret the tests and their analysis in terms of what system design changes might best make the user most effective. To do so, the systems engineer must learn enough of the basics of human–machine interactions to exercise the necessary technical leadership and system-level decision making.

Correction of Design Deficiencies

All of the previous discussions have centered on discovering potential deficiencies in the system design that may not have been eliminated in the development and test process. If the development has been generally successful, the deficiencies that remain will prove to be relatively few, but how to eliminate them may not always be obvious nor the effort required trivial. Further, there is almost always little time and few resources available at this point in the program to carry out a deliberate program of redesign and retest. Thus, as noted earlier, there must be a highly expedited and prioritized effort to quickly bring the system design to a point where full-scale engineering can begin with a relatively high expectation of success. The planning and leadership of such an effort is a particularly critical systems engineering responsibility.

How Much Development?

A key decision that must be made in planning the risk reduction effort is by what means and how far each risk area should be developed. If the development is too limited, the residual risk will remain high. If it is very extensive, the time and cost consumed in risk reduction may unnecessarily inflate the total system development cost. Striking the proper balance calls on the exercise of expert systems engineering judgment.

The decision as to how much development should be undertaken on a given component should be part of the risk management plan, as described in Chapter 5. The objective of the plan is to minimize the total cost of managing each significant risk area. This “risk cost” is the sum of the cost of such analysis, simulation, and design and testing that may be undertaken, that is, the “development cost,” and the cost of mitigating the residual risk to the low level required to proceed to the engineering design phase, that is, the “mitigation cost.” By varying the nature and amount of development, a judgment can be made as to the most favorable balance. Thus, for a critical, immature component, the balance may call for development up to the prototype stage, while for a noncritical or mature component, it would only call for analysis.

Reducing Program Risks

Objectives of the advanced development phase are to resolve the majority of uncertainties (risks) through analysis and development and to validate the system design approach as a basis for full-scale engineering. The outputs of advanced development are a system design specification and a validated development model.

Advanced development is especially critical for systems containing extensive advanced development or unproven concepts that may involve several years of development effort.

Activities encompassed by advanced development are the following:

• Requirements Analysis— relating functional requirements to needs,
• Functional Analysis and Design— identifying performance issues,
• Prototype Development— building and testing prototypes of critical components, and
• Test and Evaluation— validating the maturity of critical components.

Requirements Analysis

Analysis of system functional specifications is required to relate them to their origin in operational requirements, especially those not readily met. Their differences from those of a predecessor system are also noted.

Functional Analysis and Design

Components that may require further development include those that

• implement a new function;
• are a new implementation of an existing function;
• use a new production method for an existing type of component;
• extend the function of a proven component; and
• involve complex functions, interfaces, and interactions.

Prototype Development as a Risk Mitigation Technique

Program risks requiring development may result from a number of conditions:

• unusually high performance requirements,
• new materials and processes,
• extreme environmental conditions,
• complex component interfaces, and
• new software elements.

Development Testing

Validation testing to confirm the resolution of risks requires the development of a formal test plan (TEMP). Furthermore, test equipment must be developed; validation tests must be conducted; and test results must be analyzed and evaluated. The results of this testing lead to the correction of design deficiencies. However, special test equipment and facilities often represent a major investment. Therefore, early experimental exploration of the interface design is essential.

Models of systems and components are used extensively in system development. Simulations are increasingly important in all stages of development and are essential in the analysis of dynamic systems and software that require development and a staff of analysts and operators.

Development facilities are installations simulating environmental conditions and are used for development tests and component evaluation. They represent a major investment and require a permanent operating staff.

Risk Reduction

Risk assessment is a basic systems engineering tool, which is used throughout development, but especially during advanced development. It involves identifying sources of risk, risk likelihood, and criticality.