Place of the Operations and Support Phase in the System Life Cycle

Before discussing the systems engineering activities during the operations and support phase, it should be noted that this phase is the concluding step of the system life cycle. The functional flow diagram of Figure 15.1 shows the inputs from the production phase to be operational documentation and a delivered system, and the outputs to be an obsolete system and a plan for disposing it in an appropriate way.

Systems Engineering in the Operations and Support Phase

During its operating life, a typical complex system encounters a number of different periods when its operation is interrupted. These incidents are represented in Figure 15.2. The abscissa is time, running from system delivery to its disposal. The ordinate represents the relative level of systems engineering involvement in the various events identified by the captions in the upper part of the figure. At the start, a column is seen symbolizing the installation and test period, which is shown to take substantial time (usually weeks or months) and a relatively large systems engineering effort. The four low, regularly spaced columns represent planned maintenance periods, which may require days of system downtime. The narrow spikes at irregular intervals are meant to correspond to random system breakdowns requiring emergency fault identification and repairs. These are usually fixed quickly but may take considerable systems engineering effort to find a solution that can be effected with minimum downtime. The large column on the right represents a major system upgrade, requiring a relatively long period (many months) and a high level of systems engineering. The latter may rival the effort involved in a new system development and may itself require a multiphase approach.

System Installation

The effort required for installing a delivered system at its operating site is strongly dependent on two factors: (1) the degree of physical and functional integration that had been accomplished at the production facility and (2) the number and complexity of the interfaces between the system and the operating site (including other interacting systems). In the case of an aircraft, for example, virtually all significant system elements typically are assembled and integrated at the prime contractor’s factory site so that when the aircraft leaves the production facility, it is ready for flight. The same is true for an automobile, a military truck, or almost any kind of vehicle.

The installation of many large-scale systems on a land or ship platform may be a major operation, especially if some of its subsystems are manufactured at separate locations by different contractors and are assembled only after delivery to the operating site. For example, an air terminal control system typically consists of several radars, a computer complex, and a control tower with an array of displays and communication equipment, all of which must be integrated to operate as a system and linked to the en route control system, runway landing control systems, and an array of associated equipment required to handle air traffic in and out of an airport. The installation and test of such a system is in itself a major systems engineering enterprise. Another such example is a ship navigation system, which consists of many subsystems that are manufactured at various sites, frequently by different contractors, and which have complex interfaces with ship elements. After the initial ship systems pass integration tests at a land site, subsequent production subsystems are often assembled and integrated only after they are delivered to the shipyard. The task of interfacing elements of the system with ship structure, power, controls, and communications is usually performed at the shipyard by experts in ship installations.

Nondisruptive Installation

Some critical systems require continuous operations and cannot be stopped or paused during system installation or upgrades. This tends to be the case when installing a system into a large system of systems. The installation of a new or upgraded system into the system of systems cannot disrupt current operations. Examples include system installation into a city power grid, a complex industrial wide area network, a national communications network, a major defense system of systems, and the national air traffic control system of systems. All of these examples require 24-hour operations without significant disruption.

Installing major systems into a system of systems without disruption requires careful planning and attention to detail. In the recent past, two general approaches have emerged to assist in this area: maintaining a system of systems simulation and maintaining a system of systems test bed. Figure 15.3 depicts the first option.

With this strategy, a system of systems simulation with hardware-in-the-loop is created. This simulation is typically user-in-the-loop as well, as opposed to stand alone. This simulation facility is verified and validated against actual data collected from the operational system of systems, which interacts with the environment. Typically, the simulation would not interact with the environment (although there are exceptions to this).

The system of systems simulation is used as a test bed to determine (1) the impact the new system will have on the system of systems before it is actually installed and (2) an installation strategy that will keep operations at an acceptable level. Once a strategy has been developed and verified using the system of systems simulation facility, knowledge and confidence is gained on how to install the system into the actual system of systems.

The advantage of this nondisruptive installation mode is the cost savings and the ability to model installation procedures and techniques before installing the system into the actual system of systems. The system of systems simulation facility, while expensive and complex, is only a representation of the actual system of systems and can be scoped to desired budget and tolerance levels. Obviously, if the system of systems in question is a defense network responsibility for the survival of a nation, extremely high tolerances would be required. However, if the system of systems is a business information technology (IT) network, tolerance may be relaxed to a comfortable risk level.

The second concept used within nondisruptive installations involves the development of a duplicate system of systems, scaled down from the operational one, and is depicted in Figure 15.4. The concept is similar to the first concept in that the system is installed into the scaled-down version of the system of systems, and testing occurs. During this process, the duplicate system of systems is typically disconnected from the operational system of systems to avoid any interference or disruption. An installation strategy is developed from the experience to apply for installation into the full-scale system of systems.

Once confidence in the risk of disruption is acceptable, the system is installed into the operational system of systems. Many times, the operational system of systems is disconnected from the environment—the duplicate system of systems is used as a surrogate during the installation. This is typically performed during a low demand situation or time frame to ensure the limited capacity of the duplicated system of systems is sufficient.

Although this strategy for nondisruptive installation is expensive (you are basically building a scaled-down version of the operation system of systems), it has two major benefits: (1) the duplicate system of systems is an architecture copy of the operational system of systems and is the closest representation that is possible without duplicating both the architecture and scale of the original; and (2) during peak demand, the duplicate, scaled-down system of systems can be used to augment the operational system of systems. National communications systems use this technique to keep its networks operational continuously, and to allow for unexpected peak demand periods.

Facilities and Personnel Limitations

Neither the facilities nor the personnel assigned to the task of system installation and test are normally equipped to deal with significant difficulties. Funds are inevitably budgeted on the assumption of success. And, while the installation staff may be experienced with the installation and test of similar equipment, they are seldom knowledgeable about the particular system being installed until they have gained experience during the installation of several production units. Moreover, the development contractor staff consists of field test engineers, while systems engineers are seldom assigned until trouble is encountered, and when it is, the time required to select and assign this additional support can be costly.

The lesson to be learned is that the installation and test part of the life cycle should be given adequate priority to avoid major program impact. This means that particular attention to systems engineering leadership in the planning and execution of this process is a necessity. This should include the preparation and review of technical manuals describing procedures to be followed during installation and operation.

Early System Operational Difficulties

Like many newly developed pieces of equipment, new systems are composed of a combination of new and modified components and are therefore subject to an excessive rate of component failure or other operational problems during the initial period of operation, a problem that is sometimes referred to as “infant mortality.” This is simply the result of the difficulty of finding all system faults prior to total system operation. Problems of this type are especially common at external system interfaces and in operator control functions that can be fully tested only when the system is completely assembled in an operational setting. During this system shakedown period, it is highly desirable that a special team, led by the user and supported by developer engineers, be assigned to rapidly identify and resolve problems as soon as they appear. Systems engineering leadership is necessary to expedite such efforts, as well as to decide what fixes should be incorporated into the system design and production, when this can best be done, and what to do about other units that may have been already shipped or installed. The need for rapid problem resolution is essential in order to effect necessary changes in time to resolve uncertainties regarding the integrity of the production design. Continuing unresolved problems can lead to stoppages in production and installation, resulting in costly and destructive impact on the program.

Operational Readiness Testing

Systems that do not operate continuously but that must be ready at all times to perform when called upon are usually subjected to periodic checks during their standby periods to ensure that they will operate at their full capability when required. An aircraft that has been idle for days or weeks is put through a series of test procedures before being released to fly. Most complex systems are subjected to such periodic readiness tests to ensure their availability. Usually, readiness tests are designed to exercise but not to fully stress all functions that are vital to the basic operation of the system or to operational safety.

All systems, sooner or later, will experience unexpected problems during operational use. This can occur when they encounter environmental conditions that were not known or planned for during development. Periodic system tests provide information that helps assess and resolve such problems quickly when they occur.

Periodic operational readiness tests also provide an opportunity to collect data on the history of the system operating status throughout its life. When unexpected problems occur, such data are immediately available for troubleshooting and error correction. System readiness tests have to be designed and instrumented with great skill to serve their purpose effectively and economically—a true systems engineering task.

Readiness tests often must be modified after system installation to conform more fully to the needs and capabilities of system operators and maintenance personnel. Development systems engineers can effectively contribute to such an activity. Location of data collection test points and the characteristics of the data to be collected, for example, data rate, accuracy, recording period, and so on, also represent systems engineering decisions.

Commonly Encountered Operational Problems

Field Service Support

It is common for deployed complex systems to require field support during the lifetime of the system. In the case of military systems, this is often provided by an engineering support unit within a branch of the service. It is also common for that unit to contract with civilian agencies to provide general engineering support to keep the system operating as intended.

When system operating problems are detected, it is necessary first to determine whether the problem is due to a fault in the operational system or is a result of improper functioning of a built-in fault indicator. For example, the device may be erroneously signaling a failure (false alarm) or may be ascribing it to the wrong function. Therefore, the field engineer who is called upon to troubleshoot a problem should be knowledgeable in system operation, including especially the functioning of built-in test devices.

When any fault is encountered during system operation, the required remedial actions are more difficult to implement than they would have been during development or even during installation and test. This is because (1) user personnel are not technical specialists; (2) special checkout and calibration equipment used during installation will have been removed; (3) most analysis and troubleshooting tools (e.g., simulations) are not available at the operational site; and (4) most knowledgeable people originally assigned to the development project are likely to have been reassigned, to have changed jobs, or to have retired. Because of these factors, for operational fixes to be done reliably, they often have to be developed remotely; that is, data will have to be collected at the operational site and transmitted back to the appropriate development site for analysis; corrective action will have to be formulated; and finally, the required changes will have to be implemented at the operational site by a special engineering team.

As noted previously, facilities at the developer’s test site are excellent locations for follow-on system work because of the availability of knowledgeable people, configuration flexibility, extensive data collection and analysis equipment, and the opportunity to carry out disciplined and well-documented tests and analyses.

Scheduled Maintenance and Field Changes

Most complex systems undergo periods of scheduled maintenance, testing, and often revalidation. Nonemergency field changes are best accomplished during such scheduled maintenance periods, where they can be carried out under controlled conditions by expert personnel and can be properly tested and documented. Fortunately, this usually accommodates the majority of significant changes. In most cases, as in that of commercial aircraft, such operations utilize special facilities with a full complement of checkout equipment, have a substantial parts stockpile and an automated inventory system, and are conducted by specially trained personnel.

Any changes, large or small, to an operational system require careful planning. As noted earlier, changes should be made under configuration control and should conform to documentation requirements that specifically state how they will be carried out. All changes should be viewed from a system perspective so that a change in one area does not cause new problems in other areas. Any technical change to an operational system will usually also require changes in hardware–software system documentation, repair manuals, spare parts lists, and operating procedure manuals. In this process, systems engineering is required to see that all issues are properly handled and to communicate these issues to those responsible for the overall operation.

Severe Operational Casualties

The previous paragraphs dealt with operational problems that could be corrected during operations or short periods of scheduled maintenance. It must be assumed that a complex system built to operate for a dozen years or more may accidentally suffer a failure of such magnitude that it is effectively put out of commission until corrected, such as by a fire, a collision, or through other major damage. Such a situation normally calls for the system to be taken out of service for the time necessary to repair and reevaluate it. However, before undertaking the drastic step of an extended interruption of service, a systems engineering team should be assembled to explore all available alternatives and to recommend the most cost-effective course of action for restoring operation. The severe casualty poses a classical system problem where all factors must be carefully weighed and a recovery plan developed that suitably balances operational requirements, cost, and schedule.

Logistics Support

The materials and processes involved in the logistics support of a major operational system constitute a complex system themselves. The logistics for a major fielded system may consist of a chain of stations, extending from the factory to the operational sites, which supplies a flow of spare parts, repair kits, documentation, and, when necessary, expert assistance as required to maintain the operating system in a state of readiness at all times. Technical manuals and training materials should be considered part of system support. The effort of developing, producing, and supporting effective logistics support for a major operating system can represent a substantial fraction of the total system development, production, and operating cost.

A basic problem in logistics support is that it must be planned and implemented on the basis of estimates of which system components (not yet designed) will need the most spare parts, what the optimum replacement levels will be for the different subsystems (not yet completely defined), what means of transportation, and hence time to resupply, will be available in potential (hypothetical) theaters of operation, and many other assumptions. These estimates can benefit from strong systems engineering participation and must be periodically readjusted on the basis of knowledge gained during development and operating experience. This means that logistic plans will need continual review and revision, as will the location and stocking level of depots and transport facilities.

There are also direct connections between the logistics support system and system design and production. The sources of most spare parts are usually the production facilities that manufacture the corresponding components and may include the system production contractor and the producers of system components. Moreover, subcomponents and parts commonly include commercial elements and hence are subject to obsolescence design changes or discontinued availability.

System field changes also directly affect the logistic supply of the affected components and other spare parts. Since the process of reflecting such changes in the logistics inventory cannot be instantaneous, it is essential to expedite it, as well as to maintain complete records of the status of each affected part wherever it is stored.

It can be seen from the above that the quality and timeliness of overall support provided by the logistic system will have direct effects on operability. This is particularly true for systems operating in the field, where the timely delivery of spare parts can be crucial to survival. In the case of commercial airlines, timely delivery of needed parts is also critical to maintaining schedules. Managing such a logistics enterprise is itself an enormous task of vital importance to the successful operational capability of the system.

System Upgrade Life Cycle

The development, production, and installation of a major system upgrade can be considered to have a mini life cycle of its own, with phases that are similar to those of the main life cycle. Active participation by systems engineering is therefore a vital part of any upgrade program.

Software Upgrades

As described in Chapter 11, software is much easier to change than hardware. Such changes usually do not require an extensive system stand-down or special facilities. With increasing system functionality being controlled by software, the pressure for software upgrades tends to make them considerably more frequent than major hardware upgrades.

However, to ensure that such operations are successful, special systems engineering and project management oversight is required to manage the difficulties inherent in system software changes:

1. 1. It is essential that the proposed changes be thoroughly checked out at the developer’s site before being installed into the operational software.
2. 2. The changes must be entered into the configuration management database to document the changed system configuration.
3. 3. An analysis must be performed to determine the degree of regression testing necessary to demonstrate the absence of unintended consequences.
4. 4. Operation and maintenance documentation must be suitably updated.

The above actions are required for any system change but are often neglected for apparently small software changes. It must be remembered that in a complex system, no changes are “small.”

Obsolescent legacy programs suffer from two disadvantages. First, the number of software support personnel willing to work on legacy software is diminishing and becoming inadequate. Second, modern high-performance digital processors do not have compilers that handle the legacy languages. On the other hand, the task of rewriting the programs in a modern language is comparable to the task of its original development and is generally prohibitively costly. This presents a difficult system problem for systems in the above position. Some programs have successfully used a software language translation to greatly reduce the cost of converting legacy programs to a modern language.

Preplanned Product Improvement (P³I)

For systems that are likely to require one or more major upgrades, a strategy referred to as P³I is often employed. This strategy calls for the definition during system development of a planned program of future upgrades that will incorporate a specified set of advanced features, thereby increasing system capabilities in particular ways.

The advantage of P³I is that changes are anticipated in advance so that, when needed, the planning is already in place; the design can accommodate the projected changes with minimum reconfiguration; and the upgrade process can proceed smoothly with minimum disruption to system operations. These preplanned changes will vary in magnitude and complexity depending on the need and availability of appropriate technology. Commercial airlines, for example, will often plan for a stretched version of an existing aircraft that will carry more passengers and incorporate larger engines and new control systems. By modifying an existing aircraft instead of developing a new one, the problems of government recertification can often be alleviated. In the military, the planned upgrade process has the advantage of prior mission justification. Since the current system is operational and performing a needed function, the proposed system changes will not affect already approved mission and system objectives.

In the case of future improvements defined during initial system development, the contract for implementing them is usually awarded to the development contractor. This is the most straightforward contractual arrangement for carrying out a major system upgrade. It is also most likely to secure the services of engineers familiar with the current system characteristics to participate in the planning and execution of system changes. While even in this case the original development team may have largely dispersed, that part that remains provides a major advantage by its knowledge of the system. However, as can sometimes occur in government-sponsored programs, the pressure for competition can become especially severe and can even lead to the selection of a different contractor team for the upgrade contract. In such cases, an intensive education program will be required for the new team to learn the finer points of the system environment and detailed operation.

Sources of Operational Knowledge

A number of potential sources of operational knowledge may be available in certain situations. These include operational tests of similar systems, integration testing during system installation, system readiness tests, and maintenance operations. These activities all address the problems associated with successfully integrating the system’s external interfaces with the site and with associated external systems. These can often expose serious problems that are not adequately revealed by the interface specifications provided to the developer.

To gain the necessary operational background, the systems engineer should endeavor to witness the operation of as many systems of the type under consideration as possible. Serving as an active participant in system test operations, or even by simply acting as an observer, is a good opportunity for learning. When present at such tests, the systems engineer should make the most of the opportunity by asking questions of system operators at appropriate times. Of special importance is information regarding what parts of the system are the sources of most problems and why. Learning about operational human–machine interfaces is particularly valuable because of the difficulty of realistically representing them during development.

Assistance from Operational Personnel

In view of the limited opportunities for the developer’s systems engineers to acquire an adequate level of operational expertise, it is often advisable to obtain the active participation of experienced operational personnel during system development. A particularly effective arrangement is when the user stations a team designated to be system operators at the development contractor’s facility during the period of systems engineering, integration, and test. These individuals bring knowledge of the special circumstances of the system’s interaction with the intended operational site, as well as represent the system operator’s viewpoint.

Another source of operational expertise comes from system maintenance personnel who are experienced in the problems of servicing similar systems at their operating sites and in their logistics support. Systems engineers can gain considerable knowledge by well-planned interviews with such individuals. As noted earlier, complex systems often have maintenance support facilities that may be excellent sources of operational knowledge.

Installing, Maintaining, and Upgrading the System

The application of systems engineering principles and expertise continue to be required throughout the operational life of the system. The operations and support phase includes installation and test, in-service support, and implementation of major system upgrades.

Interface integration and test can be challenging due to a mix of various organizational units, complex external interfaces, and incomplete or poorly defined interfaces.

Installation and Test

Installation and test problems can be difficult to solve because installation staff have a limited system knowledge. Systems engineers are seldom assigned until trouble is encountered. However, periodic operational readiness testing is necessary for systems that do not operate continuously. This can help minimize unexpected system problems.

Where nondisruptive installation is required, care to plan the installation procedures, via a hybrid simulation or a duplicate system operating in parallel, is absolutely essential.

In-Service Support

System software must be subject to strict configuration control to prevent serious deterioration of software quality. In this vein, built-in fault indicators are very valuable for detecting internal faults, although they sometimes produce false alarms. Therefore, field engineers should be knowledgeable about built-in test devices.

Remedial actions to correct operational problems are difficult to implement: operational personnel are not technical specialists. Furthermore, troubleshooting tools are limited. And materials and processes involved in logistics support themselves constitute a complex system.

Major System Upgrades: Modernization

Logistics cost is a large part of system cost. Therefore, P³I facilitates improvement of systems during major upgrades. Advanced features are defined during system development, and advanced planning permits minimum disruption to system operation.

Operational Factors in System Development

Possible sources of operational knowledge include operational and installation tests—by observing system operations within its environment. Of course, assistance from operational and maintenance personnel is invaluable.