,

CHAPTER 7

COST ASSESSMENT

The accurate prediction of cost, including effort and schedule, required to accomplish various systems engineering activities in an organization is very important. Similarly, it is necessary to know the effectiveness that is likely to result from given expenditures of effort. When the costs of activities to be undertaken are underestimated, programs may encounter large cost overruns, which may lead to embarrassment for the organization. Associated with these cost overruns are such issues as delivery delays and user dissatisfaction. When costs are overestimated, there may be considerable reluctance on the part of the organization or its potential customers to undertake activities that could provide many beneficial results. Thus, cost is an important ingredient in risk management. In this chapter, we will examine costing in systems, software, and information technology development.

It is very important to note that a cost estimate is desired to predict or forecast the actual cost of developing a product or providing a service. There are many variables that will influence cost. The product or service scope, size, structure, and complexity will obviously affect the costs to develop it. The newness of the product or service to the development team will be another factor. The stability of the requirements for the product or service will be an issue as high requirements volatility over time may lead to continual changes in requirements and specifications that must be satisfied throughout the entire systems acquisition life cycle.

The integration and maintenance efforts that will be needed near the end of the acquisition life cycle surely influence costs. Something about each of these and other factors must necessarily be known to obtain a cost estimate. The more detailed and specific the definition of the product is, the more accurate we should expect the costs to be relative to the estimated costs for development and deployment of the product. One guideline we might espouse is to delay cost estimation until as late as possible in the product or service definition phase, and perhaps even postpone it until some development efforts have become known. If we wait until after deployment to obtain cost estimates, we should have an error-free estimate of the costs incurred to produce a product or system or service. This approach is hardly feasible or useful, as estimates need to be known early in the definition phase to determine whether it is realistic to undertake production.

Operating costs, once a solution is deployed, are important too. Operating costs can amount to 75% to 80% of the total costs of ownership. They can also be highly variable depending on the frequency of system use, the economy, inflation and wages, etc. Chapters 8 and 9 deal with operating costs to an extent, including uncertainties. Rouse (2010) addresses this issue in great depth as the economics of human systems integration are often dominated by these costs. Chapter 10 provides an overview of this material.

We also see that there is merit in decomposing an acquisition or production effort into a number of distinct components. A natural model for this is the life-cycle phases for production. We might, for example, consider the seven-phase software development life cycle of Boehm (1976a, b) or any of the systems engineering life-cycle models and develop a work breakdown structure (WBS) of the activities to be accomplished at each of these phases. The work elements for each of these phases can be obtained from a description of the effort at each phase, where we will concentrate on software effort:

The phases enumerated above normally take place in the sequential manner described. However, considerable iteration and interaction occur between the several phases. Each software development organization tailors the process to meet the particular characteristics of the personnel of the organization and, potentially, the needs of users of the software product.

Figure 7.1 illustrates a hypothetical WBS for this development life cycle. We could base our estimates of cost on the detailed activities represented by this WBS, and a realistic WBS would be much more detailed than this, or we could develop a macrolevel model of development cost that is influenced by a number of primary and secondary drivers. One often-used approach is called activity-based costing (ABC). An unstated assumption in the development of Fig. 7.1 is that we are going to undertake a grand design type approach for system production. This figure and the associated discussion need some modification for any approach other than the grand design type approach.

We will examine model-based costing (MBC) and ABC approaches in this chapter. We will first discuss some generic issues relating to system and software productivity. This will include the nature of the variables that are important for these measures. Then, we will consider some early approaches. After this, we will examine some MBC approaches. Next, we will look at ABC approaches. Finally we will discuss some issues related to model and system evaluation. Much of our discussion concerns the engineering of information technology, systems, and services. In the end, we turn to a brief discussion of effectiveness.

7.1 MODEL-BASED PARAMETERS FOR PRODUCTIVITY

There are a number of ways in which we could define software productivity. We might be tempted to say that software productivity is equivalent to the number of lines of source code that are delivered to the customer per person-month (PM) of effort. We immediately sense that a real problem with this definition is that complexity of the software does not enter into it. Also, there is an implied assumption that all that is important about software is programming. So we might be tempted to associate this definition more with programmer productivity than with software productivity.

Even with this more restricted interpretation, we still may not have a very complete definition of programmer productivity. Our definition involves “lines of source code.” But what is a line of source code? A line of source code might represent a high-order language (HOL) line of source code. Even here, we must be concerned with whether we should include only retainable lines of code in the resulting compiled code or should also include such items as remarks statements that may be extraordinarily helpful in providing cues to the software structure. Also, we should perhaps include only those lines of source code that are delivered to the customer, and not all lines that were written and then perhaps discarded.

There are many concerns with an approach that suggests that the number of lines of code is an important and primary driver of software costs and schedule. Let us briefly examine some of these. An object line of code is a single machine language instruction either written as a machine language instruction or generated through some translation process from a different language. We immediately see a problem. Lines of code can be meaningful only at the end of a project after operational executable code is delivered and accepted by the client. Surely, delivered lines of code are not a meaningless measure of productivity. However, it is assumed that one line of code is equal to and has precisely the same value as any other line of code. Clearly, there is no reason whatsoever to accept this assertion.

Yet the number of source lines of code (SLOC) is often used as the primary driver of software cost estimation models. Often, this is modified to the term “delivered SLOC” (DSLOC) through the simple artifice of multiplying the estimate of SLOC by some fraction, which would generally depend on programmer experience and other variables. Boehm (1976a and b) delineates some of the difficulties with this metric, which are as follows:

We could continue this listing by noting that maintenance costs may well increase through the implied encouragement to produce many lines of code if these are produced without regard to maintainability. Transitioning to a new environment is especially difficult when many lines of carelessly structured code are involved.

Even if we could agree that SLOC or DSLOC is a worthwhile driver of costs, there would still remain a major problem in effectively estimating either quantity prior to the start of a software acquisition effort.

Productivity could also be defined in terms of an importance weighted sum of delivered functional units. This would allow us to consider the obvious reality that different lines of code and different software functional units will have different values. This is the basis for “function points” measurement method. It leads to an approach where software costs are measured in terms of such quantities as input transactions, outputs, inquiries, files, and output reports. These are software product quantities that can be estimated when the requirements definition document is developed. Such an approach will enable us to estimate the cost and schedule required to develop software, and perhaps even the size of the computer code that is a part of the software, in terms of:

Other taxonomies of factors for software development are possible. For example, we might identify seven cost and schedule drivers illustrated in Fig. 7.2. We could seek to develop a model for costing based on these drivers. In general, the important factors will include the people, process, environment, product, and computer system to be used. The major requirement is that the factors be significant, general, measurable, observable, and independent. This poses major difficulties, both for software costing and for software acquisition, as we shall soon see. We now turn our attention to some early models for software costing. Then we look at some more contemporary efforts that are, to some extent, based on these early efforts.

7.2 EARLY MODELS FOR ESTIMATION OF COSTS, INCLUDING EFFORT AND SCHEDULE

Ideally, a software or system cost estimation model should describe the relationships among the characteristics of the overall acquisition effort so that we can determine the effort and schedule for acquisition from these relevant characteristics. The needed characteristics would usually exist at the level of the software or system specifications. It would be ideal if the estimation could be accomplished directly from user requirements, but this is generally not possible without many additional definitional efforts and assumptions.

Use of the software-level requirements specifications would also be satisfactory. If we had an automatic way of transitioning from system-level requirements to software requirements, and perhaps were able to solve the inverse problem as well, then knowledge of one would be equivalent to that of the other.

The features useful as input to cost and schedule estimation models should include descriptions of not only the software to be developed but also the characteristics of the developing organization, such as personnel experience and capability levels, microlevel and macrolevel software development tools, and program management approaches.

This effectively assumes a deterministic software acquisition process, whereas a number of uncertain and imprecise matters are associated with any large effort to acquire a new system. A deterministic approach to a stochastic acquisition process will generally not produce correct answers. The results generally obtained will be precise; in reality, the results should be stochastic and imprecise. Hopefully, however, the deterministic estimates will provide a reasonable approximation of the realized cost.

Software cost models usually involve at least two types of estimates. These are effort and schedule. These may be expressed according to the phases of the software life cycle. Many existing software cost models estimate the number of weeks or months needed to produce a given software system. A time limit on system development is often identified by the client as one of the system-level requirements. This is generally assumed as a constraint in the transition of system-level requirements to software-level requirements and associated-effort requirements. For this reason, most cost models focus on effort requirements over a prescribed schedule horizon. As we will discuss, generally some virtually irreducible minimum time is required to obtain a new system, and increasing the acquisition costs for the system may well not be associated with a reduction in the associated acquisition schedule.

We will use effort estimates as input variables that result in the calculation of costs of development in terms of the human resources required to build the target system. Typically, this estimate is expressed in terms of the number of people needed integrated over the time to be spent, so that the measure of effort is labor-months, PMs, or person-years. The actual cost in dollars can be generated from an effort estimate by multiplying the effort by the average cost of labor per unit time. More sophisticated models may be based on the fact that the cost per person-hour of effort depends on the type of labor involved. A development effort that involves much use of systems engineering talent for technical direction and rapid prototyping would thus be costed at a higher amount than an effort that involves little of this effort and a great deal of routine coding.

Many simple software cost models describe the size of the software development program only in terms of the size of the software product. This size is often measured in terms of lines of code delivered to the customer. Complexity figures, in terms of the number of programming operators and operands, and functionality measures are used in some models as we have noted. For example, Walston and Felix (1977) allow comment lines up to half of the delivered lines of code. Boehm (1976a) does not allow the inclusion of comment lines at all. Bailey and Basili (1981) include the total number of lines of new code, including remarks, but only one-fifth of the total number of lines of reused code, including comments. There is no problem with any of these approaches, although it does make routine use of the same information on all models, as well as comparison of the various approaches, more difficult. The meaning of the expression “lines of code” must simply be interpreted for each model before using the model.

An experiential model uses expert-based wholistic judgment as the primary input to the cost estimation or forecasting process. The accuracy of the prediction is a function of the experience and perception of the estimator. In its simplest form, an estimate based on expert judgment involves one or more experts who make educated guesses about the effort required for an entire software development program, or for a project thereof. The estimate may be derived from either a top-down or a bottom-up analysis of the proposed system. Often, experts are asked to make three predictions in a simple approach to forecasting: A, optimistic; B, pessimistic; and C, most likely. If A represents the optimistic cost estimate, B the pessimistic one, and C the most likely cost estimate, then the final estimate of cost or effort is presumed to be given by

(7.1)

This is simply a weighted average in which it is assumed that the weight of the most likely estimate has a worth of four times that of either the most optimistic or most pessimistic estimate. Alternately, the resulting estimate can be said to follow what is called a beta probability distribution. The Delphi technique (Porter et al., 1991) can be used to generate A, B, and C as themselves averages of many individual responses. Of course, there is no scientific basis to support use of this cost relation. Many people would, in a judgmental situation such as this, simply provide optimistic and pessimistic estimates that are equally spaced about the most likely estimate. In this case, the most likely estimate is the average, which is half way between the optimistic and pessimistic estimates.

A number of other more formal models add more structure to the expert judgment approach. Wolverton (1974) developed a software cost matrix approach in which matrix elements represent the cost per line of code as calibrated from historical data. The multipliers in a vector product equation represent a phase–activity distribution as derived from the judgments of experts. The choice of element depends on expert judgment as to the type of software, novelty, and difficulty. This model contains as many as 8 phases and 25 activities per life-cycle phase. Use of the model is generally predicated on breaking the software acquisition effort into phases and estimating their costs individually. A software effort may represent a new or old development effort depending on the familiarity of the software developer.

Wolverton’s software cost–effort model is based on subjective estimates of software complexity. These estimates are made by considering the different types of software to be developed and the development difficulty expected. A software cost matrix can also be used as an alternate to a graphic approach. For example, we have the data shown in Table 7.1 for the six types of software considered in this early work. Clearly, the cost for developing time critical software is large at $75 per line of code. Obviously, this includes all costs, not just the direct programming costs to write a line of code.

TABLE 7.1. Wolverton Dollar Cost Estimates for Old and New Code Development

In this table, the costs are based on the type of software, represented by the row name, and the difficulty of developing such software, represented by the several columns. The difficulty is determined by two factors: whether the problem is old (O) or new (N), and whether it is easy (E), moderate (M), or hard (H). The matrix elements are the cost per line of code as calibrated from historical data. To use the matrix, the software system under development may be partitioned into modules i, where i = 1, 2, . . ., n. An estimate is made of the size Si of each module, as measured in the number of lines of developed uncommented code, for each of the n modules. This estimate does not consider software process variables, except to the extent that they influence Table 7.1. Fundamentally, the software cost estimate is a function only of product characteristics unless one is to have a different graph or table for each change in process, and management, characteristic. It is not even clear, for example, how questions of acquisition management approach, software development organization experience, availability of CASE tools, and other process-related factors affect the resulting cost estimates.

The cooperative programming model (COPMO) (Thebaut and Shen, 1984) incorporates both software size and software development staff size by expressing effort in terms of team interactions. Effort is defined as

(7.2)

where E is overall effort, E1(S) is the effort required by one or more people working independently on a size S module development who require no interaction with other modules, E2(W) is the effort required to coordinate the development process with other software development team members, and W is the average number of staff assigned to software program development.

Additional needed relations are

(7.3)

(7.4)

The parameters a, b, c, and d are determined using past development efforts. These equations are not especially different from the effort estimation equations used in the other models. The value of d can be inferred to represent the amount of coordination, management, and communication needed on a project. Similarly, c is a measure of the weakness of the communication paths among the individuals working on the software effort.

A resource estimation model has been proposed by Fox (1985), who identifies a total of 27 contributors to software development costs. The eight major contributors to the cost of software development are identified in three major categories as follows:

Each of these factors is assumed to contribute to costs per line of delivered code. The code costs can vary up to $83.00 per line, as this is the sum of the largest numbers for each of these factors. Fox encourages use of a circle-like diagram on which is sketched the relative difficulty for each factor.

The Bailey and Basili (1981) model was developed using data from a database of 18 large projects from the NASA Goddard Space Center. They make no claims that the model, based on empirical evidence from specific projects, applies to projects other than those of the same general type. Most of the software in the NASA database used by Bailey and Basili was written in Fortran, and most of the applications they considered were scientific. Thus, the database is very homogeneous. The total number of lines of code is defined to be the total amount of new code written plus 20% of the old code that is reused. Comment lines of code are included in each.

The Walston–Felix (1977) model uses a size estimate adjusted by factors determined from the subjective answers to questions about 29 relevant software acquisition topics, as follows:

These process-related factors are quite specific. In the aggregate, they represent an attempt to measure the understanding of the development environment, personnel qualifications, proposed hardware, and customer interface. Walston and Felix used these factors to supplement their basic equation with a productivity index. Using a large database of empirical values gathered from 60 projects in IBM’s Federal Systems Division, they produced the foregoing list of 29 factors that can affect software productivity. The projects were reviewed to determine parameters for the model that varied from 4000 to 467,000 lines of code, were written in 28 different high-level languages on 66 computers, and represented from 12 to 11,758 PMs of effort. For each mean productivity value xi calculated from the database, a composite productivity factor P was computed as a multi-attribute utility theory (MAUT)-type calculation.

The Walston–Felix model of software production effort that results is fundamentally based on the equation E = 5.25S0.91, where S is the number of SLOC, in thousands. A similar equation is obtained for schedule, in terms of time to project completion, as T = 2.47E0.35 = 4.1S0.35, which is also obtained from a standard least-squares curve fit approach. The effort, or cost, S is estimated in PMs of effort and the schedule in months. We note that this implicitly specifies the project workforce. Many of the models that we consider assume that there is a reasonable minimum development time and that it is very unwise to compress this. Other equations of interest in this model are L = 0.54E0.06 = 0.96S0.055 and Z = 7.8E1.11 = 49S1.01, where L and Z represent the staffing requirements in people and the pages of documentation that will be needed, respectively.

The commonalities, and the differences, among these approaches are interesting. Table 7.2 summarizes the central and basic features of many of the estimators we have discussed in this section, and provides the basic COnstructive COst MOdel (COCOMO) results that we will discuss in the next section.

TABLE 7.2. Basic Estimate Relations for Various Software Cost Models

E = 5.5 + 0.73S1.16	Bailey–Basili
E = 2.4S1.05	Boehm—basic organic COCOMO
E = 3.2S1.05	Boehm—intermediate organic COCOMO
E = 3.0S1.12	Boehm—basic semidetached COCOMO
E = 3.0S1.12	Boehm—intermediate semidetached COCOMO
E = 3.6S1.20	Boehm—basic embedded COCOMO
E = 2.8S1.20	Boehm—intermediate embedded COCOMO
E = 5.29S1.047	Doty
E = 5.25S0.91	Walston–Felix

These show different economies of scale and multipliers. This is not unexpected since the data used to provide the estimators are different. The Boehm model equations are the most sophisticated of these, by far the most used, and allow for different environment assumptions. Each of these equations is necessarily associated with errors. According to DeMarco (1981), uncontrollable factors in the development process will almost always result in 10% to 20% error in the accuracy of this type of estimate. Sadly, estimates using models of this sort are rarely this good, unless very careful attention is paid to the parameters that go into the estimates. We will now examine a number of MBC, or effort and schedule, models to illustrate the similarities and differences among them. We will begin with the COCOMO models of Boehm. After discussing this and other approaches, we will examine some concerns relative to modeling errors.

7.3 THE CONSTRUCTIVE COST MODEL

Barry Boehm has been intimately involved in almost all of software engineering, and especially involved in developing and refining his COCOMO and associated databases. His book on this subject (Boehm, 1981) is generally regarded as a software engineering classic. There have been a number of extensions to this work for software, including COCOMO II, and more recently for systems engineering, COSYSMO and COSISIMO by Boehm and his doctoral students. There are a plethora of related papers as well. The COCOMO model, or in fact “models” as there are three fundamental models involved, was derived from a database of 63 projects that were active during the period 1964 to 1979. The database included programs written in Fortran, COBOL, PL/1, Jovial, and assembly language. They range in length from 2000 to 1,000,000 of code, exclusive of comments. The COCOMO database is more heterogeneous than most that have been used for software cost projection, incorporating business, science and engineering-based, and supervisory control software. In addition to estimating effort, COCOMO includes formulae for predicting development time and schedule and a breakdown of effort by phase and activity. This is a very extensive and thoroughly investigated model to which much serious thought has been applied. It is often used in practice. Also, there have been a number of extensions to the initial COCOMO model developments, and we will discuss some of these.

There are three COCOMO levels, depending on the detail included in, or desired in, the resulting cost estimate: basic, intermediate, and detailed. Each form of the COCOMO model uses a development cost estimate of the form

(7.5)

where M(x) represents an adjustment multiplier. The units of E are PMs, or labor-months of effort. If we multiply the labor-months by the average cost of labor, we obtain the direct costs of the acquisition effort. Of interest also is the development schedule. The schedule time for development is given by

(7.6)

where E is the cost estimate just obtained. The parameters δ, ϒ, a, and b are determined by the model and development mode, as we will soon discuss. In most representations of COCOMO, the units for S are thousands of lines of delivered instructions (KDSI), the units for E are PM or labor hours (LM) of effort, and the units for T, the time for development (TDEV), are months. A PM of effort is assumed to be comprised of 19 person days of 8 hours each, or 152 person-hours. This figure allows for time off for sick leave, holidays, and vacations.

For the basic COCOMO model, the cost driver is always 1.0. For the intermediate and detailed models, it is a composite function of 15 cost drivers x1 through x15. By setting these cost drivers to 1 in the intermediate model, we obtain an adjustment multiplier M(x) = 1. Thus, the form of the intermediate COCOMO model reduces to that of the basic model when the cost drivers are at their nominal value of 1.0.

The values of a and δ are not obtained from least-squares regression, as is the case with many other cost estimation models. Boehm incorporates his own experience, the subjective opinion of other software managers, the results from use of other cost estimation models, and trial and error, to identify the structural relations that lead to the cost model as well as the parameters within the structure of this cost model. His initial parameters were fine-tuned using the TRW database. Thus, COCOMO is a composite model, basically combining experiential observations and wisdom together with statistical rule–based adjustments.

The parameters a and δ depend on what Boehm calls the development mode of a project. He labels a project as belonging to an organic, embedded, or semidetached mode, depending on the project’s independence from other systems. In the initial effort, these first three modes were described. Later effort extended this to Ada programming language development efforts (Helm, 1992). These three modes may be described as follows:

The a and δ parameters for the basic and intermediate COCOMO model are presented in Table 7.3.

TABLE 7.3. COCOMO Parameters for Basic and Intermediate Models

In the basic COCOMO model, the cost driver or adjustment multiplier M(x) = 1 for all xi. If we are considering semidetached mode development, the a and δ parameters are the same for both the intermediate and the basic COCOMO model. For organic and embedded development, the a parameters are different. This suggests that semidetached development is the nominal development mode for which the intermediate model COCOMO cost drivers are obtained. That the a parameter is different in the basic model than in the intermediate model, for other than semidetached developments, suggests that the meaning and interpretation of scaling of the cost drivers is different across these different development modes. In other words, the interpretation of the term “nominal” must be different for organic development efforts than it is for embedded development efforts. Using the basic model a parameters and the scaling indicated results in cost and effort values that are too low for organic mode development and too high for embedded mode development. This indicates that these cost-driver parameters are not independent of one another.

7.3.1 The Basic COCOMO Model

A single cost estimate, in terms of development effort, is obtained in the basic COCOMO model. This single effort measure is assumed to be distributed across the various phases of the software acquisition life cycle. In Boehm’s COCOMO effort, a four-phase life cycle consisting of conceptual design, detailed design, programming and unit test, and integration and test is assumed to be used.

For the basic COCOMO model, no effort is assumed to be allocated to such deployment efforts as maintenance. The distribution of effort does vary across phases as a function of the size of the product, in terms of the number of lines of code that are developed—although the variation is not large. Figure 7.3 illustrates how this effort, in labor-months, is distributed across these four phases, for the three different development modes. As we would expect, the simpler organic mode of development results in a greater percentage of effort devoted to actual coding of a software product, and less effort associated with the conceptual architecture and detailed design phases. Of course, the simpler products also result in less effort being needed for integration and test since there is less need for a large number of development modules. Often the percentage effort for programming and unit test may be less with embedded software than it is for organic and semidetached software. However, the actual amount of development effort will generally be greater. Even if the number of lines of code is the same, the fact that the a multiplier is larger for the embedded software will almost assure this. Figure 7.4 shows the companion results for the percentage distribution of schedule time across these four development phases. As we would expect, little effort needs to be devoted to conceptual design for organic software development. We can proceed almost directly to the detailed design and coding phases of development in this situation.

The very simple cost and schedule equations for the basic COCOMO model are

(7.7)

(7.8)

where the parameters are as specified in Table 7.3. Figures 7.5 and 7.6 illustrate the development costs and schedule for the three modes of operation. Figure 7.5 illustrates how much more labor intensive development of embedded software is, as compared with organic software. For 1000 KSDI, which represents a very large software acquisition effort indeed, it takes approximately 4.23 times as much labor (14.33 labor-months per KSDI) to produce a delivered line of code in embedded software, as contrasted with the labor (3.39 labor-months per KSDI) required for organic software. As shown in Fig. 7.3, the total development time that should be allocated to development is not appreciably different across the three development modes. For a 1000-KDSI effort, the development time is approximately 54 months, or 4.5 years, regardless of the mode of development.1 We can now use the effort and schedule distribution across phase results illustrated in Figs. 7.3 and 7.4 to determine phase-wise distributions of effort and schedule for the basic COCOMO model. This results in a relatively complete set of effort and schedule plans for software development, as based on the assumptions inherent in the basic COCOMO model.

We might wonder why there is an amount calculated for effort, in labor-months, and an amount calculated in schedule time, months, through use of the basic COCOMO model. It might seem that it should be possible to obtain virtually any schedule time desired simply through adjusting the number of people working on an effort. Sadly, this turns out to be the case only within relatively narrow limits. A very classic work illustrates this specifically for software development. We digress briefly to discuss this important result here.

Fred Brooks, in a truly seminal work (Brooks, 1975), was perhaps the first to suggest a power-law relationship between lines of code, S, and software programming effort, E. The generic relationship for this is given by

(7.9)

where the d and β parameters depend on the application being developed, programmer capability, and programming language used. In the work of Brooks, the parameter β was assumed to be 1.5, although few today would associate this large a diseconomy of scale with this relationship. This is the structure of the basic COCOMO model where β ranges from 1.05 to 1.20.

Brooks attempted to increase the power of this model by such efforts as adding terms to represent communication difficulties among the N programmers assumed to be working together on the programming project. This might result in a relation such as

(7.10)

where c is a communication cost. The division of labor is such that programmer i produces code of size Si. If each programmer has equal ability, then we have

(7.11)

Since a total of S lines of code are produced, we have

(7.12)

where the summation extends over all N programmers. It is interesting to obtain the optimum allocation of effort among the programmers. To do this we use elementary variational calculus principles. We adjoin the constraint equation of Equation 7.12 to the cost function of Equation 7.10 by means of a Lagrange multiplier. We assume that each programmer has the same cost of communications. We also assume that each programmer produces the same amount of code such that S = NSi. Then, we minimize the effort relation

(7.13)

where we have automatically satisfied the equality constraint of Equation 7.12. In this particular case, it is very interesting to sketch the relationship

(7.14)

for several values of the economy of scale factor β and the communications effort ratio E/d.

The general results for negative economies of scale look like that shown in Fig. 7.7. The very interesting observation that the amount of effort required does not continually decrease with an increase in the number of programmers involved in the effort is evident. Only for very small values of the communication effort ratio would this be the case. We would like to have the ideal result noted in the figure. But we will get this only if these are perfect communications such that E = 0. In general, there will be an optimum number of programmers for maximum code production. Increasing the number of programmers results in a decrease in the amount of code that they can produce in a given amount of time.

All of this is potentially rather distressing. Not only do we have diseconomies of scale, but also the addition of programmers to aid an already late software development project may well result in further delays! While there is nothing generic to software in this, software is perhaps more likely to suffer from diseconomies of scale than other production technology efforts.

What this sort of simple analysis suggests is that we simply must bring about automated software production technologies, that is to say, those technologies that take advantages of the macroenhancement efforts of rapid prototyping, reusable software, and automatic code generation approaches to code production if we are to cope with truly large systems efforts. It seems highly doubtful that there are many major advances left to be obtained simply through only increases in programmer productivity. Of course, this is, in no way, an argument against programmer productivity, which is very necessary—but not at all sufficient. We note that we need both a cost equation and a schedule equation to deal with these issues.

7.3.2 Intermediate COCOMO

The basic COCOMO model is very simple and assumes a large number of nominal development parameters. The intermediate COCOMO model allows for judgment relative to 15 important cost drivers. These cost drivers are sometimes called environmental factors.

For the intermediate COCOMO model, the adjustment multiplier is calculated as the product of individual cost drivers, as given by

(7.15)

The 15 cost drivers or environmental factors are grouped into 4 categories as illustrated in Fig. 7.8. Each of the 15 categories illustrated in this figure is associated with an appropriate multiplier factor m(xi). In the basic COCOMO model, each cost factor m(xi) is assumed to be equal to 1, and so each of the adjustment multipliers is equal to 1. In the intermediate and detailed COCOMO models, the cost factors m(xi) vary as a function of the quantitative values of the various cost drivers. For each category, a scale of five or six values is used to represent the possible choices for each category. For many of the cost drivers, values were obtained for the COCOMO database using the classical Delphi technique to elicit expert opinion. Reliability cost, for example, can be rated from very low to very high, in accordance with the equivalence between verbal or semantic descriptors and a corresponding numerical driver shown in Table 7.4.

TABLE 7.4. Reliability Categories and Multipliers (RELY)

Rating	Effect of lack of factor	Multiplier
Very low	Slight inconvenience if there is a problem	0.75
Low	Losses easily recovered due to problem	0.88
Nominal	Moderate difficulty in recovery	1.00
High	Major financial loss if problems develop	1.15
Very high	Risk to human life	1.40

There is, in effect, a table like Table 7.4 for each of the 15 cost drivers in the database. It is assumed that one of the specific values called for in these tables will be used in that the model is validated only for these values. This should not preclude careful use of parameter values within the given ranges.

Appropriate descriptions for each of the 15 cost drivers in the intermediate COCOMO model are as follows:

Table 7.5 illustrates the values for the intermediate COCOMO model effort multipliers as initially presented by Boehm. The overall or aggregate effects of these cost-driver multipliers may be large if they are set close to the extreme positions. If each driver is set at the lowest possible value given in this table, the product of the cost drivers is 0.0886. The largest possible product of the cost drivers is 72.379. It is, of course, unreasonable to expect this sort of range in practice. The lowest possible value would be associated with a truly excellent development team attacking an extraordinarily simple acquisition effort. The largest value would be associated with a virtually incompetent team attacking an extraordinarily difficult effort.

TABLE 7.5. Cost Drivers for COCOMO Function Multipliers

At these extremes, there would be little reason to suspect that the COCOMO model would have great validity. It is more reasonable that there would be correlation between product, project and computer complexity, and development team capability. If this is the case, then the multiplier products vary from 2.601 for a very-low-capability team working on a simple problem to 2.706 for a high-capacity team working on a most difficult acquisition effort.

Intermediate-level COCOMO uses the same approach as basic COCOMO to allocate costs, and schedules, across a four-phase development life cycle.

Let us describe the use of the intermediate COCOMO model in some detail. The steps given here can be shortened such that they are applicable to the basic COCOMO model. The intermediate COCOMO model estimates the effort and cost of a proposed software development in the following manner:

Figure 7.9 illustrates the general set of steps to be followed in obtaining a cost and schedule estimate using COCOMO. It suggests, as does this discussion, the initial use of basic COCOMO and then a switch to intermediate COCOMO to accommodate environmental factors that differ from the nominal.

The amount of variability of the 15 drivers about the nominal value of 1 varies considerably, as we see in Fig. 7.10. In this situation, the order of the drivers is arranged such that those associated with greater variability are to the right of the curve. These tend to be the drivers associated with human capabilities and problem complexity, not the ones associated with the availability of automated tools.

There are a number of possible cost drivers that are not included in the intermediate COCOMO model. Among these are such factors as requirements volatility, management quality, customer (interface) quality, amount of documentation, hardware configuration, and privacy and security facets. Boehm discusses these factors, and several others. He suggests that parsimony is among the major reasons why other factors are not included and that inclusion of many of these factors would unnecessarily complicate the analysis while not providing for greatly enhanced fidelity of cost and schedule predictions. This does not mean that these factors are necessarily neglected. They are assumed to be at some nominal values and that changes in them will not materially affect results.

It is not difficult, in principle, to incorporate other parameters. For example, we might wish to modify intermediate COCOMO to a two-level version such that it becomes possible to consider requirements volatility, or requirements creep, which is interpreted to mean the tendency for requirements to change after they have been initially identified and where software development has proceeded beyond the requirements phase.

The first intermediate COCOMO 1 model or level is restricted to the cost drivers listed in Fig. 7.8. Our new intermediate COCOMO 2 model or level would include an additional driver for requirements volatility, with values ranging from a low of 0.91 to a high of 1.62. This assumes that these multipliers are experientially determined to be correct for given amounts of “volatility.” Table 7.6 presents the effort multipliers for requirements volatility. This is then assumed to be a 16th cost driver in the COCOMO model equations obtained earlier. We can describe this 16th cost driver as follows:

An interesting question arises concerning whether or not the other drivers in the intermediate COCOMO model need to be changed to accommodate this new driver. If we can assume that the drivers are independent, the answer is no. While there is no strong reason to assume independence, we can assume near independence for small enough changes in the driver parameters.

There are a number of similar extensions of the COCOMO model. One very useful one has led to what is known as a Revised Intermediate COCOMO (REVIC) model. REVIC predicts the development of life-cycle costs for software acquisition across the three basic phases of definition, development, and deployment. It encompasses costs and schedule from requirements analysis to completion of the software acceptance testing and later maintenance. As with COCOMO, the number of DSLOC is the basic cost driver.

There are two major extensions to the development life cycle assumed in intermediate COCOMO that enables REVIC to include many aspects of definition and deployment. Six issues are of importance, which are discussed next:

There are 19 environmental factors in REVIC. Those four not included in the intermediate COCOMO model are as follows:

• RVOL—requirements volatility;
• RUSE—required reusability;
• SECU—DoD security classification; and
• RISK—risk associated with platform.

We have already described requirements volatility. Appropriate descriptors of the other three are as follows:

Thus, REVIC is essentially intermediate-level COCOMO with the addition of four additional environmental factors, an updated database, and considerable ability for “what if” type analysis, assessment, and sensitivity studies.

A standard set of two-letter descriptors associated with each of the seven cost multipliers, one more (extra–extra high) than generally described in intermediate COCOMO, is used to describe the extent to which each environmental factor differs from the nominal amount:

• VL—very low;
• LO—low;
• NM—nominal;
• HI—high;
• VH—very high;
• XH—extra high;
• XX—extra–extra high.

Some anchoring of the scoring for the factors is used to calibrate the use of the REVIC algorithms. This anchoring is particularly needed as many of the environmental factor terms need specific definition across the people responsible for providing estimates, if these estimates are to be compatible and consistent. The environmental factors and calibration presented in Table 7.7 were used in version 9 of the REVIC software. Many of the parameters are precisely the same as those used in the intermediate COCOMO model.

TABLE 7.7. REVIC Environmental Factor Calibration and Scoring

Product attributes
Software product complexity—CPLX
VL	Simple	0.70
LO	Data processing	0.85
NM	Math routines	1.00
HI	Advanced data structures	1.15
VH	Real-time and advanced math	1.30
XH	Complex scientific math	1.65
Database size—DATA
LO	DB bytes/program SLOC, D/P < 10	0.94
NM	10 ≤ D/P < 100	1.00
HI	100 ≤ D/P < 1000	1.08
VH	D/P ≥ 1000	1.16
Required software reliability—RELY
VL	Slight	0.75
LO	Easy	0.88
NM	Moderate	1.00
HI	MIL-STD/high finance	1.15
VH	Loss of life	1.40
Computer attributes
Main storage constraints—STOR
VL	None	1.00
LO	None	1.00
NM	None	1.00
HI	70% utilization	1.06
VH	85% utilization	1.21
XH	Utilization > 95%	1.56
Execution time constraints—TIME
VL	None	1.00
LO	None	1.00
NM	60% utilization	1.00
HI	70% utilization	1.11
VH	85% utilization	1.30
XH	Utilization > 95%	1.66
Computer turnaround (response) time—TURN
VL	<6 min	0.79
LO	<30 min	0.87
NM	<4 h	1.00
HI	>4 h	1.07
VH	>12 h	1.15
Virtual machine volatility—VIRT
VL	No changes	0.87
LO	1 change in 6 months	0.87
NM	1 change in 3 months	1.00
HI	1 change every month	1.15
VH	Several changes every month	1.30
XH	Constant changes	1.49
Personnel attributes
Analyst capability—ACAP
VL	15th percentile	1.46
LO	35th percentile	1.19
NM	55th percentile	1.00
HI	75th percentile	0.86
VH	90th percentile	0.71
Project application experience—AEXP
VL	Less than 4 months	1.29
LO	1 year	1.13
NM	3 years	1.00
HI	6 years	0.91
VH	Greater than 12 years	0.82
Language experience—LEXP
VL	None	1.14
LO	Less than 1 year	1.07
NM	From 1 to 2 years	1.00
HI	2 years	0.95
VH	Greater than 2 years	0.95
Programming team capability—PCAP
VL	15th percentile	1.42
LO	35th percentile	1.17
NM	55th percentile	1.00
HI	75th percentile	0.86
VH	90th percentile	0.70
Virtual machine experience—VEXP
VL	None	1.21
LO	Less than 6 months	1.10
NM	Less than 1 year	1.00
HI	From 1 to 2 years	0.90
VH	Greater than 2 years	0.90
Project attributes
Modern programming practices—MODP
VL	None	1.24
LO	Beginners	1.10
NM	Some	1.00
HI	In general use	0.91
VH	Routine	0.82
Existence of required development schedule—SCED
NM		1.00
Use of software development tools—TOOL
VL	Very few primitive	1.24
LO	Basic microcomputer tools	1.10
NM	Basic minicomputer tools	1.00
HI	Basic mainframe tools	0.91
VH	Extensive, but little tool integration	0.83
XH	Moderate, in an integrated environment	0.73
XX	Full use, in an integrated environment	0.62
Additional drivers in REVIC
Requirements volatility—RQTV
VL	None	0.91
LO	Essentially none	0.91
NM	Small, noncritical redirection	1.00
HI	Occasional, moderate redirection	1.19
VH	Frequent moderate, occasional major	1.38
XH	Frequent major redirection	1.62
Required reusability—RUSE
NM	No reuse	1.00
HI	Reuse of single mission products	1.10
VH	Reuse across single mission products	1.30
XH	Reuse for any and all applications	1.50
Classified security application—SECU
NM	Unclassified	1.00
HI	Classified (secret, or top secret)	1.10
Management reserve for risk—RISK
VL	Ground systems	1.00
LO	MIL-SPEC ground systems	1.20
NM	Unmanned airborne	1.40
HI	Manned airborne	1.60
VH	Unmanned space	1.80
XH	Manned space	2.00
XX	Manned space	2.50

What results from this is a somewhat enhanced version of intermediate COCOMO and a software package especially suited for systems management prediction of the effort and schedule needed for software development. The possibility of performing various sensitivity analysis studies for changing parameters is especially attractive. We can also reuse the model to develop the several estimates needed for incremental and evolutionary development of software.

Of course, it is easily possible to implement intermediate COCOMO on any of the currently available microcomputer spreadsheet programs. One of the objectives in any of these implementations is to predict costs and schedules for the various life-cycle phases, such as to obtain the sort of results identified in Figs. 7.11 and 7.12.

There have been a considerable number of efforts to evaluate the accuracy of COCOMO models. In one of the more detailed studies (Helm, 1992), based on a study of seven new software development efforts, it was found that COCOMO was able to predict costs and schedule for all seven projects with less than 20% relative error. To make predictions of this quality requires detailed familiarity with past organizational practices and careful calibration of the COCOMO model to reflect these characteristics.

7.3.3 The Detailed COCOMO Model

The detailed COCOMO model adds refinements not included in the basic and intermediate COCOMO models. Detailed COCOMO introduces two more components to the model. Phase-sensitive effort multipliers are included, as a set of tables, to reflect the fact that some phases of the software development life cycle are more affected than others by the cost-driver factors. This phase-wise distribution of effort augmentation is the primary change over intermediate COCOMO. It provides a basis for detailed project planning to complete the software development program across all of the life-cycle phases.

In addition, a three-level product hierarchy is employed, so that generally different cost-driver ratings are supplied for the assumed four phases of effort for 4 cost drivers that primarily affect software modules (CPLX, LEXP, PCAP, and VEXP) and for the 11 drivers that primarily affect subsystems (ACAP, AEXP, DATA, MODP, RELY, SCED, STOR, TIME, TOOL, TURN, and VIRT). These drivers are presumably used at the level at which each attribute is most susceptible to variation. As in basic and intermediate COCOMO, the value of all nominal cost drivers is unity.

Multipliers have been developed for detailed COCOMO that can be applied to the total project effort and total project schedule completion time to allocate effort and schedule components to each phase in the life cycle of a software development program. Four distinct life-cycle phases are assumed to exist. The effort and schedule for each phase are assumed to be given in terms of the overall effort and schedule by the same general equations as for intermediate COCOMO, but with the use of phase-sensitive effort multipliers. Figures 7.13 and 7.14 illustrate the way in which two of these multipliers vary across the four assumed life-cycle phases. In general, there is usually not much difference in the multipliers for the detailed design phase and those for the code and unit test phase between detailed and intermediate COCOMO.

For some of the cost drivers, such as the two illustrated in Figs. 7.13 and 7.14, there is not much difference in the conceptual design phase (denoted as the requirements and product design phase by Boehm) between intermediate and detailed COCOMO. There is a major difference, however, in the integration and test phase. We would expect this with respect to such factors as programmer capability, modern programming practice, use of software tools, and needed system reliability. For others, such as analyst team capability and project application experience, we would expect a considerable difference in the conceptual design phase as this is the phase in which these experiences are most critical. Figures 7.15 and 7.16 illustrate the variations in these multipliers over the four development phases. In each case, the multiplier associated with the intermediate COCOMO is a composite of these multipliers across the four phases shown. Much more detail concerning implementation of the detailed COCOMO model is available in Boehm (1981).

7.3.4 Other COCOMO-Based Models

The COCOMO model also has provisions for adjusting estimates of software costing when the software is developed, in part, from existing code. This involves a reuse COCOMO model. The adjustment is made at the module level, and each module is evaluated to determine how much code will be used without modification and how much will be modified.

Another variation of COCOMO is called Ada COCOMO (Boehm and Royce, 1987). This model is very similar to the original COCOMO model with only a few differences to allow for explicit incorporation of rapid prototyping, the spiral life cycle, risk management, and requirements volatility.

In general, the COCOMO model appears most appropriate for the grand design waterfall life cycle in which software is built right and completely in a single pass through the life cycle. It is easily modified to allow for other nongrand design life-cycle models and modes of development. In this approach, we calculate an effort for each of the successive, iterative, or evolutionary builds. This has been suggested by Balda and Gustafson (1990) and by a number of other researchers.

The previous model developments were macrolevel models based on the assumption of a given structure for the model and then determining parameters within this structure to match observed data. An alternate approach is possible in which we postulate a theoretically based model for resource expenditures in developing software.

It is not unreasonable to postulate that development time will influence the overall project effort and cost. Norden (1963) was perhaps the first to suggest a time sequencing of typical project effort. In investigating projects at IBM, he observed that the staffing of these research and development projects resembled a Rayleigh distribution. This sort of distribution is what might be empirically observed in practice, or obtained as a result of the sort of modeling effort that might have led to Fig. 7.11.

Putnam and Myers (1992) used the observations of Norden (1963) for project effort expenditures to develop a model of software costing. Putnam initially studied 50 projects from the Army Computer Systems Command and 150 other projects, thus developing one of the largest databases to support cost estimation. This resource allocation model, which has been developed under the name Software Lifecycle Management (SLIM), includes several effort factors: the volume of work, a difficulty gradient that measures complexity, a project technology factor that measures staff experience, and delivery time constraints. An investigation of the accuracy of SLIM (Conte et al., 1986) indicates that the model exaggerates the effect of schedule compression on development. Another potential problem evolves from rather heavy reliance on the project lines of code and development time characteristics. Kemerer (1987) has also performed an evaluation of the SLIM model, as well as COCOMO and several other software packages. Jensen (1984) has proposed a useful variation to Putnam’s SLIM model.

There are a number of alternative approaches to costing. ABC, which is not really a COCOMO variant and thus could be discussed in a new section if we presented a detailed treatment of this, is another approach to costing (Johnson and Kaplan, 1991; Kaplan, 1990; Kaplan and Cooper, 1990; Cooper and Kaplan, 1988; O’Guin, 1991). The ABC approach is based on activities required to produce a product that consume resources and that have value (or lack thereof) for customers. One of the major objectives of an ABC system is that of associating costs with the activities that lead to products. This is usually accomplished by first assigning all resources to either products or customers. The resources of an organization include direct labor, materials, supplies, depreciation, and fringe benefits. The organizational support units are separated into major functional units. Each of these functional units should have a significant cost associated with it, and each should be driven by different activities. The support unit, or department, costs are separated into a number of functional cost pools. These department costs should be fully burdened (one goal with ABC is to dramatically decrease undifferentiated burden), and all costs should be associated with the set of departments. The costs for each department are assigned to the various functional units such that each function has its own cost pool. The functional unit cost pools are next assigned, using first-stage cost drivers, to each of the activity centers. Measures of activity, such as customer orders, are very important first-stage cost drivers in that they are also measures of productivity. The organizational support costs are assigned to products or customers through these activity centers. An activity center is a functional economic grouping that contains similar or homogeneous sets of processes. The product-driven activity centers are those to which costs associated with the product lines or processes are assigned. The customer-driven activity centers are those driven by customers, such as sales and order department costs, and those that represent the costs of supporting customers and markets. The substance of an ABC system is associated with the set of second-stage cost drivers. The total cost of every activity center is separated into a number of cost-driver pools. The costs associated with each cost-driver pool are assigned to products using a second-stage cost driver. These second-stage cost drivers lead to resource consumption.

COCOMO is now available as COCOMO II and represents an extension of the initial COCOMO model described here. It is described in detail in Boehm et al. (2000). The URL http://sunset.usc.edu/csse/research/COCOMOII/cocomo_main.html is relatively extensive with much information of value, especially in terms of cost estimations efforts by Barry Boehm and/or his colleagues at the University of Southern California.

There have been several efforts to extend cost modeling COCOMO-like efforts to systems engineering. The papers by Lane and Valerdi (2007) and Lane and Boehm (2008) are noteworthy. The Handbook of Systems Engineering and Management (Sage and Rouse, 2009) describes a number of these as well, especially its Chapter 6 on cost management by Blanchard and Chapter 29 on project planning by Buys. Many contemporary systems engineering subjects that are described in detail may not formally involve costing in their initial description but must necessarily be incorporated to bring about realization of implementable systems concepts of cost analysis and assessment. One of the many such instances might involve the role of cost analysis and assessment in incorporating service-oriented architectural concepts for model-based systems engineering efforts (Andary and Sage, 2010).

7.4 SUMMARY

In this chapter, we have examined a number of issues surrounding the costing of software. While most of our efforts concerned costing, we did provide some comments concerning valuation, effectiveness, and other important issues. Modeling and estimation of software effort and schedule is a relatively new endeavor. Modeling and estimation of systems engineering, including systems engineering effort and schedule, is an even newer endeavor. Most such models are validated from databases of cost factors that reflect a particular user and developer organization characteristics. Such models are useful when developer organizations are stable and continue to produce systems, or services, similar to those developed in the past for users who have similarly stable environments and needs. However, models have little utility if the mixture of personnel experience or expertise changes or if the development organization attempts to develop a new type of system or a user organization specifies a new system type.

It would be desirable to build models that are independent of either user or developer organization, so that the models can be transported to other development teams and user situations. This is clearly foolish, however, as it assumes context-free solutions to problems, and this is very unrealistic. The major conclusion from this is that it is absolutely necessary to consider developer and user organization characteristics in the cost estimation models that we build.

A second problem is the use of cost-influencing factors and the relationship expressed among them. It is not clear that the appropriate factors have been identified; nor is it clear that they have been related correctly mathematically. Many of the cost models use adjustment multipliers in which the cost factors are summed or multiplied together.

A third issue involves changes in technology, tools, and methods. The contribution made to cost by these three aspects of project development is not as well understood as it should be for high-quality valuation purposes. Consequently, the appropriate change required in the cost equation is not known when technology, tools, and methods differ from those in place when the supporting database for a model was generated. Many of the databases used for contemporary cost and effort models use decades-old information, which is surely not reflective of current practices. Very few of the cost-estimating models are able to incorporate the potential benefits (or disbenefits) brought out by the use (misuse) of CASE tools. Nor do they consider the macroenhancement approaches of reusability, prototyping, and knowledge-based systems for development.

The development process itself presents a fourth set of needs. Some models examine the product or service rather than the software development process to obtain information about cost- and effort-influencing factors. An examination of only product- or service-related variables will generally never provide sufficient information to allow us to estimate effort and cost. More work needs to be done to determine what aspects of the development process contribute most to cost. The major improvements in productivity will come about through better development processes, and not at all through only implementing methods that ensure higher worker programmer productivity.

Most of the available methods select a single important deliverable SLOC for software, and use this variable as the core variable for estimation purposes. Various multipliers of this variable are proposed that presume to represent realities of conditions extant as contrasted with the nominal conditions under which the effort relationship to SLOC was determined. While this is not necessarily an appropriate way to go, it might be more meaningful to develop a program development plan in terms of client needs and then cost each of the phase-distributed efforts of the plan.

It is vitally important to develop cost estimation, including effort and schedule estimation, models that can be used very early in the development process. The purpose of these models is to predict development life-cycle costs and efforts as soon as possible and to provide information about the costs and effectiveness of various approaches to the acquisition process and its management. This will allow us to develop accurate information about development programs that clients might wish to fund and to allocate the appropriate resources to these developments. It will also result in feedback of important development information that will predict how changes in user requirements (software, system, or service), specifications, design, maintenance, and other factors will affect cost, effort, performance, and reliability.

A study of cost estimation (Lederer and Prasad, 1992) provides the following nine guidelines for cost estimation:

While these guidelines were established specifically for information system software, there is every reason to believe that they have more general applicability. This study identifies the use of cost estimates as selecting projects for implementation, staffing projects, controlling and monitoring project implementations, scheduling projects, auditing project progress and success, and evaluating project developers and estimators. Associated with this, we suggest, is evaluating the costs of quality, or the benefits of quality, and the costs of poor quality.

BIBLIOGRAPHY AND REFERENCES

Andary JF, Sage AP. The role of service oriented architectures in systems engineering. Inf Knowl Syst Manag 2010;9(1):47–74.

Bailey JW, Basili VR. A meta-model for software development resource expenditures. Proceedings of the Fifth International Conference on Software Engineering; 1981. p 107–116.

Balda DM, Gustafson DA. Cost estimation models for the reuse and prototype software development life-cycles. ACM SIGSOFT Software Eng Notes 1990;15(3):1–18.

Boehm BW. Improving software productivity. IEEE Comput 1976a;20(9):43–57.

Boehm BW. Software engineering. IEEE Trans Comput 1976b;25(12):1226–1241.

Boehm BW. Software engineering economics. Englewood Cliffs, NJ: Prentice-Hall; 1981.

Boehm BW, Royce W. Ada COCOMO: TRW IOC version. Proceedings of the Third COCOMO Users Group Meeting; November 1987. Carnegie Mellon University, Software Engineering Institute; Pittsburgh PA.

Boehm B, et al. Software cost estimation with COCOMO II. Englewood Cliffs, NJ: Prentice-Hall; 2000.

Brooks FP. The mythical man-month. Reading, MA: Addison-Wesley; 1975.

Conte SD, Dunsmore HE, Shen VY. Software engineering metrics and models. New York: Benjamin Cummings; 1986.

Cooper R, Kaplan RS. Measure costs right: making the right decisions. Harv Business Rev 1988;66(5):96–103.

DeMarco T. Controlling software projects. New York: Yourdon Press; 1981.

Fox JM. Software and its development. Englewood Cliffs, NJ: Prentice Hall; 1985.

Helm, JE. The viability of using COCOMO in the special application software bidding and estimating process. IEEE transactions on engineering management 1992; 39 (1):42–58.

Jensen RW. A comparison of the Jensen and COCOMO schedule and cost estimation models. Proceedings of the International Society of Parametric Analysis; 1984. p 96–106.

Johnson HT, Kaplan RS. Relevance lost: the rise and fall of management accounting. Boston, MA: Harvard Business School Press; 1991.

Kaplan RS. Measures for manufacturing excellence. Boston, MA: Harvard Business School Press; 1990.

Kaplan RS, Cooper R. Cost management systems. Englewood Cliffs, NJ: Prentice Hall; 1990.

Kemerer CF. An empirical validation of software cost estimation models. Commun ACM 1987;30(5):416–429.

Lane JA, Boehm B. System of systems lead system integrators: where do they spend their time and what makes them more or less efficient. Syst Eng 2008;11(1):81–91.

Lane JA, Valerdi R. Synthesizing SoS concepts for use in cost modeling. Syst Eng 2007;10(1):297–308.

Lederer AL, Prasad J. Nine management guidelines for better cost estimating. Commun Assoc Comput Machinery 1992;35(2):51–59.

Norden PV. Useful tools for project management. In: Operations research in research and development. New York: Wiley; 1963.

O’Guin MO. The complete guide to activity based costing. Englewood Cliffs, NJ: Prentice Hall; 1991.

Porter AL, Roper AT, Mason TW, Rossini FA, Banks J. Forecasting and management of technology. New York: Wiley; 1991.

Putnam LH, Myers W. Measures for excellence: reliable software on time, within budget. Englewood Cliffs, NJ: Pergamon Press; 1992.

Rouse WB, editor. The economics of human systems integration: valuation of investments in people’s training and education, safety and health, and work productivity. New York: Wiley; 2010.

Sage AP. Systems engineering. New York: Wiley; 1992.

Sage AP. Systems management for information technology and software engineering. New York, Wiley; 1995.

Sage AP, Palmer JD. Software systems engineering. New York: Wiley; 1990.

Sage AP, Rouse WB, editors. Handbook of systems engineering and management. 2nd ed. New York: Wiley; 2009.

Thebaut SM, Shen VY. An analytic resource model for large-scale software development. Inf Processing Manag 1984;20(1–2):293–315.

Walston C, Felix C. A method of programming measurement and estimation. IBM Syst J 1977;10(1):54–73.

Wolverton RW. The cost of developing large scale software. IEEE Trans Comput 1974;23(6):615–636.

1It should be remarked that it would be unreasonable to expect organic software development of as large a system as would be represented by 1 million lines of delivered source instructions (1000 KDSI). In a similar manner, it is probably also unrealistic that there could be an embedded software development as small as 1000 lines of delivered source instructions (1 KDSI).