Chapter 3
Designing a Metrics System

Metrics System

Several rules have been prescribed to plan metrics in an organization. Methodologies have been developed to make effective use of metrics. In the past ten years there has been substantial progress in this direction. We applied these methodologies to several situations and worked with practitioners who have attempted implementation of their favored methodologies. Each time we tried an application, we realized its strong points and also encountered some difficulties.

We have studied metrics from a practical angle and identified factors that control the success rate of metrics implementation. What emerged from our findings is a new approach, which is presented here.

Metrics are best viewed as a system. We cannot design metrics in isolation from the environment. Metrics are connected to measurements by mapping rules. Metrics are connected to goals through decision rules. The architecture of metrics system is built around the information highway of organizations, which feeds decision centers. The objective of such a metrics system is to provide model-based decision support.

Designing metrics system architecture is the first step in metrics planning. The second step is to implement the system by working out a set of phased operational plans. Managing these two steps — design and implementation — is what metrics application is all about.

Information-Based Metrics Architecture

We propose a modern architecture in which metrics fulfills information needs in all the decision centers in an organization. The elements in metrics system architecture are

• Goal system
• Decision centers
• Models: knowledge capsules
• Metrics: indicators–signals
• Measurement: sensors

Exhibit 1. Metrics system architecture.

The architecture treats the organization as a network of processes delivering results to the customer. The concept of a value chain is used to identify processes for measurements. Processes that do not add value are not considered.

The metrics system architecture is illustrated in Exhibit 1. It may be noted that the system architecture is in agreement with the knowledge creation sequence discussed in Chapter 1, and moves data upwards. The architecture lives in an information environment, and can elegantly and naturally fit into MIS or decision support system (DSS) networks that exist in the organization.

Goals: The Drivers

It is well established that goals drive organizations. When we go into the details, we find that goals are translated and distributed across all organizational systems, decision centers, and problem-solving initiatives. Information systems need a “pull” from goals, human systems require goals for motivation and guidance, management systems use goals to define objectives, and goals are known to influence the very structure of all systems under their guidance. Decision making, at any level, depends on goals even more. Then, for problem-solving cycles, goals are the starting point and the very recognition of problems is achieved by comparing actual results against goals.

Exhibit 2. Goal tree.

Now, systems have sub-systems and, correspondingly, goals have sub-goals. Decisions are made in organizational layers and, correspondingly, goals disperse into goal layers. Problem-solving cycles (as in Kaizen) are distributed across the organization, and we think of a suitable goal distribution. Placed together, all these constitute a complex goal system that can be modeled as a goal tree, shown in Exhibit 2. The metrics system architecture provides for an elaborate goal structure, as indicated in Exhibit 2. The design begins with defining goals. In reality, goal definition is a lengthy process; it happens in waves, each widening its reach.

Because the role of metrics is to support systems, decisions, and problem solving, when goals are not defined, metrics are futile because there is nothing to support.

When an organization launches its metrics program, it starts discovering its goals and perceives the goals with clarity. When the metrics culture sets in, goals are defined quantitatively.

The goals–metrics interaction is very dynamic and creative; each shapes the other. All known metrics initiatives, such as Basili’s GQM paradigm and Park et al.’s adaptation called GQ (I) M, are goal-centered frameworks.

Exhibit 3. The new organization.

The information flow shown in Exhibit 1 is directed toward goals. In organizations goals exist in multiple layers and are translated for “deployment.” We recognize and strive for congruence among organization goals, project goals, team goals, and individual goals. Hence the complexity of a goal system can be seen.

Decision Centers: The New Organization

An organization is a network of processes. A few of the process centers also perform like decision centers, as shown in Exhibit 3. We look to the decision centers in an organization for metrics application. In this Information Age and era of the knowledge worker, the process-centric model of organizations, actively promoted by Total Quality Management (TQM) and business process reengineering (BPR) enthusiasts, is now superceded by the decision-centric model. The shift from process centers to decision centers occurs when the process is run by a dedicated “process owner,” replacing mechanical and ritualistic “operators.” A process center becomes a decision center when empowered with knowledge and decision-making freedom. There could be several process centers in an organization, but the number of decision centers could be dismally low. Decision centers shift constantly in unstable environments.

While the ritualistic process centers reject metrics, decision centers have a natural appetite for metrics. The rejection will then wrongly be ascribed to “failure of metrics. ”

We have to map the decision centers in an organization and trace them while developing a metrics plan. It is quite likely that the decision centers can be found in a hierarchical order and the metrics plan will inherit the hierarchy of decision centers, as shown in Exhibit 4.

Exhibit 4. Hierarchy in decision making.

Models: Knowledge Capsules

Models are abstractions of realities, which allow us to learn by inquiry as an economic alternative to trials. Models help in visualizing the process behavior. Models can be in the form of mathematical equations, graphs, and matrices that would represent real-world entities. They also help in forecasting, prediction, and what-if analysis.

Models are created using metrics, perhaps in knowledge centers. These models are consumed by all centers in the organization.

In decision centers, models meet with a special challenge, which arises from a very basic feature of decision making: search for alternatives. Decision analysis benefits from flexible, intelligent, and even interactive data presentations instead of frozen statistical predictions. Decision-making practices draw heavily from “management games” and probability assessment. Dynamic models can support decision making better than static models.

Based on the measurement scale employed, models can be in several forms: cognitive, iconic, semantic, visual patterns, quantitative structures, and, with the help of computers, artificial intelligence (AI). They can be built to address all process areas, as illustrated in Exhibit 5, mapping model building potentials to process areas. We can conceive a library of models, each an intellectual asset of immense value. The growth of software engineering can be traced to the discovery of new models for software quality, reliability, and estimation. From fixed assets management to knowledge management, all major disciplines have constructed and published models.

Published models need to be calibrated before use in an organization. Perhaps one has to choose between calibrating a ready-made model and constructing a new one. Calibration takes minimum effort, while construction is a project on its own merit. Both depend on metrics.

Exhibit 5. Model building potential.

Metrics: Indicators–Signals

The measurements–metrics–models presented in Chapter 1 become an integral part of the metrics architecture. To begin with, metrics are indicators, built from observations. Metrics are compared with goals, and the comparison provides the first level report. At the next level, models can be created.

That metrics support the subsequent information processes in the architecture is best seen from a communications system analogy. Well-designed metrics can play the role of corporate “signal generators,” feeding information networks in the organization. The signal must emerge above noise and communicate messages clearly.

An example would help in appreciating metrics as origins of objective communications. Use of metrics in managing the training process is rather well known. Here is a common list:

• Rating of trainer
• Relevance of course
• Application potential of ideas presented
• Rating of training material
• Rating of environment (hall, light, video, audio)
• Rating of amenities provided (lunch, tea, water)
• Knowledge score before training
• Knowledge score after training
• Number of absentees
• Cost of training

In the example cited here, despite good monitoring of all the attributes of training process and presenting the reports quantitatively to the concerned managers and decision makers, one problem persisted. There were always absentees. All the nominated persons did not participate in the actual training. The training manager devised a new metric that better communicated the problem. He presented a new metric to the organization: cost of training per person per program. This metric may also be thought of as “dollar productivity” of the training process. Expressed in monetary units, the metrics caught the attention of senior managers, and an analysis revealed that the cost of training per person per program was almost double the budgeted cost. Had all the nominated people participated in the training, the budgeted dollar productivity would have been achieved.

The senior managers reacted to the messages provided by this metric, and the result was straightforward: it was communicated to all employees that if a nominated person did not attend training sessions, he would have to explain the reason to the management council in person. Attendance improved, as did dollar productivity of training.

While the earlier metrics set failed to represent and communicate the problem, the new metric effectively communicated the problem and resulted in organizational transformation.

Measurement: Sensor System

To measure is to observe or to sense processes, and hence measurement elements constitute a sensor system. The sensors planted at various stages of the process chain help in deriving the data. Locations of these sensors, the data collection points, are chosen to meet the information needs expressed in the form of desired metric and models.

The measurement system is distributed across the entire organization in a multitude of forms, ranging from automated tools to human observation. The characteristics of sensors vary correspondingly in consistency and bias.

The metrics system architecture integrates all measurements from a systems standpoint.

Data Collection

Collecting metrics data is perhaps the hardest part. Many a metrics program has failed on account of difficulties encountered in data collection. Hence, data collection systems must be carefully designed, avoiding the known pitfalls, and adopting lessons learned from metrics installations.

One sure thing is that we must design a metrics database. Different database technologies are available, including intelligent systems. The problem, however, is in the design of interfaces between the database and process centers.

In a common scenario, process owners tend to keep their personal databases. Project managers keep their data on a planning tool. Centralized metrics tools collect quality-related data but project data and some process data are not available. An integrated database seems to be an ideal beyond common reach. We may have to accept the reality of a heterogeneous and distributed database, as illustrated in Exhibit 6.

Implementing the Metrics System Architecture

Preparation

After designing the metrics system architecture, the task is to implement it. A good design is easy to implement, a poor design may demand a trial-and-error approach. The steps involved in the implementation address realizations of the elements of the architecture:

Exhibit 6. A heterogeneous and distributed metrics database.

• Recognize goals and define them.
• Recognize decision centers and list information needs.
• Build a suitable library of models.
• Define an economic metrics set.
• Define measurements.

Start with Small Scope

It is better to start a metrics plan with a small scope. One can limit the number of projects or take portions of the software life cycle. The most successful metrics system is evolutionary in style. The goals, decisions, models, and metrics are continually evaluated, and the metrics system changes to match the needs of the organization. Metrics and approaches are pilot-tested and discarded if inappropriate.

Begin with Lower Number of Metrics

Experience from successful metrics programs suggests that a minimal set of measures and metrics is usually adequate for beginning a program and sufficient to fulfill priority goals. If a single measure is sufficient to address the organization’s goal, keep just one. It is not the number of metrics but what we do with them that is going to make a mark.

Phased Expansion

Many organizations begin measuring in the small, by applying measurements to a hot spot or difficulty, or by measuring to quantify the improvement in a process, product, or resource. Eventually, as staff becomes more comfortable with measurement, the small pockets of measurement spread to encompass most or all of an organization’s software activities. This bottom-up approach to a measurement program is both popular and effective, as staff buys into the value of measurement, one project or problem at a time.

Exhibit 7. Goal–metrics correlation matrix.

Metrics
Goals	M1	M2	M3	M4	M5	M6	M7	M8	M9
G1	r11	r12	r13	r14	r15	r16	r17	r18	r19
G2	r21	r22	r23	r24	r25	r26	r27	r28	r29
G3	r31	r32	r33	r34	r35	r36	r37	r38	r39
G4	r41	r42	r43	r44	r45	r46	r47	r48	r49
G5	r51	r52	r53	r54	r55	r56	r57	r58	r59
G6	r61	r62	r63	r64	r65	r66	r67	r68	r69
G7	r71	r72	r73	r74	r75	r76	r77	r78	r79
G8	r81	r82	r83	r84	r85	r86	r87	r88	r89
G9	r91	r92	r93	r94	r95	r96	r97	r98	r99

In the horizontal expansion, the metrics program is implemented in more than one component of the software process or activities. In the vertical expansion, this metric is implemented in two or more software projects.

Goal–Metrics Correlation (GMC): Metrics Choice-Checking Tool

One can use a matrix structure to cross-check and analyze how the established goals and metrics go together, using the goal–metrics correlation structure presented in Exhibit 7. GMC seeks a correlation between goals and metrics but does not seek to trace out the reasons for choice. If GMC analysis discovers goals that are not correlated to metrics, these are orphan goals, left out of the renaissance in the organization.

If GMC shows metrics without correlation to goals, these metrics are lone rangers, drifting around listlessly. These can be dropped, of course. But our experience shows that the team, which introduced the particular metric in the game plan, was indeed addressing an unarticulated and hidden goal. Hence, the odd metric should be examined for its roots before any decision is made about its withdrawal from the plan.

Metrics Planning Approaches

Long-Term Plan: Core Metrics

Metrics have life cycles. They can become obsolete after the goals are attained. At least one can say that metrics acquire the life cycle of goals themselves, apart from acquiring other characteristics of goals such as hierarchy.

It feels good to operate on firmer ground and identify the core metrics that will be pertinent for a longer period. Core metrics have another requirement to fulfill. They must be relevant to all types of projects being executed in the organization, a constraint tougher than longevity. The most popular core metrics are schedule variance, effort variance, defect density, and productivity.

Short-Term Plans

By supporting the permanent core metrics we can identify short-term plans that can be rolled out across projects, capturing the specific business and process goals of the individual project. Technology-driven considerations would introduce metrics that are meaningful to categories of projects coming under technology umbrellas. Sometimes we use special metrics for special studies, such as in Six Sigma initiatives or Kaizen where the chosen problems may require metrics, which become useless after the study is over or the problem is solved. Separating the short-term from the long-term metrics has avowed benefits.

Metrics Planning Document Checklist

At the center of a metrics plan in an organization is the formal construction of metrics, which addresses the following details:

• Name
• Purpose
• Definition
• Example
• Data source
• Expected range of values
• Periodicity of data collection
• Format for data collection
• Metrics database structure
• Resources identified
  • Collection
  • Analysis
  • Reporting
• Model analysis

As a vehicle that carries organization data, the metrics plan is designed to pick out process intelligence with reliability and support the organization in decision making and process improvement. Metrics can yield bene-fits to the organization, as much as the plan will allow or provide for.