Chapter 14 Metrics System Implementation

Toward Truth

The metrics journey is a movement toward truth. We implement metrics because we wish to deal with true values, to see the true picture, and to arrive at true solutions. This is the foundation on which metrics implementation seems to rest. This quest lies dormant, subdued by business pressures, and waits to be invoked. Implementation of metrics is invocation of this spirit.

No Universal Method

There is no universal technique when it comes to implementing metrics. Every organization must build its own method. Of course, there are lessons learned which are to be found in the references cited from which one can work out a system of avoiding the pitfalls. Advice such as “do not begin by measuring performance” and “start small” could be certainly useful but, by themselves, are not prescriptions for success.

One can also pick up clues from the ERP (enterprise resource planning) implementation experience of the last decade; from the well-publicized business process reengineering (BPR) problems and how some have solved them; from Six Sigma project initiatives and the eventful stories of Black Belts who changed organizations. Implementing metrics is not too different from these experiences of change management. As these stories would testify, there is no “off the shelf” solution, no ready-to-use strategy.

Roadmap?

Successful implementation of metrics could be the result of a long chain of preparatory events, from designing an appropriate metrics system to creating the right applications. The whole process begins with a desire to have a new culture that accepts transparency and statistical thinking. Then it evolves, fuelled by an emphasis on humanism, which asserts mastery of the human mind over environment. It is very difficult to trace evolutionary paths of metrics, much less to prescribe roadmaps.

The tree of metrics taxonomy can really branch into an intricate network, covering the deep fathoms of software engineering on the one hand, and penetrating process layers on the other. It could be a simple study of project management effectiveness or an elaborate research on product architecture. There is no single track to pick or beaten track to follow.

But perhaps, if one gathers the field experiences of metrics champions, one can collect some key concepts, clear principles, and some fundamentals, which help metrics implementation. We attempt to give a modest compilation of such ideas in this chapter.

Effective Use of Metrics

The central point of implementation is effective use of metrics. Implementation begins with showing results from metrics, however modest they may be. In fact, it is recommended to begin the metrics system in a low key but execute an improvement cycle in a key business area. It could be, for example, developing a simple metric called cost of poor quality and using the findings to reduce cost — a tangible benefit that everyone appreciates. In an organization such a positive and result-oriented move creates interest and orientation toward metrics implementation.

Metrics are mirrors that reflect realities. Fear of exposure inhibits the mind from accepting metrics. When the mind becomes ready to see change and reality, effective use of metrics is possible.

Effective use is possible with “bias for action,” which has the power to overcome ambiguity inherent in software metrics. Let us not use metrics to motivate people. Metrics produces results in the hands of motivated people.

Looking at Metrics Data

Goal Activation

Perhaps implementation starts when people “look” at data. Looking at data is a process that is influenced by “goals” and we see what we want to see. Also we see that part of data alone that is related to the problem at hand. Thus, a problem-solving culture and heightened sense of goals are required for recognizing data. Goals need to be refreshed periodically, otherwise they rise to their peak and fade away in the organization preview. Correspondingly, the context set by goals by metrics interpretation could also rise and fall. To keep metrics alive, therefore, the goals and metrics context must be activated periodically.

Knowledge Discovery from Data

Metrics systems contain meaning in several layers and one has to mine meaning through the knowledge discovery process. Implementation here means extracting knowledge and putting it to use. Extraction of knowledge

can be done through a simple method that makes use of the power of pictorial representation.

Data Visualization. There are numerous ways of analyzing data statistically. Any process behavior can be represented in three dimensions, namely, time, frequency, and relationship. There is no process behavioral pattern that cannot be captured in these three dimensions. Accordingly, looking at process data involves three fundamental analytical views:

1. Time domain analysis
2. Frequency domain analysis
3. Relationship domain analysis

Each analysis by itself can result in great understanding of the process. It is a diagnostic tool. Also, the analytical framework can simultaneously be a “process model” helping in forecasting.

A synthesis of these three analytical results could prove to be more useful and comprehensive than many sophisticated analysis using rigorous statistical methodologies. To implement metrics, we must keep the analysis as simple as possible.

It is a mistake to collect a lot of data but not analyze it adequately. A better ROI comes about when we collect minimum data and perform maximum analysis. Successful implementation of metrics has one cardinal principle: cut down the data collection cost.

Applying Metrics

The purpose of having metrics, interpreting them, and discovering knowledge is to apply them to the business. All applications must be well integrated with the business process flow. Other applications, even if technically feasible and very attractive, must be rejected. Applications can promote metrics better than procedures, guidelines, and instructions. Applications could be infectious; one breeds another.

Application Categories

Metrics have been put to a lot of applications, from business to science. The applications are as numerous as the management approaches that prevail. The applications are as numerous as the number of problems waiting to be solved. All these applications fall under six categories:

1. Creating estimation models: Having estimation models builds a capability to foresee problems and supports planning, a must for excellence in software engineering.
2. Creating process models: Process models are process assets, knowledge capsules that pave the way for innovation in the workplace.
3. Online use of metrics: Selecting core metrics and responding to them online makes one “vigilant” and “intelligent” regarding work progress.
4. Using metrics for managing defects: Defect is a telling manifestation of process characteristic. Managing defects is managing process.
5. Building a decision support system (DSS) from metrics: In the final analysis, metrics provide intelligence. Building a decision support using metrics will establish a modern “nerve center” for the organization.
6. Creating strategic vision: Metrics can help in seeing the intangible and in quantifying the abstract; hence, a great support for building a strategic vision.

These are possible directions of applying metrics; one need not travel in all the directions to implement metrics. Substantial benefits of metrics can be realized even in one category of applications.

For example, an organization that is not comfortable with IT innovations may skip DSS and instead choose the well-established process modeling, or one can transform the organization just from defect metrics. There have been instances when strict adherence to project discipline and online application of project metrics has led projects to outperform others. And of course, it is well known how foresight (derived by using metrics) could give competitive advantage.

Value Generation

Every successful application, whatever its direction, creates value. So waves of application in an organization will create a new value system that may compete with existing values. Those who implement metrics must be prepared for this. Application of metrics also results in creation of intellectual assets, knowledge units, and process assets. Therefore, application of metrics amounts to creation, protection, and utilization of this wealth.

Deconstruction

Metrics play as symbols, heralding a new culture. The established thought system is often destroyed, and a new system slowly comes into being. The suspension of existing symbols and assumption leaves the scenario empty.

The series of models that issue forth from metrics analysis create a strong set of higher-level symbols of a different kind. The old symbols conveyed meaning by operating as referential icons. The new symbols allow one to think and construct one’s own meaning. The former is “stock response,” the latter “intelligent creation. ”

Implementing metrics involves a transition from mechanical application of decision rules to “decision analysis” and optimal construction of meaning. This is seen as a painful transition that the organization fights involuntarily.

In one instance, trying to measure goals and track their progress changed the goals themselves. In another, measurement of size led to radically new concepts about software size. In both these examples, it turned out to be a case of self discovery and deconstruction. Prefabricated notions were examined and discarded in favor of more valid ones.

Creating Decision Centers

Metrics could eventually bring about changes in the way the organization thinks. By sharing data with people and making them see and think, we will be creating knowledge centers in the work area. By allowing them to analyze the situation and solve problems, we will be taking those knowledge centers to a higher level of organizational culture. These will evolve into decision centers that are empowered to act upon knowledge.

Creating an organization with decision centers is a postmodern trend in management. Most likely those who apply metrics will find themselves facing the emergence of this new organization. This encounter could lead to problems if the top management is not willing to create such a new organization. Unwittingly, metrics champions meet with a conflict of which they never dreamed. Therefore, it is up to the top management to prepare itself and others for the organizational changes before implementing metrics.

The full use of metrics can be realized only through decision centers. This could bring in sweeping improvements across the organization.

Equip People with Knowledge at Less Cost

Implementation of metrics involves data analysis, as well as decision making — an essentially human process that cannot be mechanized and automated completely. Human involvement and the human ability to deal with complex, real-life situations alone can lead to success with metrics.

Tools can be made use of, for a price, to support the human initiative. It is recommended with the pick-and-choose, low-cost tools that perform selected functions and install them in the decision centers. With some training on the basic statistics and on the use of tools, the decision centers are well equipped.

Attempts at using tools providing global solutions and possessing higher levels of intelligence failed in implementing metrics. These solutions are prohibitively expensive. For the high cost they have serious limitations.

Exhibit 1. Statistical functions.

AVEDEV AVERAGE AVERAGEA BETADIST BETAINV BINOMDIST CHIDIST CHIINV CHITEST CONFIDENCE CORREL COUNT COUNTA COUNTBLANK COUNTIF

COVAR CRITBINOM DEVSQ EXPONDIST FDIST FINV FISHER FISHERINV FORECAST FREQUENCY FTEST GAMMADIST GAMMALN GEOMEAN GROWTH

HARMEAN HYPGEOMDIST INTERCEPT KURT LARGE LINEST LOGEST LOGINV LOGNORMDIST MAX MAXA MEDIAN MIN MINA MODE

NEGBINOMDIST NORMDIST NORMINV NORMSDIST NORMSINV PEARSON PERCENTILE PERCENTRANK PERMUT POISSON PROB QUARTILE RANK STANDARDIZE STDEV

STDEVA STDEVP STDEVPA STEYX TDIST TINV TREND TRIMMEAN TTEST VAR VARA VARP VARPA WEIBULL ZTEST

On balance, the cost of high-level tools is several times higher than the cost of equipping people with analytical skills.

Implementation is fundamentally the human process with or without tools. Human beings cannot effectively implement prefabricated decisions. Analysis, decision making, and action are organically embedded in human systems. Implementing metrics must draw from this inherent human potential. Any other technical “alternatives” fade in comparison. The only way to implement metrics is through invoking the human initiative.

The Marvelous Spreadsheet

The spreadsheet can be used to great advantage in making decisions using data. While implementing metrics, practice on Excel’s analytical capabilities has been found supportive. There are more than 80 statistical functions available in Excel that will help in performing a wide variety of analysis (see Exhibit 1). For example, a simple function such as NORMDIST can be used to generate a Gaussian model of processes and estimate risk as a percentage.

Similarly, Excel has financial, logical, mathematical, and many other functions that will help in deeper analysis of metrics data.

Special macros are available for making complex data analysis easier. For signature analysis there is the Fourier transform macro. Similarly, powerful macros are available for ANOVA, t-Test, regression statistics, and frequency analysis. Excel allows a user to record personal macros to do repetitive tasks. Macro scripts can be edited and improved easily using Visual Basic.

Excel has very good graph-making capabilities. We can customize the graph styles to suit the business presentations. By selecting different datasets, we can generate dynamic views of graphs. This creates multiple scenarios quickly using the same data.

In addition to this, database management facilities including sort, filter, and pivot table are very useful in managing metrics data. Special tools such as Goal Seek, Scenarios, and Wizard will be very useful in decision-making applications. Excel also allows adding personal macros to the list to increase productivity of data processing and report generation. Implementation becomes easier with Excel.

Things to Remember during Implementation

• Before introducing a metrics plan, the project management systems and engineering processes must be well defined and documented. Without this foundation, metrics will end up with conflicting numbers.
• Project managers need some essential metrics: cost, schedule, resources, performance, and customer satisfaction. Add one for human assets and another for fixed assets. These seven areas are of concern. Each can have sub-areas and process areas.
• Goals first, metrics next. This rule may be applied to the sub-processes.
• Avoid information overload and “analysis paralysis. ”
• When analysis starts, strike a balance among EDA, SPC, and DOE.
• As much you wish to act upon, so much you measure.
• Human factor “perception to action.” Do not underestimate the human power for interpretation of data. The metrics system should be agile and flexible and not overly mechanical.
• Define the core metrics and allow metrics to evolve.
• Diagnose the process defects using process models.
• Extract wealth from metrics using KDD (knowledge discovery in database).
• The metrics system is the brain of the organization.
• Measure market environment at a strategic level.

Lead with Numbers

Implement metrics beginning from the leadership zone. Let the highest decision makers in business and technological issues use the first metrics, and demonstrate that they respect the data. Implementation could mean the construction of a cost model for the project, a complexity model for the product, or customer satisfaction analysis for a market segment. Let the senior-most people lead the way with numbers. The others will find it easy to follow.

Integrated Management

Metrics integrate organizational processes. Gain from this natural advantage. History was made in fixed assets management by announcing a metric called OEE (overall equipment effectiveness), which measures equipment effectiveness by considering three factors: downtime, defect rate, and productivity (hitherto the concern of three separate departments). OEE was the ace metric used in the total productive maintenance movement, which brought in spectacular results in Japan and the rest of the world.

The discussion of metrics in one forum brings all the core concerns of the organization to that forum. We get the opportunity to relate the usually separately viewed factors: productivity and quality, effort escalation and delivery slippage, employee satisfaction index, and defect density.

Implementation of a metrics culture establishes integrated project management. Quantitative methods bring in great improvements in the planning process using aids such as Monte Carlo simulation, resource balancing, capability matching, and risk modeling. In a similar vein, the estimation process also gets reinforced by use of multivariate models, mathematical templates, and statistical techniques. Metrics create a new bondage — a synergy between planning and estimation.

Metrics data analysis produces analytical views — new symbols — which are integrated in the report. Persistent integration of symbols is a forerunner of integration of subcultures.

Mirror, Microscope, and Telescope

An analogy would help in getting the complexity of implementing metrics: implementation of metrics resembles installation of mirrors, microscopes, and telescopes.

Implement metrics and you install mirrors in the organization. Mirrors can be placed in vantage points.

The investigative power of metrics unfolds in two ways. The first is the way metrics provide details on selected key process areas. Metrics put the process under a microscope and reveal hidden details not seen by cursory glances. The universes within the universe — the processes within each process — emerge and appear in the eyepiece of the metrics microscope. Without this microscope, the primary research tool, new discoveries in software engineering are impossible. The second is how metrics are used as telescopes by surveyors, the explorer, the traveler who seeks to conquer new frontiers, and astronomers who reach out to the galaxies. Aided by the metrics telescope, we can see risks and opportunities more clearly. Market research, customer requirements research, competitor analysis, threats within the process network, defect production tendencies, failure patterns within the organization, and similar scientific studies are examples of how metrics can give foresight.

One can implement metrics as microscopes and as telescopes to suit the discovery agenda one has charted out. But microscopes and telescopes are mere tools. The user makes the difference.

Unlimited Scope

In this book, we presented a few simple and economic ideas for designing, analyzing, and implementing metrics. We have been extremely conscious of the constraints in organizations to make do with existing tools and cut costs. However, growth of technology such as artificial intelligence, data mining, and Bayesian Belief Nets (BBN) in recent times holds attractive promises for better analysis of metrics. Also, new directions in metrics application are continuously evolving, and correspondingly new discoveries will be made in metrics models. We have taken a human-centric approach that envisages the birth of decision centers in an organization in order to tap the best out of metrics. There could be other approaches to suit the emerging scenarios. Metrics have great potential; there are unlimited possibilities.