Chapter 13 Metrics for Strategic Vision

Beyond the Obvious

Making use of metrics to look beyond the obvious is quite important. Metrics have been seen as a tool to look at the details, a costly research microscope. Metrics can be equally powerful in looking into the future, viewing beyond the immediate neighborhood. This kind of application of metrics is in tune with proactive management initiatives. When the management process changes its approach from merely responding to a situation to working with vision, from myopic schemes to strategic vision, the predictive abilities of metrics will be in vogue.

Model-Based Approach

A step in the direction of building strategic vision from metrics is to build models, as many of them as possible, and use these models in management thinking. The agenda for models is prediction, forecasting, and estimation. We have seen in the previous chapters how we can build empirical models from metrics and apply them for decision analysis. In almost each of such applications, the objective has been to predict future from the several perspectives provided by those models.

Seeing the future through metrics-based models is like making two-dimensional drawings of a three-dimensional object. Several views such as the front view, the top view, the side views, isometric views, and cross-sectional views are required to understand the true object. We have to apply our imagination while looking at drawings. Likewise, we have to apply innovation while running models. We build vision by running these models iter-atively, scanning frame after frame of the scenario.

The Vision Called Integration

The metrics system integrates an organization in a manner not very different from how information technology integrates enterprises. The very establishment of metrics, from definition to deployment, is based on an integrated point of view of processes. The empirical formulas that use multiple variables also integrate, conceptually, the corresponding process variables, allowing us to see process as a system. Metrics are champions of integration, constantly bringing together those ideas, parameters, and process indicators.

Integration of process indicators, from data fusion to process modeling, presents a new outlook and an enduring vision, which becomes the cornerstone of postmodern management.

In this chapter, we will see a few applications of metrics in vision building. These are complementary to similar applications that lie scattered in the earlier chapters.

Metrics in Project Management

Applying metrics to the project amounts to scaling down the organization’s metrics plan to the project requirements. Such a tailoring is based on a focus on the project goals, which are relatively short term. Also metrics can play a crucial role in capturing customer requirements quantitatively. A much-acclaimed use of metrics is in risk analysis and forecasting. Goals, customers, and risk represent three critical factors that a project must reckon with.

Tailoring Metrics for the Project

From the larger list of metrics proposed in the metrics system, the project manager selects those that correlate with the project goals. Hence, the selection rule has to resolve the problem of clearly defining and articulating the project goals, and also positioning them with the relevant organization goals. In matured projects even process goals can be added to this list.

Then the goals must be quantified and prioritized. In framing the goals it is possible to set directly measurable goals such as timely delivery. Sometimes the goals do not have a direct measure. But they can be related to a metric with affinity. For example, optimum resource utilization can be a goal; many find it difficult to measure it directly, whereas a simple metric such as effort variance points to resource utilization. Schedule variance along with effort variance can provide information about resource utilization.

For each goal a weighting factor or rank can be ascribed and the goal list can now be rearranged according to its significance. Setting measurable goals in a structured way such as this brings vision and clarity to the project. And this remains among the finest contributions of metrics.

Setting Quantitative Goals: Goal–Metrics Correlation (GMC)

Vision is expressed through goals. Vision building and goal setting are interdependent. Goals, for better understanding and application, must be expressed quantitatively. The metrics system in the organization, initially an offspring of goals, eventually supports in giving a quantitative expression to goals.

In principle, any goal can be measured directly. Some goals are easily measurable; some are abstract and can be measured through survey forms and rather elaborate procedures. Economy in quantifying goals is achieved by the GMC method.

Exhibit 1. Goal/metrics correlation table.

			Metrics
	Goals	Rank	SV	EV	DD	RE	P	REN	A	DT
Corporate Goals Project Goals Process Goals	cs PFT MS TD REL CMM REV	1 1 1 1 2 3 2	1 0 0.5 1 0 0 0 2.5	0.3 0.7 0 0 0 0 0 1	0.7 0.8 0.5 0.3 1 0 0 3.3	0.2 0.5 0 0.3 0.7 0 1 2.7	0.2 1 0 0.3 0 0 0 1.5	0 0.6 0 0.5 0 0 0 1.1	0 0.7 0 0.8 0 1 0.2 2.7	0 0.7 0 0.2 0 0 0 0.9	2.4 5 1 3.4 1.7 1 1.2

As the name suggests, goal–metrics correlation is a matrix structure, as shown in Exhibit 1. The GMC structure has goals in the columns and metrics in the rows. The weightage factor is also indicated along with the goals in a separate column.

Metrics
SV = Schedule Variance%
EV = Effort Variance%
DD = Defect Density (Defect/KLOC)
RE = Review Effectiveness%
P = Productivity LOC/Month
REN = Risk Exposure Number
A = Attrition
DT = Downtime of Assets, Hours/Month

Goals
CS = Customer Satisfaction
PFT = Profit
MS = Market Share
TD = Deliver in 3 Months
REL = 97% Reliability
CMM = SEI CMM Level 5
REV = Better Reviews

The goal list is carefully prepared. The organizational goals, the project level goals, and whatever goals the teams are required to follow, need to be

brought in the list. Conflict between goals, if any, is resolved. The project manager, who is going to use the GMC as a planning tool, will ensure that accepted goals appear in the table.

Every cell in the matrix bears a correlation coefficient, which measures the strength of the relationship between the associated goal and metric. Instead of directly quantifying a goal, we are going to be satisfied if they correlate well with a metric. We are transferring numeric quality from metrics to goals by association.

By examining the GMC matrix we can easily find and pick those metrics that fit into the goal system. They are the ones that post correlation figures. We know that only such metrics will survive in the goal-dominated project environment. Hence, after the GMC study the fittest metrics are selected for the project and a practical metrics system is deigned (other metrics are spotted and isolated for study). This way, the metrics plan gets tailored to the project plan and the commonly felt problem of the gap between planned metrics and actually used metrics is preempted.

Following is a list of the overall benefits of GMC:

• Setting measurable goals
• Tailoring metrics plan
• Firm foundation for project plan
• Good communication tool for goals and metrics to project team
• Overview of entire goals and metrics system at a glance

GMC Analysis

Metrics Effectiveness

In the GMC matrix, we first add the columns and with the resultant values we can plot a bar graph with the metric name in the x-axis and the sum in the y-axis. This is known as the effectiveness profile of metrics. This pro-file is shown in Exhibit 2.

Effectiveness of metrics can be defined, in predictive style, as the association it has with goals. A metric that has scored high has a high association with the defined goal. Lower scores indicate poor association.

Metrics that do not connect with any known goals are “lone rangers” and must be taken away from the active list.

The metric effectiveness profile also provides strategic information about the most effective metrics in the project that need to be supported at any cost.

Exhibit 2. Metrics effectiveness map.

Goal Deployment

The row totals indicate a measure of metric support to the goals. The profile of goals scores is shown in Exhibit 3. Larger scores indicate that the goals stand a good chance of being interpreted in measurable terms. The larger the score, the larger the degree of effective goal deployment in the project.

If this profile has large imbalances, we normally revisit the choice of goals and metrics. Maybe we will have to improve the metric system (if the profile scores match the ranking ascribed to goals, then the metrics system already reflects goal preferences).

Exhibit 3. Goal deployment profile.

Iterative Process

GMC matrix analysis is iterative. The first analysis may show incongruities that will have to be ironed out and refined before the second run. For example, as is a common experience GMC may reveal the absence of a goal but still have metric that addresses the goal. Conceptually, people put metrics in place first, and then define goals.

After a couple of iterations one can expect a refinement in a metrics system for the project.

Exhibit 4. Customer requirements topology.

What
Customer Requirements
Category	Requirements	Rank
A
B
C

Quality Function Deployment (QFD)

Metrics can be applied to capture the customer’s voice using the well-known quality function deployment (QFD) structure. The complete QFD known as the house of quality can be built in four stages.

Stage 1: Defining Customer Requirements

Customer requirements are defined and prioritized in this stage. These definitions must use a minimum number of words, almost encoding the quintessence of customer requirements. The precision involved in this exercise makes it the measurement of requirements in linguistic scale, as shown in Exhibit 4.

Stage 2: What–How Analysis

Next, the process capabilities and facilities available in the project are identified and listed in a row. A correlation mapping is done between the requirements and capabilities using a WHAT-HOW matrix, as illustrated in Exhibit 5. The WHAT column represents the voice of the customer and HOW column represents the organization’s ability to respond to the voice.

This mapping will expose the weaknesses in the organization’s ability to meet the customer needs. Sometimes we also come across the capabilities and facilities that do not relate to customer requirements. This also acts as a resource planning tool. A mismatch between resources and requirements forewarns process risk.

Exhibit 5. What/how matrix.

What			How (Features, Capabilities)
Customer Requirements			Group 1		Group 2		Group 3
Category	Requirements	Rank	F1	F2	F3	F4	F5	F6	F7	F8
A
B
C

Exhibit 6. Correlation matrix.

F8								1
F7							1
F6						1
F5					1
F4				1
F3			1
F2		1
F1	1
	F1	F2	F3	F4	F5	F6	F7	F8

Stage 3: Process Analysis

In Stage 3, a relationship study is made between the process capabilities, and the target values for each capability are expressed quantitatively, as illustrated in Exhibit 6.

This stage involves a scientific study of correlation between processes and meticulous determination of capability baselines and process goals. Such a study will bring an in-depth understanding of the interline between process elements and help to detect and solve some hidden problems in the process.

Exhibit 7. House of quality.

			Correlation Matrix
		F8								1
		F7							1
		F6						1
		F5					1
		F4				1
		F3			1
		F2		1
		F1	1
			F1	F2	F3	F4	F5	F6	F7	F8
What			How (Features, Capabilities)
Customer Requirements			Group 1		Group 2		Group 3
Category	Req.	Rank	Fl	F2	F3	F4	F5	F6	F7	F8	Benchmark I
A
B
C
	Benchmark II

Stage 4: Benchmarking

In this stage, two benchmarking studies are conducted. The first is to compare how our best three competitors fail in meeting customer requirements. The second is to compare our process capabilities with the best three competitors.

Comparison with the market condition gives major input to strategic planning. It is quite likely that different organizations may employ different measurement scales to measure the process parameter. Care must be taken to arrive at a common scale. See Exhibit 7 for illustration. By benchmarking we measure the market scenario to which the customer is going to be exposed. Today’s competition is tomorrow’s customer requirement. Hence, we measure indirectly the future requirements of the customer.

Thus, QFD is a business survival tool. It measures the most important parts of business: customer and market. QFD maps all processes to the market and exposes mismatches. QFD can be applied to all processes right from marketing to maintenance. This series of applications will translate the customer’s voice to the process.

Risk Estimation

Most software metrics activities are carried out for the purpose of risk analysis of some form or another. Forecasting risk by the traditional analysis of metrics has been found to be insufficient. Special techniques are required. We are presenting two simple methods for measuring and forecasting risk. One is simulating schedule risk by a computerized planning tool and the other is mapping risk using a risk exposure number (REN) matrix.

Simulating Schedule Risk

Risk arises from unexpected variations in deliveries. There is a finite probability that the schedule may vary, which is customarily computed from probabilistic models derived from data. In the absence of historical data construction of models from data is not possible. Instead we simulate the project scenarios on a computerized planning tool and trigger variations by altering the project elements.

Variations in the project duration can be traced to three project elements:

1. The work breakdown structure (WBS)
2. The schedule estimations of individual tasks
3. Network architecture (sequential, concurrent, or a combination of both)

The WBS can have different tasks lists based on the planner’s experience and approach. Schedule estimation for each task may vary in similar fashion from person to person. For a given task list and set of estimates one can think of a variety of task networks with a combination of sequential and concurrent arrangements. All three elements could vary simultaneously and give rise to a large number of project scenarios, each having its own duration. Creation of scenario is running a simulation run in the software. It can be a series of simulations and capture all possible schedules. In Exhibit 8 and Exhibit 9, two scenarios are presented for illustration.

The results of all simulation runs can be summarized in the format given in Exhibit 10.

From this table one can arrive at the first-order estimate of risk by computing descriptive statistics as minimum duration, maximum duration, mean, median, mode, and standard deviation. If the number of simulation runs is sufficiently large, from this data we can create probability distribution of schedule, as illustrated in Exhibit 11. Perceiving risk from the probability distribution is discussed in Chapter 5.

Exhibit 8. Scenario 1.

W1

W2

W3

W4

W5

W6

W7

W8

W9

W10

W11

W12

W13

W14

W15

W16

W17

W18

Start

Fix scope

Meeting with customer

Req. analysis

Prototype

Prepare SRS

End

Exhibit 9. Scenario 2.

Wl

W2

W3

W4

W5

W6

W7

W8

W9

W10

Wll

W12

W13

W14

WIS

WIG

W17

WI8

Start

Interview customer

Document

Internal review

Document

Risk analysis

Document

GUI prototype

Meet customer

Final draft SRS

End

Exhibit 10. Format: summary of simulation runs.

Scenario Number	Duration Weeks

Mapping Risk Using Risk Exposure Number

Mapping risk in a project involves first recognizing risk elements. To a large extent, recognition depends on past experience of the analyst. Each recognition must be defined without ambiguity and expressed in a concise form. Some even give IDs for each risk so that they are traceable.

For each risk element we estimate the likelihood of occurrence and the magnitude of damage that would be caused if the risks were to be attacked. Both the likelihood and damage are expressed in convenient quantitative scales. One possibility is to express likelihood on a probability scale of 0 to 10 and the damage on a scale of 0 to 10. Risk exposure is now computed by multiplying these two, as illustrated in Exhibit 12.

Analysis of REN

The first-cut analysis is to sort out the table according to risk exposure number. This will give a focus on critical risks.

One can also generate the risk Pareto chart, as shown in Exhibit 13. Apply the 80/20 principle and identify the 20 percent of risk elements that account for 80 percent of the damage.

Exhibit 11. Frequency distribution of simulation output.

Exhibit 12. Risk exposure number.

Risk Exposure Matrix LEVEL: SM Risk	Probability	Loss	REN	CREN	CREN%
Price cut Order cancel Review failure Wrong requirements Attrition Defect leakage Delivery slippage Technology change	9 2 4 2 1 3 1 0.5	6 10 4 5 9 3 5 3	54 20 16 10 9 9 5 1.5	54.0 74.0 90.0 100.0 109.0 118.0 123.0 124.5	600.0 822.2 1000.0 1111.1 1211.1 1311.1 1366.7 1383.3

Having short-listed risk elements, we can arrive at a mitigation plan to cut down probability and a contingency plan to minimize damage.

The sum total of REN count is taken as overall measure of risk, which can be tracked from time to time in the project.

Exhibit 13. Risk Pareto chart.

Six Sigma Renaissance

Six Sigma Vision

At the core of the Six Sigma movement is a new vision that combines quantitative methods with leadership. This powerful combination has achieved breakthrough improvements. It has also brought all the tools for improvement developed in the past five decades to focused use.

The Six Sigma cycle uses metrics to advantage in almost all the phases in recognizing, defining, measuring, analyzing, improving, and controlling; the Define, Measure, Analyze, Improve, Control (DMAIC) model revolves around measurement and analysis.

Metrics in the Boardroom

The magic of Six Sigma has its origin in the boardroom and happens when top management believes in data and is willing to train people to look at and analyze data and apply the knowledge gained to improve the situation. Top management has taken this training a bit seriously and enlisted support from all. All Six Sigma case studies are case studies of leadership faith in numbers. The Six Sigma breakthroughs start with achieving quantitative understanding of processes, as Bill Smith, the originator of the Six Sigma concept, did in Motorola.

When the metrics system is taken seriously by the board, the organization takes a new shape, perhaps in the name of Six Sigma.

Money, the Greatest Metric

Perhaps Six Sigma succeeds where other initiatives have failed; it is because Six Sigma uses financial benefits as the criteria. Profits and bottom lines are the watchwords.

Money seems to be the greatest metric ever. Crosby knew it when he insisted that cost of failures must be measured in monetary terms. Now sigma improvement is judged by cost savings. On the surface it looks like a short-sighted business drive. But there is a lot of wisdom behind this choice. Modeling process behavior in terms of cost functions is an established scientific approach (used widely in optimization algorithms). Defin-ing a cost function or a profit function enables one to see performance clearly and relate the result to influences now called cost drivers. Converting process variables into cost variables helps to combine several processes for modeling.

And, in the organizational context, money represents great value, health of the projects, and a very communicative indicator.

In money, Six Sigma has achieved the great convergence.

Metrics Black Belts

Six Sigma Black Belts (application experts, as they were called in yesteryear) take professional training on measurement and analysis of metrics data and changing the organization through a series of improvement projects. The body of knowledge (as in the ASQ Black Belt certification curriculum) that the Black Belts are expected to master includes several data analysis methods:

• Measurement Scales
• Metrology
• Types of Data
• Methods of Data Collection
• Descriptive Statistics
• Inferential Statistics
• Probability
• Graphical Methods
• Frequency Distributions
• Process Capability
• Exploratory Data Analysis
• Simple Linear Regression
• Multiple Linear Regressions
• Design of Experiments

Metrics data analysis is a basic Black Belt skill. Metrics application for improvement is the very purpose of Black Belt learning. For Black Belts, application is the key.

Measurement Capability

As defects are reduced to part-per-million levels, process measurement capability must improve to match. A normal measurement system has the ability to detect one tenth the variation it tries to measure. The ability to detect even the smallest process drift or deviation is not easily achieved in software projects. The organization “sees” what it measures. Where measurement capability is less, many process problems are buried beneath the carpet. Where measurement practice is absent, even larger problems are not seen; they do not exist for all practical purposes, and hence there is no perceived need for process improvement!

Six Sigma programs realize this early in their project phases. Attempts to build models from metrics data will reveal such inadequacies related to precision of measurements.

Consummate Vision

We have presented in this chapter four components of vision. The first is related to goals, seeing them, defining them, and deploying them. Goals are expressions of vision. Then comes a vision that covers market forces and the customer; QFD is good way of consolidating this vision. Risk perception is next, seeing well ahead what could fail later is what this is about. Finally, the combination of data analysis and leadership offers a unique vision with great power; it could transform organizations.

The purpose of metrics, ultimately, is to build a consummate vision and give the user strategic benefits.