Chapter 9

The Outcomes Index

WHY AN ORGANIZATION MIGHT TRACK THIS
Questions Answered
  • Do our programs actually produce valuable outcomes for customers and stakeholders?
  • Can we see a direct link between our processes and activities and outcomes?
  • Which outcomes have improved and which ones have not?
  • Have changes in our approaches or processes led to better outcomes or reduced costs?
  • Did our product or program work with other efforts to contribute to broader outcomes?
  • Are there any negative side effects caused by our product or program that could be worse than the problem being addressed?
Why Is This Information Important?
The United States has one of the highest health care costs in the world, and our quality is ranked in the fiftieth percentile worldwide. We continue to dump more money into education but many high schools still have a 25 percent dropout rate, and graduating students can’t read or do math at a ninth-grade level. The government spends millions on a program like “Just Say No to Drugs” that has failed to decrease drug use. In today’s world of restricted funding for government programs, each activity, initiative, and program is being questioned as to its value to society and taxpayers. Since everyone now has to have health insurance and pay something for the services they receive, government, organizations like Medicare, and insurance companies are looking to avoid paying for brand-name drugs and expensive surgeries, making it harder for pharmaceutical companies and health care providers to make millions or billions as they used to.
Any organization that has a product or service that is supposed to produce an outcome is expected to demonstrate results. I worked with Long Beach Unified School District (the second largest in California), and they had big grants from both the Dell Foundation and the Broad Foundation. Both charities are very generous in their support of education, but the grants had to result in improved education outcomes or the money would stop flowing. When the teachers’ union refused to accept the idea that there would be positive and negative consequences for teachers based on the performance of their students, the grants were not renewed. The foundations understood that for performance to improve you need goals, measures, feedback, and consequences.
The Community Development Department in the City of Los Angeles is a shining star among city departments. Through the use of measurement, feedback, and accountability, the Community Development Department was able to dramatically reduce its costs while increasing the number of disadvantaged people they placed into jobs. Its performance was so impressive that the Community Development Department and many of its contracted WorkSource Centers won the prestigious California Award for Performance Excellence.
In health care there is a set of 75 metrics used to evaluate organizational performance, called Healthcare Effectiveness Data and Information Set, or HEDIS. But while many of these metrics are good process measures, very few look at health outcomes. The same is true of many education evaluations that focus more on classroom behavior of teachers (process) and curricula (inputs) than on outcomes like student success. Long Beach Unified School District does not measure how many graduates enroll in college, get a job, or go to jail. In fact, the school district doesn’t measure any outcomes other than the percentage who graduate, and even that is a gamed statistic by counting the percentage of 12th graders who graduate versus the percentage of 10th graders who graduate high school. Outcome metrics in health care, education, and many government programs are a joke.
The other problem I see with outcome metrics is a lack of shared accountability. The government likes to hold one agency or department accountable for one outcome so blame or credit is easy to assign. So who do you blame for the high unemployment of the last few years? Clearly jobs and full employment are important outcomes for a healthy economy, but who is held accountable for that metric? The Commerce Department, Treasury, the Federal Reserve, the SEC, Congress, the Senate? When I worked with the Federal Highway Administration, one of the key outcome metrics it was held accountable for was highway fatalities. Even though it doesn’t set or enforce laws, the administration got the blame if fatalities increased. The major causes of traffic fatalities are speeding and driving under the influence. Texting or talking on the phone is probably right up there these days as well. The point is that there are multiple federal, state, city, and county organizations that might have some accountability for traffic fatalities. Certainly local police departments, as well as state police, would have some accountability. What I have not seen is a bunch of government departments getting together to collectively decide what outcomes they should be accountable for and who owns what percentage of the accountability. Instead what happens is that if performance is good or improves, they all stand up and take credit, and if it is bad or declining, they blame others.

TYPES OF ORGANIZATIONS WHERE THIS METRIC IS APPROPRIATE

This metric is appropriate for any city, state, county, or federal government organization that is tasked with providing some sort of outcome for taxpayers or other stakeholders. Some government organizations are easier to measure than others. The IRS has outcomes that are measured in dollars every month. An agency like the Department of Energy may have a tougher challenge coming up with outcomes, but this department was originally formed to eliminate the United States’ dependence on external sources of energy. I don’t think that outcome has been achieved and I think the trend is going in the wrong direction. The Energy Department’s current mission statement is much vaguer and harder to tell whether it has been achieved:

The mission of the Energy Department is to ensure America’s security and prosperity by addressing its energy, environmental and nuclear challenges through transformative science and technology solutions.

It would be tough to come up with some good outcome metrics for prosperity and security, but one of them might be how much of income the average American spends on gasoline, electricity, and natural gas.

Not only do government organizations need to measure outcomes, but so do organizations that sell to government, like defense companies, or are paid by the government, such as health care providers, as well as schools and government programs. Even profit-making manufacturers and service companies might measure outcomes if the product is designed to save people time or money, or somehow improve the quality of their lives.

HOW DOES THIS IMPACT PERFORMANCE?

What is scary is that a lack of demonstrable outcomes sometimes does not impact performance in government organizations. If an organization fails to achieve its mission, it continues to exist, get funded, and maybe even get more money. Luckily, there are some rewards for organizations that do achieve positive outcomes. U.S. News and World Report and other organizations rank hospitals based on key outcome metrics like mortality, and this seems to be having an impact. More people check into the good hospitals, so their financial health improves, they can hire better staff and buy better equipment, and health care improves even more. Two recent examples are Henry Ford Hospital in Michigan, which recently won the Baldrige Award, and Southcentral Foundation, a community health care provider in Anchorage, Alaska, that also won a Baldrige Award. Both organizations not only demonstrated improved health outcomes, but also lower costs. Demonstrating improved outcomes might also help ensure that you get your share of the budget when things get tight and make you much less of a target when leaders are looking for programs and places to cut. Most leaders are loath to cut programs that are clearly producing measurable outcomes, particularly those that are important to politicians and taxpayers. So one of the reasons to have a good outcome index is that you demonstrate a genuine desire to improve your performance. A secondary reason may be side benefits like recognition and rankings, awards, ability to hire better staff, avoidance of budget cuts, and promotions for managers and others.

Another important way this affects performance is that the data allows you to detect processes, programs, and initiatives that do not produce outcomes or cost more than the value they produce. Outcomes always need to be compared with costs. Some countries have much better services for their citizens than we have in the United States. People also pay much more in taxes. If you think it is bad here, go to Holland or some other European countries like Denmark. Being able to demonstrate improved outcomes at lower costs is really what gets attention and recognition.

COST AND EFFORT TO MEASURE

The cost to develop and implement a system for tracking outcomes ranges from low to high, depending on the type of organization you’re in. If you are a health care provider, I am sure you already track key outcomes like successful surgeries, mortality, patients discharged, infections, and other factors. If you are in education, you have a lot of work ahead of you. School systems track test scores of academic achievement, but this is not really an outcome metric. An outcome metric is something like the percentage of graduates who get accepted to college and what colleges they get accepted to, or the percentage of graduates who are able to find a decent job. Our local Manhattan Beach school system is one of the best in the state and tracks many of these key outcome metrics. Harvard Business School tracks what percentage of its MBAs are running big corporations or holding high-level government jobs.

Where the cost will come is in collecting the data rather than designing the metrics. Getting access to data that may be in other databases is sometimes a challenge. If the Just Say No to Drugs program wanted to really track their effectiveness they could keep track of how many people are arrested for possession of drugs, how many are in rehab, how many are in jail, how many dealers are arrested, and how much pot is being dispensed in states like Colorado and Oregon and California. In other words, getting data on drug use would require accessing many different databases, which would take lots of effort and probably cost a bit of money.

HOW DO I MEASURE IT?

Step 1 in developing an outcome index is to decide on the broad categories of outcomes that will be measured. A good reference point is to go back to your mission statement. If Disney’s mission is to make people happy, then happiness should be the key output the company measures, and it certainly does measure it and does a great job of managing it. The Community Development Department I work with in the City of Los Angeles strives to provide its clients with independence and the improved self-respect that come with having a decent job. Therefore key outcomes might look at independence, self-respect, and employment. These might be the broad categories of outcomes that the City of Los Angeles measures. For some organizations that have a very broad mission like the World Bank or the Inter-American Development Bank, broad categories of outcomes might include factors such as:
  • Economic outcomes
  • Health outcomes
  • Education outcomes
  • Transportation and access outcomes
Step 2 is to develop four to six outcome metrics within each of the categories. Unlike most of the HEDIS metrics for health care, outcome measures are just that. Screening people for diabetes is not an outcome metric. Reducing the incidence is an outcome metric. Graduating students is an outcome, but so is preparing them for work or higher education, so a measure for high school would be the percentage of students who are employed or enrolled in college. When you are selecting outcome metrics, don’t ask whether you can control the outcomes, but ask whether your organization can influence them. A school district might not be able to control the number of kids who are in gangs, but it can certainly influence it. The Long Beach schools did not want to measure the health of the students even though it feeds them sometimes two meals a day and provides them with exercise. Student health is clearly a metric that a school can influence and could be considered an outcome.
Other than the degree to which you can influence the outcome, you need to think about the integrity of the data. If it is an objective measure tracked using reliable methods by an unbiased source, then the metric should probably make your short list. It is often helpful to create a list of eight to 10 outputs for each area and then narrow them down to the two to four best ones. Where this gets complicated is an organization like a hospital that has lots of health outcomes. The most basic outcome is that people recover from whatever is wrong with them and don’t develop additional problems (like infections) from being in the hospital. A hospital administrator or a CEO should be able to look at one overall health outcomes index and get an assessment of how the organization is performing at its healing mission. Yes, there will be lots of submetrics that break down the data into various categories by process, illness, or procedure performed. For example, delivering healthy babies is one of the more positive outcomes for a hospital. Hospitals also have to deal with cancer, death, accidents, and not so pleasant outcomes as well, so the idea behind this metric might be to sort the outcomes into a positive and a negative category.
Step 3 is to assign weight to each of the outcomes. Metrics assigned a high percentage should be those with the strongest link to the organization’s mission, the greatest integrity, and the greatest degrees of influence and control. For example, for a university, a measure of graduates finding jobs in their field of study would be a really important outcome metric, as would getting accepted into graduate school. Less important outcomes might be student and parent satisfaction with their education.

FORMULA AND FREQUENCY

An example of an outcome index for a community development department in county or city government might be:

Jobs 40%
Number of new jobs
 10%
Number of people employed
 20%
Promotions
 5%
Turnover
 5%
Economic development 40%
Number of new businesses started
 15%
Slums and blight eliminated
 15% (square footage)
Turnover of new businesses
 10%
Youth 20%
Percent graduating high school
 10%
Percent completing vocational training
 10%

The exact formula that you use will be up to you, and there is no right answer. However, what you don’t want to do is take 50 to 100 metrics and roll them all into an index. Data should be stacked in a pyramid with the outcome index being the peak of the pyramid that breaks down into three to six dimensions, which break down into four to six metrics, which might break down into four to six submetrics.

VARIATIONS

One variation is just to have a couple of outputs and not create an index or analytic metric comprising lots of different measures. This might work if you are a simple organization with a single product or service. An airline, for example, is a pretty simple business, and it might just measure the percentage of passengers who are transported on time to their desired destination. One outcome metric certainly makes it simple. However, an airline has other outputs like bags and cargo delivered on schedule. Cargo and bags undamaged is another important outcome. Most organizations are sufficiently complex that they have multiple outcomes for multiple services and products, and each needs to be measured and combined so that management does not have to review 50 charts to see how the organization is performing on its core mission.

Another variation is to combine process, output, and outcome metrics into a single index. For example, a hospital or medical clinic might take all of the 75 or so HEDIS metrics and combine them into a single index. Most of the HEDIS metrics are input or process metrics. I think this is indeed possible, but it is not recommended. It is better to keep process and outcome metrics separate, even though there should be a link. Washing hands after each patient is probably a good process measure for preventing infections, but infections are an outcome that should be tracked separately. It is possible to be diligent about hand washing and still spread infections by other means, like those ties doctors still wear.

TARGETS AND BENCHMARKS

Outcome metrics are probably the easiest to get averages, benchmarks, and all sorts of comparative data. Regulators often track performance of various organizations, as do professional associations and trade groups. Each individual outcome metric needs to have a unique target set based upon customer and stakeholder requirements, industry averages, your own capacity and history, resource constraints, and benchmarks. As with most composite indices, “green” is usually defined as 80–100 percent compliant or greater. This may vary greatly depending on the industry and metric. A rating of 80 percent safe landings or 80 percent successful surgeries would probably not be good standards. With many outcome metrics we are looking for Six Sigma (less than three defects per million), or very close to perfection. On the other hand, a school system that had 80 percent of its graduates go on to college would probably be doing a great job.

BENEFITS OF DATA

There are huge benefits to having one aggregate index that can be viewed every day that tells leaders how the organization is performing on its key outcomes besides the financial ones. Having one overall metric also minimizes the chances of leaders micromanaging or obsessing about minor outcomes rather than focusing on the big ones. A CEO I worked with obsessed about getting ranked in the bottom 25 percent of a survey of customers, when those customers only represented about 10 percent of his business. Staking the outcome data in layers by importance and dimension allows leaders to drill down as necessary into the detail when there are problems but most of the time stay focused on the 30,000-foot view of the organization. Having good outcome data like this can also be helpful when it comes time to present to your board, donors, finders, the community, and other key stakeholders. Being able to demonstrate the value you provide to society, the taxpayer, or whoever your customers and stakeholders are is a huge advantage that many of your peers may not have.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset