Planning the Process for Outcome Assessment

For outcome assessment to have maximum value, the process for that assessment must be well planned and executed. The first step in that regard is for the organization's leaders to be committed to the effort. Ideally, an initiative of this kind will begin with the chief executive of the nonprofit organization but, in any event, the chief executive and the organization's board should understand and support the initiative. Support includes recognizing and accepting that outcome assessment could uncover unwelcome truths about program performance. There is no point in taking the time to develop and obtain performance data if those in charge are not committed to using the data.

Assuming this support is assured, outcome assessment should be undertaken on a program-by-program basis. For a nonprofit organization with multiple programs, that guideline means that each program will require separate outcome assessment planning. A specific individual should be assigned primary responsibility for that planning for each program, preferably as part of a small team. (The discussion that follows will use the term decision makers to encompass both possibilities.)

Regardless of whether a team is used, the planning process should entail extensive involvement of staff and perhaps even clients who are involved with the program. That involvement serves at least two functions. First, it assists in information gathering. Since those who are involved with a program know it from the inside, they can provide valuable intelligence on the program's desired outcomes and possible means for measuring their achievement. Second, involvement can build ownership in the outcome assessment process. If program staff and clients have the opportunity to speak to how the program will be assessed, they are more likely to buy into the eventual results of the assessment.

Finally, the process should also be linked to the organization's information technology. Improved information technology, by facilitating the recording and analysis of performance data, is a major factor underlying the recent push for better performance assessment in both the nonprofit and public sectors. Building a strong performance assessment system requires that the system be planned in conjunction with the organization's technology.

Defining Program Goals

Outcome assessment is a goals-based process; programs are assessed relative to the goals they are designed to achieve. Defining those goals can prove to be a difficult task since the project leader or team must define and differentiate several types of goals while navigating the often-difficult politics of goal definition. This section first explains several goal types and then discusses how to define them in a political context.

Measuring Goals

Once the goals have been defined, attention must turn to how to measure them. Before thinking about specific measures, decision makers should become familiar with some basic measurement concepts and with the various types of measures available.

Data Collection, Analysis, and Reporting

Once the necessary measures have been defined, decision makers should plan for data collection, analysis, and reporting. They must ensure first that procedures are established for recording observations on the measures. They must also determine how the new data collection process will be integrated with the agency's information technology, including evaluating whether new software will be needed for the effort.

Before putting the full outcome assessment in place, the agency should pilot test the measures and the data collection procedures to see how well they work. Measures sometimes prove not to produce the anticipated information. Convenience surveys, for example, sometimes elicit only partial responses from program clients, which could require either improving or abandoning that instrument. Problems can also arise in the recording of data, perhaps necessitating rethinking the recording procedures.

Decision makers, certainly including the agency's chief executive, should also establish a schedule for regular reporting and review of the data. Depending on agency preferences and perceived needs, reviews might be planned as frequently as weekly or as infrequently as annually. Or, less intensive reviews might be planned more often with more intensive reviews scheduled only occasionally (e.g., on a quarterly or annual basis).

The details of the schedule are probably less important than that a schedule is established and implemented. Judging from the findings of one study (Morley, Hatry, and Cowan, 2002, p. 36), many nonprofit agencies that collect outcome information do not systematically tabulate or review the data, instead “leaving it to supervisors and caseworkers to mentally ‘process’ the data to identify patterns and trends.” Agencies unnecessarily hamstring themselves when they make such choices. If systematic outcome data are available, agencies should ensure that the data are tabulated and reviewed.

Growing numbers of governments and some nonprofit agencies (see Carman, 2007, p. 65) are taking the additional step of putting program effectiveness and efficiency data into comprehensive performance management systems and then reporting the data through summary “scorecards” or “dashboards.” Popularized by Kaplan and Norton (1996), the so-called “balanced scorecards” are designed to present a 360-degree picture of an organization's performance at any given time. With their growing popularity among governments (see, for example, Edwards and Thomas, 2005), these scorecards seem likely to become increasingly common among nonprofit agencies in the coming years.

Actual review of the data can go in a number of directions depending on what the data look like and what questions the agency has about the programs. At the outset, initial data on any new measures can be at once the most interesting, yet the most difficult to interpret. Novelty accounts for the likely high interest: agency executives and staff may be looking at outcome readings they have only been able to guess at before. However, with initially only one data point to analyze, those readings may seem uninterpretable. Interpretation becomes easier as readings accumulate over time, permitting comparisons of current performance to past performance.

To increase interest and potential utilization of the data, agencies should consider asking key staff to predict results in advance. Poister and Thomas (2007) have documented that asking for such predictions (albeit on a limited number of measures) increased interest among state administrators in the results of stakeholder surveys. Prediction questions can be asked relatively simply, too (e.g., “what proportion of program clients will say they are satisfied with the program?”).

The focus of the interpretation depends on a variety of factors. If the data show an unexpected trend or pattern—such as an unanticipated decline on an outcome measure from one quarter to the next—attention may focus on explaining that pattern. More generally, though, the analysis of the data should be driven by the questions and concerns of the agency. Is there a concern about whether a program is working at all? Or, might the concern instead be whether a new program component is achieving desired improvements?

At the same time, care should be taken not to over-interpret outcome data. In particular, outcome data should not by themselves be read as implying causality—that is, to conclude that any observed changes resulted from a specific program or programs. Such changes could have resulted from other factors (a change in the economy, for example) that are wholly independent of the program. Outcome assessment data by themselves speak to important questions of whether progress is being made on key agency objectives, but cannot explain the part the agency has played in inducing those changes.

When questions about program performance turn toward these issues of causality, agency executives must move a step beyond outcome assessment to conduct a full program evaluation. Program evaluations, in essence, start from a foundation of strong outcome assessment and add the techniques of comparison and control necessary to speak more definitively to the role of specific programs in producing desired outcomes.

Two Approaches to Program Evaluation

Program evaluation can seem a frightening prospect, raising the specter of outside experts “invading” the organization, seeking information in a mysterious and furtive manner, and ultimately producing a report that may contain unexpected criticisms. Such fears are not ungrounded. The traditional approach to program evaluation, sometimes termed the objective scientist approach, often proceeds along those lines.

Borrowed from the natural sciences, the objective scientist approach entails several elements. To begin with, objectivity is valued above all else. To achieve objectivity, the evaluator seeks to maintain critical distance from the program being evaluated in order to minimize possible influence by program staff, who may be biased in the program's favor. The objective scientist also strongly prefers quantitative data, recognizing qualitative data to be subjective by nature—the antithesis of objectivity. Finally, the usual purpose of an evaluation for the objective scientist is to determine whether or to what extent the program has achieved its goals. Is the program sufficiently effective to be continued, or should it be terminated? The objective scientist takes little interest in how a program's internal mechanics are functioning.

Two decades of experience have revealed shortcomings to this approach. Evaluators who insist on keeping their distance miss the unique insights staff often have about their programs. Disdaining qualitative data further limits the ability to assess a program because the goals of most public and nonprofit programs are too subjective to be measured only by quantitative techniques. Finally, the insistence on critical distance combined with an exclusive focus on program outcomes can result in evaluations that fail to answer the questions decision makers have.

Recognition of these problems led to the development of an alternative, the approach that Michael Quinn Patton has termed utilization-focused evaluation. As Patton (2008, pp. 451–452) has explained, this approach begins with the goal of balance rather than objectivity. Where objectivity implies taking an unbiased view of a program by observing from a distance, balance recommends viewing program operation from up close as well as from afar, thus to discern important details as well as broad patterns. Achieving balance also requires qualitative as well as quantitative data because the latter are unlikely to capture all that is important about programs whose goals are subjective. A balanced assessment necessitates multiple perspectives.

The balanced approach also rejects outcome assessment—“did the program work?”—as the only purpose of an evaluation. A utilization-focused evaluation seeks information for use in modifying and improving programs, too. Getting close to the program helps by putting the evaluator in contact with the program administrators who have questions about how programs should be modified as well as the authority to implement those modifications.

The balanced approach is not appropriate for every program, every evaluator, or every nonprofit executive. In getting close to a program, an evaluator can risk being “captured” by the program and, at the extreme, becoming only a “mouthpiece” for those who are vested in the program. For that reason, if there are serious questions about the quality of a program or about the competence of its staff, the nonprofit executive may prefer an evaluation performed from the critical distance of the objective scientist.

For the most part, though, nonprofit executives will find that the utilization-focused evaluation approach promises both a more balanced assessment and information more likely to be useful in program development. As a consequence, the discussion that follows assumes a utilization-focused approach to evaluation.

Who Does the Evaluation?

A first question, when planning a program evaluation, is who should conduct the evaluation. Here the principal options are (a) an internal evaluation performed by the organization's staff, (b) an external evaluation performed by outside consultants, and (c) an externally directed evaluation with extensive internal staff assistance.

An internal evaluation is possible only if the organization has one or more staff members with extensive training and experience in program evaluation. Unlike outcome assessment, full-scale program evaluation is too technical a task to attempt without that expertise. An internal evaluation also requires that the nonprofit executive give essentially a free rein to the evaluation staff. Since inside evaluators may face strong pressures to conform their findings to the predispositions of program staff, standing up to those pressures is possible only if the nonprofit executive has made an unequivocal commitment to an unbiased evaluation.

As a practical matter, although many nonprofit organizations prefer to conduct evaluations internally (e.g., Carman, 2007, p. 70), most appear to lack sufficient in-house expertise to produce high-quality evaluations. They are probably better advised to seek outside assistance from United Way or private-sector consulting firms, management assistance agencies for the nonprofit sector, or university faculty (often found in public administration, education, or psychology departments).

Hiring an outside consultant carries its own risks. Perhaps the greatest risk is that the external evaluators, perhaps trained in the objective scientist tradition, may resist getting close to the program and consequently conduct the evaluation with insufficient concern for the organization's needs. A preference for critical distance may blind them to the questions and insights the agency has about the program.

To minimize this risk, the nonprofit executive should discuss at length with any prospective evaluators how the evaluation should be conducted, including whether they are capable of taking a utilization-focused approach. It is also wise to negotiate a contract that specifies in detail how the nonprofit organization will be involved in the evaluation.

Perhaps the best means for conducting an evaluation is through a combination of outside consultants and internal staff. In this mode, outside consultants provide technical expertise plus some independence from internal organizational pressures while internal staff perform much of the legwork and collaborate with the consultants in developing the research design, collecting data, and interpreting findings. The idea, as documented in one study of successful evaluations (Minich et al., 2006, p. 186), is to “not expect program staff to be researchers” and, in that spirit, to “shift measurement tasks to full-time evaluators.”

There are several advantages to this approach. First, it provides the necessary technical expertise without sacrificing closeness to the program. Second, greater staff involvement should produce greater staff commitment to the findings, increasing the likelihood that findings will be utilized. Third, the evaluation can be used to train staff to serve a greater role in future evaluations. Finally, having staff implement much of the legwork could reduce the out-of-pocket costs for the outside consultants. This reduction is possible, however, only if care is taken that working with the staff does not require too much of the consultants' time.

That time commitment can be limited by creating a small advisory team to oversee the evaluation. This team should include the outside evaluators, the nonprofit executive (or the executive's representative), and at least one to three other staff members in the nonprofit organization. The team should serve as the central entity to which the evaluator reports, reducing the time necessary for working with program staff. Keeping its size small (in the range of three to five members) facilitates the team ability to provide clear and prompt feedback to the evaluation process. A team of this kind is probably desirable for wholly internal or external evaluations, too.

The only way to assure that the chief executive's concerns about the program are addressed is for that executive to be personally involved in the evaluation, optimally as a member of the evaluation advisory team. In addition, as the literature on organizational change attests (see, for example, Fernandez and Rainey, 2006), programmatic change is unlikely to occur through an evaluation unless the chief executive is involved and committed to the process.

The goal of this involvement should not be to obtain the “right” answers—answers that conform to the executive's predispositions—but to ensure that the right questions (the questions crucial to the program's future) are asked. The chief executive should emphasize this distinction to the evaluator(s) up front, and then monitor to be sure the distinction is observed as the evaluation proceeds.

Determining the Purpose of the Evaluation

The first task of an evaluation is to define its purpose. That is, what sort of information is desired and why? How will the information be used? Answers to these questions will be crucial in determining the other elements of the evaluation.

Discussion of evaluation purposes typically begins with a dichotomy between summative and formative purposes (see Rossi, Lipsey, and Freeman, 2004, pp. 34–36). A summative purpose implies a principal interest in program outcomes, in “summing up” a program's overall achievements. A formative purpose, by contrast, means that the principal interest is in forming or “re-forming” the program by focusing the evaluation on how well the program's internal operations function. In reality, though, the purposes of evaluations are much more complex than a dichotomy can convey. Saying an evaluation has a formative purpose, for example, does not indicate which of the program's internal mechanisms are of interest.

An evaluation's purpose should reflect the concerns key stakeholders have about the program. The process of defining this purpose thus should begin with the nonprofit organization's executive: What questions does he or she have about how the program is working? What kinds of information might speak to anticipated decisions about the program? Opinions of other stakeholders, including funders, may also be solicited.

In the end, a number of purposes is possible, depending on the perceptions of stakeholders and the specific program. An evaluation performed primarily for funders, who may be most interested in whether the program is having the desired impact, is likely to have a summative purpose. By contrast, a program that has only recently been implemented may be a good candidate for an implementation assessment—an evaluation of how well a program has been put into operation—but a poor candidate for a summative evaluation because the program has not been operating in the field long enough to expect an observable impact. Evaluations designed mainly for program staff are likely to have principally formative purposes to help staff modify and strengthen the program.

Because this purpose will guide decisions at all subsequent steps in the evaluation, a mistake at this stage can hamper the entire effort. The nonprofit executive should consequently review this purpose and make certain it reflects his or her concerns as well as the concerns of other key stakeholders. It is also true, though, that an evaluation's purpose may become clearer as the evaluation progresses. Stakeholders may be able to articulate their questions about programs only as they consider program goals and measures. Evaluators should be open to this possibility.

Evaluators and nonprofit executives must also be alert to the possibility of so-called covert purposes, unvoiced hidden purposes for an evaluation (Weiss, 1972, pp. 11–12). Program managers, for example, sometimes have an unspoken goal of “whitewashing” a program by producing a favorable evaluation. The responsible chief executive will reject such an evaluation as unethical as well as incapable of producing useful information.

It is at this stage that the evaluator and the organization's chief executive should also consider whether the evaluation is worth doing. Revelation of a dominant covert purpose would provide one reason to bow out. Or, it may be impossible to complete an evaluation in time to inform an approaching decision about the program. The resources necessary for a program evaluation are difficult to justify unless the results can be meaningful and useful.

Experiments

Experimental designs offer the strongest internal validity. The classic experimental design takes this form:

refers to randomization, meaning that subjects are assigned by chance—for example, by lot or by drawing numbers from a hat—to the experimental or control group in advance of the experiment.

Randomization is a crucial defining element of experimental designs. With the inter-group and across-time components of this design testing for covariation and time order, randomization establishes the final condition of causality, nonspuriousness, by making the experimental and control groups essentially equivalent. As a consequence of that equivalence, the control group provides a test of “change across time”—the changes due to maturation, regression, history, and so forth, which could affect program outcomes. Comparing the experimental and control groups can thus separate program effects from other changes across time, as this simple subtraction illustrates:

Unfortunately, many practical problems work against the use of experimental outcome designs in evaluations. In particular, randomization poses a number of difficulties. First, it must be done prior to the beginning of an intervention; participants must be randomly assigned before they receive treatment. Second, ethical objections may be raised to depriving some subjects of a treatment that other subjects receive, or political objections may be raised to providing treatment on anything other than a “first come, first served” basis. Experiments can also be costly, given the need to establish, maintain, and monitor distinct experimental and control groups. Since many programs are still changing as they begin operation, it sometimes proves impossible to (as an experiment requires) maintain the same program structure throughout the length of the experiment.

But the possibility of conducting an experiment should not be dismissed too quickly. The need for prior planning can sometimes be surmounted by running an experiment not on the first cohort group of subjects, but on a second or later cohort group, such as a second treatment group of substance abusers. Ethical and political objections often can be overcome by giving the control group a traditional treatment rather than no treatment. That choice may make more sense for the purpose of the evaluation anyway, since the ultimate choice is likely to be between the new treatment and the old, not between the new treatment and no treatment.

Quasi-Experiments

If an experimental design cannot be used, the evaluator should consider one of the so-called quasi-experimental designs. These designs are so named because they attempt, through a variety of means, to approximate the controls that experiments achieve through randomization. The strongest of these designs come close to achieving the rigor of an experiment.

A first quasi-experimental design is the nonequivalent control group:

Here, in lieu of randomization, a comparison group is matched to the experimental group in the hope that the pre-post comparison of the two groups will furnish an indication of program impact.

This design is as strong—or weak—as the quality of the match. The goal of matching is to create a comparison group that is as similar as possible to the experimental group, except that it does not participate in the program. A good match can be difficult to achieve because the available comparison groups often differ in crucial respects from the experimental group.

Consider a hypothetical job-training program for the unemployed that takes participants on a first come, first served basis. The obvious candidates for a comparison group are would-be participants who volunteer after the program has filled all of the available slots. The evaluator might select from those late volunteers a group similar to the experimental group in terms of race, sex, education, previous employment history, and the like—similar, in other words, on the extraneous variables that could affect the desired outcome of employment success.

The difficulty arises in trying to match on all of the key variables at once. Creating a comparison group similar to the experimental on two of those variables—say, race and gender—may be possible, but the two groups are unlikely then also to have equivalent education levels, employment histories, and other characteristics. In addition, the two groups may differ on some unrecorded or intangible variable. Perhaps the early volunteers were more motivated than late volunteers, accounting for why they volunteered sooner. If that difference were not measured and incorporated in the analysis, the program could erroneously be credited for employment gains that actually stemmed from the differences in motivation. In cases such as this, no match is preferable to a bad match.

A second kind of quasi-experimental design is the interrupted time series design, diagramed as follows:

The defining elements of this design are three or more observations recorded both before and after the program intervention. Multiple observations are important because they provide a reading on trends, thereby controlling for most changes over time (maturation, regression, and so on), which experimental designs achieve through randomization. Those controls give this design relatively good internal validity.

History is the principal weakness of this design, with respect to internal validity. There is no control for any event that, by virtue of occurring at the same time as the program, could affect program impact. A program to improve the situation of the homeless could be affected, for example, by an economic upturn (or downturn) that began at about the same time as the program.

Obtaining the necessary multiple observations can also prove difficult. On the front end, preprogram observations may be unavailable if measurement of key outcome indicators began only when the program itself began. On the back end, stakeholders may demand evidence of program impact before several post-program observations can be obtained.

One of the strongest of the quasi-experimental designs is the multiple interrupted time series:

The strength of this design results from combining the key features of the interrupted time series and the nonequivalent group design. The time series dimension controls for most changes across time; the nonequivalent control group dimension controls for the threat of history.

The problems with this design derive from the possible weakness of its component parts. A bad match can provide a misleading comparison; the lack of longitudinal data can rule out use of this design at all.

Other Designs and Controls

The realities of many programs preclude the use of either experimental or quasi-experimental designs. Perhaps no one planned for an evaluation until the program was well under way, thereby ruling out randomization and providing no preprogram observations. Finding a comparison group may also prove too difficult or too costly. Under these conditions, the evaluator may be forced to rely on one or more of the pre-experimental designs as the principal outcome evaluation design, leaving the evaluation susceptible to many threats to internal validity.

Fortunately, means are available to compensate for if not to eliminate these design weaknesses. A first possibility is to use statistical controls. If their numbers and variability are sufficient, the subjects of a program can be divided for comparison and control. For example, a one-group pretest/posttest might be subdivided into those receiving a little of the program (x) and those receiving a lot (X). The resulting design becomes more like the stronger nonequivalent control group design:

There remains the question of whether the two groups are comparable in all respects other than the varying program involvement. If that comparability can be established, the design can provide a reading on whether more program involvement produces more impact, substituting for the unavailable comparison of program versus no program. The option to strengthen designs through statistical controls can also be useful with quasi-experimental and experimental designs. When a nonequivalent control group design is used, the evaluator may want to subdivide and compare subjects on variables on which the matching was flawed. If the two groups were matched on race and gender but not on education, the experimental and control groups might be compared while statistically controlling for education. Or, where a time series design is employed, additional data might be sought to control for threats of history. In a study of how the 55-mile-per-hour speed limit affected traffic fatalities, researchers examined data on total miles traveled to test an alternative explanation that fatalities declined as a consequence of reduced travel (amid the 1974–1975 energy crisis), not as a consequence of reduced speed (Meier and Morgan, 1981, pp. 670–671). The data added to the evidence that reduced speed was the cause.

Combining several outcome evaluation designs can also add to the strength of the overall design. Many evaluations will employ multiple designs, each for a different measure. Stronger designs on some measures might then help to compensate for the weaker designs necessary for other measures.

Assuming an outside evaluator is involved, these design decisions will be made principally by that individual. Still, to the extent that executives and staff understand these basic principles of evaluation design, they will be able to advise evaluators on these decisions. The nonprofit executive can perform an even more important role by monitoring the design planning to assure its fit to the purposes of the evaluation. The most rigorous design will be of no use unless it speaks to the issues of concern to the organization's board, executive and/or stakeholders. It is up to the executive to ensure that the evaluation remains relevant and appropriate to the organization's needs.

Process Evaluation

With most program evaluations, nonprofit executives will want to evaluate the program process as well as its ultimate impact. Outcome evaluation designs usually indicate only whether a program is working, not why. Process evaluation may be able to discern what steps in a program's process are not working as intended, perhaps pointing to how a program can be changed to increase its effectiveness. These suggestions will often prove the most useful.

The techniques of process evaluation are both simpler and less systematic than those for outcome evaluations (see also Thomas, 1980). In essence, process evaluation entails examining the internal workings of a program—as represented largely through activity goals—both for their functioning and for their role in producing the desired outcomes. It usually begins with the development of a good logic model (see Savaya and Waysman, 2005), then progresses to an examination of specific parts of that model.

The executives and staff of nonprofit organizations should be key actors in any process evaluation. To begin with, they should attempt to define at the outset the specific questions they have about the program's process. Conceivably, they may already feel adequately informed about performance as it pertains to some activity goals, and so may not desire new information there. They will then want to be certain that the evaluation includes the questions they do have about program process.

The basics of a process evaluation can be illustrated by the case of an affirmative action program designed to increase the hiring of minority firefighters by a municipal government. The activity goals of interest in this evaluation included the following:

1. Increase the number of minority applicants.
2. Increase the success rate of minority applicants on the written examination.
3. Increase the success rate of minority applicants on the physical examination.

These activity goals are designed to lead to this outcome goal (among others):

1.  Increase the proportion of minority firefighters in the fire department.

The several activity goals can illustrate how a process evaluation can be useful. Data on these various activities could indicate where, if at all, the program might be failing. Are too few minorities applying? Or are minorities applying only to be eliminated disproportionately by written or physical exams? Answering these questions could help a program administrator to decide whether, or how, and where to change the program.

A good process evaluation often can help to compensate for weaknesses in the outcome evaluation designs. When the difficulty of controlling for all threats to internal validity in an outcome evaluation design leaves unanswered questions about the linkage of program to outcomes, the process evaluation could provide an additional test of this linkage by documenting whether the program activities have occurred in a manner consistent with the observed outcomes. If an impact evaluation shows significant gains on the outcome measures and the process evaluation shows high levels of program activities, the evaluator can argue more convincingly that the program caused the impact. By contrast, evidence of low activity levels in the same scenario would cast doubts on the possibility that the program is responsible for outcome gains.

Most program evaluations should contain some form of process evaluation. Though less systematic than outcome designs, process evaluation techniques will often provide the more useful information for nonprofit executives.

Data Development, Report Writing, and Follow-Up

Nonprofit executives should plan to involve themselves and the program staff extensively in analysis and review of evaluation findings. This involvement is necessary first for accuracy: staff review of data and reports minimizes the risk of outside evaluators reporting inaccurate conclusions. Staff members also are more likely to utilize findings and implement recommendations that they helped to develop.

When outside evaluators are used, the best approach to this involvement may be to ask for the opportunity to review and comment on interpretations and reports while still allowing the evaluators to retain final authority on the substance of reports. Most evaluators should welcome this arrangement for self-protection; no evaluator wants to go public with conclusions that are subsequently shown to be erroneous. Staff might also be involved in basic data interpretation as, for example, by meeting with evaluators to review data printouts. As suggested earlier, interest among staff might be heightened by asking them to predict some of the results before the findings are in (Poister and Thomas, 2007).

The chief executive must also decide what final written products to request. A comprehensive evaluation report is usually desirable, both for the historical record and as a reference in case questions arise, along with a brief executive summary of one to three pages for broader distribution and readership. Other reports may be desirable for particular types of clients.

The job of the outside evaluator customarily concludes at this point, but the agency's chief executive and program staff should consider if and how the program should be changed in light of the evaluation. A program evaluation can provide both a direction and an impetus for change, but often with a limited window of opportunity to achieve any change. The agency's chief executive should take advantage of that window by discussing the evaluation with staff and, where appropriate, developing plans for what changes to make and how. Since the evaluation data presumably came from the agency's outcome assessment system, this is also a good time to consider any need to change that system. Only through such efforts can a nonprofit agency gain the full value of a program evaluation.

Summary

Nonprofit agencies today confront increasingly strong demands to demonstrate that their programs work. To meet these demands, contemporary nonprofit agencies must engage in systematic outcome assessment, measuring and monitoring the performance of their programs.

Outcome assessment data can speak to important questions of whether progress is being made on key agency objectives. As a result, every nonprofit agency, if it has not already done so, should consider if and how it can develop, collect, and analyze these data on a continuing basis.

Outcome assessment data alone can not speak to issues of causality, that is, to whether any observed changes resulted from a specific agency program or programs. Agency executives who wish to investigate those kinds of causal connections should consider taking a step beyond outcome assessment to conduct a program evaluation, too. Program evaluations build from a foundation of strong outcome assessment, adding the techniques of comparison and control necessary to speak to the role of specific programs in producing desired outcomes.

Success in these efforts may not come easily. For one thing, nonprofit agencies often need to find additional funds to support new initiatives in either outcome assessment or program evaluation. Yet as Carman (2007, p. 71) has observed, “although funders and other stakeholders may be asking [nonprofit agencies] to report on evaluation and performance information, most are not receiving separate funds or additional grants to collect this information.” In the long term, the solution may lie in these agencies “investing in their own evaluation capacity,” as Carman (2007, p. 73) recommends, but that strategy offers no help in the near term.

Even if the necessary funding can be found, success in either outcome assessment or program evaluation also requires a delicate balance of analytic and scientific expertise with group process skills. On the analytic side, nonprofit executives and staff should acquire at least a basic expertise, which can be supplemented as necessary with the talents of skilled consultants. On the group process side, nonprofit executives must ensure that any outcome assessment planning or program evaluation includes extensive participation of the agency's stakeholders, including at least the program staff and funders. Achieving that balance can give the executives and staff of nonprofit organizations the knowledge necessary to provide better programs and services.

References

1. Carman, J. G. Evaluation Practice Among Community-Based Organizations: Research into the Reality. American Journal of Evaluation, 2007, 28(1), pp. 60–75.
2. Edwards, D. J., and Thomas, J. C. Developing a Municipal Performance Measurement System: Reflections on the Atlanta Dashboard. Public Administration Review, 2005, 65(3), 369–376.
3. Fernandez, S., and Rainey, H. G. Managing Successful Organizational Change in the Public Sector. Public Administration Review, 2006, 66(2), 168–176.
4. Hatry, H. P. Using Agency Records. In K. E. Newcomer, H. P. Hatry, and J.S. Wholey (eds.), Handbook of Practical Program Evaluation (4th ed.) San Francisco: Jossey-Bass, 2015 325–343.
5. Hatry, H., and Lampkin, L. Key Steps in Outcome Management. Washington, D.C.: The Urban Institute, 2003.
6. Herman, R. D., and Renz, D. O. Advancing Nonprofit Organizational Effectiveness Research and Theory: Nine Theses. Nonprofit Management & Leadership, 2008, 18(4), 399–415.
7. W. K. Kellogg Foundation. Logic Model Development Guide. Battle Creek, Michigan: Author, 2004.
8. Kaplan, R. S., and Norton, D. P. The Balanced Scorecard: Translating Strategy into Action. Boston: Harvard Business School Press, 1996.
9. Kress, G., Springer, J. F., and Koehler, G. Policy Drift: An Evaluation of the California Business Enterprise Program. Policy Studies Journal, 1980, 8, 1101–1108.
10. McLaughlin, J. A., and Jordan, G. B. Using Logic Models. In K. E. Newcomer, H. P. Hatry, and J.S. Wholey (eds.), Handbook of Practical Program Evaluation (4th ed.) San Francisco: Jossey-Bass, 2015. 62–87.
11. Meier, K. J., and Morgan, D. P. Speed Kills: A Longitudinal Analysis of Traffic Fatalities and the 55 MPH Speed Limit. Policy Studies Review, 1981, 1, 157–167.
12. Minich, L., Howe, S., Langmeyer, D., and Corcoran, K. Can Community Change Be Measured for an Outcomes-Based Initiative? A Comparative Case Study of the Success by 6 Initiative. American Journal of Community Psychology, 2006, 38, 183–190.
13. Morley, E., Hatry, H., and Cowan, J. Making Use of Outcome Information for Improving Services: Recommendations for Nonprofit Organizations. Washington, D.C.: The Urban Institute, 2002.
14. Newcomer, K. E., and Triplett, T. Using Surveys. In K. E. Newcomer, H. P. Hatry, and J. S. Wholey (eds.), Handbook of Practical Program Evaluation (4th ed.) San Francisco: Jossey-Bass, 2015, 344–382.
15. Patton, M. Q. Utilization-Focused Evaluation, 4th ed. Thousand Oaks, Calif.: Sage, 2008.
16. Poister, T. H., and Thomas, J. C. The ‘Wisdom of Crowds’: Learning from Administrators Predictions of Citizen Perceptions. Public Administration Review, 2007, 67(3), 279–289.
17. Rea, L. M., and Parker, R. A. Designing and Conducting Survey Research: A Comprehensive Guide, (4th ed.) San Francisco: Jossey-Bass, 2014.
18. Rossi, P. H., Lipsey, M. W., and Freeman, H. E. Evaluation: A Systematic Approach (7th ed.) Newbury Park, CA.: Sage, 2004.
19. Savaya, R., and Waysman, M. The Logic Model: A Tool for Incorporating Theory in Development and Evaluation of Programs. Administration in Social Work, 2005, 29(2), 85–103.
20. Stone, M., Bigelow, B., and Crittenden, W. Research on Strategic Management in Nonprofit Organizations: Synthesis, Analysis, and Future Directions. Administration & Society, 1999, 3, 378–423.
21. Thomas, J. C. ‘Patching Up’ Evaluation Designs: The Case for Process Evaluation. Policy Studies Journal, 1980, 8, 1145–1151.
22. Thomas, J. C., Poister, T. H., and Ertas, N. Customer, Partner, Principal: Local Government Perspectives on State Agency Performance in Georgia. Journal of Public Administration Research and Theory, 20(4), 2010, 779-799.
23. United Way of America. Outcome Measurement Resource Network. http://www.liveunited.org/outcomes/. Visited December 17, 2009.
24. Weiss, C. H. Evaluation Research: Methods of Assessing Program Effectiveness. Englewood Cliffs, N. J.: Prentice-Hall, 1972.