4
CHAPTER FOUR
Interpret and Communicate Evaluation Findings
Evaluations fail to live up to their full promise when what is learned is not appropriately communicated or goes unused. It's important that stakeholders understand the evaluation process doesn't end when all of the data have been collected, analyzed, and interpreted. On the contrary, this stage is the time to revisit the reasons the evaluation was originally commissioned. To ensure that your evaluation findings will be used to support learning, implement decisions, and make improvements, you need to communicate the results. Depending on your role, you may also be involved in identifying specific courses of action, developing a detailed action plan, and monitoring the action plan's implementation.
The activities we describe in this section create a greater likelihood that the evaluation recommendations will be carefully considered and translated into action and that the actions taken will be realistic and sensitive to the relevant cultures and contexts. These activities will also help stakeholders identify and manage any potential barriers or obstacles to change, and they will allow those individuals affected by the changes to get involved in planning the changes. Perhaps most importantly, these activities lay the groundwork for the stakeholders to use the evaluation findings as part of a continuous process of improvement and learning (Preskill & Torres, 1999). When consulting with stakeholders about findings, the purpose is not to change the findings but to discuss the findings and understand and integrate stakeholder perspectives. For instance, footnotes about actions that will be taken in response to negative findings may make it easier to communicate the negative findings and to point out that action is already being taken as a result of the evaluation. As you work through the communication needs, consider the following:
Who needs to know?
What do they need to know?
Why do they need to know?
How can they best know?
When do they need to know?
Include and Engage Stakeholders
Evaluations can result in a final report that is submitted to the primary stakeholder, client, or foundation. All too often, few people read that final report, so the degree of change that occurs as a result of the evaluation is limited. Evaluators can counter this tendency by carefully considering their interpretation and reporting strategy while keeping the goal of a usable evaluation in mind. Organizations will more often use evaluation reports that are credible, relevant (to stakeholders' interests, expectations, and communication preferences), accurate, and fair.
To effectively communicate, you will need to consider what needs to be communicated to whom and how you will distribute the information. Including and engaging stakeholders, as appropriate, throughout the process by distributing interim and draft reports before the release of a final report is a useful and effective sensemaking and communication tactic. This transparent and frequent communication enables you to better understand what information is helpful as well as see how different groups interpret and use data.
Developing and applying an evaluative rubric (described in Chapter Two) is one way to engage stakeholders in interpretation. You may have developed an evaluative rubric during the planning phase of the evaluation. If so, now is a good time to revisit, update, and apply the rubric. If you didn't previously create a rubric, it isn't too late. Although creating a rubric during the planning phase helps make stakeholder expectations explicit before data collection begins, introducing a rubric approach during the interpretation phase helps to make the interpretation within and across stakeholder groups transparent. Rubrics can be used to combine information from different sources to arrive at an evaluative judgment—for instance, is the program good enough to warrant continued investment? Developing a rubric can document what data were taken into account in making that judgment. In short, the rubric documents how evidence was used to arrive at an evaluative decision of “good” or “bad,” or what needs improvement and what functions well.
Using more visual approaches for data sharing can help stakeholders understand data more quickly and engage with it more readily. See below for some tips on how to make your visualization effective.
Exhibit 4.1 – Data Visualization Tips
Communicating with data visualization is increasingly commonplace and necessary for better understanding and use of evaluation. Pictures and graphs can be extremely valuable for communicating information quickly and easily to multiple audiences. Some suggestions for data visualizations include:
Consider the elements of an effective visual, including
• data density (the amount of data in a given space is not too overwhelming);
• visual appeal (the balance of size, color, and shape of the visual); and
• ease of understanding (the interpretability of the information presented).
Try to understand the context and informational needs of your client group.
Consider using software that you already use to produce pieces of powerful visuals. For example, you can expand your skills using presentation or spreadsheet applications.
Use data labels and color but understand how and when to use them.
Balance creativity with a simple and clean design.
Isolation and color can be a great tool for stressing an important point or finding.
Use Multiple and Targeted Communication Approaches
Evaluation results are not the final goal—moving toward improvement and impact are. To make sure the evaluation reaches people and that the information is acted upon, consider using multiple methods for communicating and reporting the process of the evaluation and its results. A broad communication strategy can help distribute the lessons arising from development initiatives and emphasize that the results of an evaluation provide information that can and should be used in an organizational-learning process. Informing a variety of audiences about the findings of the evaluation and how the results will be used increases the credibility of the evaluation effort, communicates to stakeholders that they were heard and that the results will be acted on, and prepares stakeholders for future evaluations by demonstrating their potential value. At a broad level, communication channels for evaluation information can include e-mail, web pages, face-to-face and virtual meetings, and paper-based collateral (flyers, posters, and so on). Some of the more specific approaches we use most often include:
presentations with handouts (in person or virtual);
executive briefings of key information;
interactive dashboards of key data;
illustrative case studies or vignettes;
e-mail summaries of highlights and recommendations;
videos of stakeholders talking about the importance, process, or impact of the initiative;
infographics of key data and insights;
photo essays that show the process or impact of the initiative; and
town-hall-style meetings.
Tailoring and presenting information to effectively reach and inform stakeholders is critical. For example, high-level leaders and executives do not have time for or interest in a complete and detailed account of findings. They often want results and information that helps them make necessary changes or decisions. Finally, when a development initiative is designed to occur in stages or has separate components, evaluation reports can be issued at each stage as well as rolled up in a final, comprehensive report. We outline some of the basic elements of such a report in Exhibit 4.2.
Tailoring the Report to Stakeholder Interests
Information about the stakeholder group's background, communication preferences, and information needs should be available from the initial planning phase of the evaluation; if it is not available, ask for it. What information do stakeholder groups want? When do they need or want the information? How do they best receive information? The most wonderful evaluation is not worth much if the key stakeholders do not know about it or understand how to make sense of or use the information. Strategically thinking about your communication approach will improve the likelihood that people are getting the information they need and can use it.
Exhibit 4.2 – Basic Elements of a Final Evaluation Report
A final evaluation report should be clearly written and tailored specifically to its intended audience. Avoid words that may be unfamiliar to the intended audience, and, if you must use technical language (or jargon), explain it in layperson terms. Use graphs or narratives to illustrate a point. The report should be attractive and professional, and it should not be overwhelming; use headings and page numbers to help orient the reader. When appropriately used, color images, photographs, call-out boxes, and dashboards can add interest and clarity to the report. Do not neglect to pay attention to correct grammar, spelling, and punctuation. For organizing purposes, the list below can be a helpful guide.
The most useful evaluation reports provide both critical findings and recommended actions. Once you have identified these key elements of the report, you will want to make sure they stand out and are easily understood. The actions must be specific and follow logically from the evaluation results.
We recommend Stephanie Evergreen's book, Effective Data Visualization, as an excellent resource for how to organize an evaluation report so it is most useful and interesting to stakeholders (Evergreen, 2016). Although most evaluation reports include the following key elements, descriptions of the initiative and evaluation methods can often be included in appendices so the report focuses primarily on what the data tell us about the initiative.
Executive Summary: Highlight the most important parts of the findings, including recommendations.
Purpose of the Evaluation: Indicate why the evaluation was conducted, including the core evaluation questions and stakeholders.
Conclusions and Recommendations: This section should be extremely clear so that your stakeholders know exactly what has occurred and what changes they may need to make.
Summary of Data (with multiple subsections): The section could be divided by evaluation questions, by results related to the implementation of the initiative as opposed to the outcomes (short, mid, and long term) of the initiative, or by overall lessons learned. The organization of sections depends very much on what was examined by the evaluation, what was found, and what would be the most salient way to present them to your stakeholders.
Description of Initiative and Sample: Include the title of the initiative and what the initiative sought to accomplish and who participated in it. Basic demographic information about participants can be helpful to include, as well as a description of the stakeholders from whom data were collected for the evaluation.
Overview of Processes Used and Analysis: What data were collected, how, and when? Was there anything out of the ordinary about the methodology or processes used? Provide samples of your evaluation methodologies as appendices (for example, survey questions).
Limitations of the Evaluation: Usually evaluators are not able to gather data from all stakeholder groups or all people within a stakeholder group. It is important to acknowledge what information or perspectives might be missing.
Appendices (supporting materials): These can include more detail on methods used, a copy of data-collection tools, data-analysis tables (detailed), and other resources that are pertinent to understanding the report but may not be of interest to all readers.
Checklist – Chapter Four
Interpret and Communicate Evaluation Findings
Draft the final evaluation report with key stakeholders in mind. Consider how best to inform and engage the stakeholders so the findings will be used to learn, make improvements, and increase impact.
Include and engage stakeholders by asking for their feedback and input on draft reports and recommendations.
Revisit and use your evaluative rubric to synthesize information from different sources and determine evaluative judgments.
Use graphics and visual approaches to engage stakeholders and make it easy to understand data highlights.
Use multiple and targeted communication approaches tailored to how stakeholders will best receive the information (for instance, one-page handout with summary of findings and recommendations, video summary, and PowerPoint).