Measuring Level 1

Level 1, or reaction, can be easily measured during the training event or immediately following it, preferably before participants leave the classroom. A questionnaire, composed of both questions with a rating scale and open-ended questions, is usually used. Some trainers allow participants to take the evaluation with them and/or go online to complete it after the session. The drawback is a lower return rate.

How do you begin?

1. Determine what you want to learn about the training.

   You will most likely want to know something about the content, including its relevance to the participants’ jobs and the degree of difficulty. You will also want to gather data about the trainer including effectiveness, communication skills, the ability to answer questions, and approachability.

2. Design a format that will be both easy for participants to complete and presents a way for you to quantify participant responses.

   Many formats exist, and the design will be a factor of the first question — What do you want to learn? You may choose to have statements rated on a one- to seven-point scale representing strongly disagree to strongly agree (or poor to excellent). You may also wish to ask open-ended questions, or you may choose to do a combination of both types of questions. Even if you develop a format that has questions that are rated on a scale, it is a good idea to add space at the end for comments.

3. Plan to obtain 100-percent response ratings that are complete.

   How do you accomplish all that? Most of it is in the timing. Ensure that you plan time into the training design to complete the questionnaire. Twenty minutes prior to the end of the training session pass out the evaluations and ask participants to complete them. It should take only ten minutes, allowing you time to facilitate your closing activity ensuring the learners’ last interactions are memorable and positive.

Trainers desire honest responses to their evaluations. To ensure honesty, allow for anonymity. Don’t require participants to sign their evaluations. In addition, ask them to drop the evaluations off on a table on their way out versus leaving them at their places. If you have a valid reason for wanting to know who made specific comments, for example, to provide feedback, you may wish to make signing the evaluation optional.

Try SurveyMonkey or another similar format completed at the end of the class on the learners’ tablets or laptops. Easy, fast, and still anonymous.

Your training department will most likely have determined an acceptable standard against which you will measure results. For example, 5. 5 on a 7. 0 scale may be considered acceptable by your training department.

If you follow these guidelines, you will be well on your way to finalizing an effective Level 1 evaluation. A comprehensive training-competency checklist can be found at the end of this chapter. You may want to use some aspects of it to create your Level 1 evaluation. You and your colleagues may want to use the evaluation to provide feedback to each other on your training skills.

Some training practitioners snub Level 1 evaluation as too insignificant or perfunctory. Perhaps this is because trainers can conduct more sophisticated evaluations or because it’s prudent to tie training to the bottom line. The “smile sheet,” as Level 1 is known, has become a pejorative term to some. Don’t fall into that way of thinking. Dr. Kirkpatrick’s Four Levels are built upon each other. Level 1 is a prerequisite when other levels are used. Even though you’re “only” measuring reactions, these are the first-hand reactions of the customers of the training. There are very few more effective and efficient ways for trainers to obtain feedback about performance. You may have the best training design, but if it is delivered poorly, it does affect participants’ ability to learn. Level 1 evaluation also serves as a way to focus questions on the content. The key here is to design an instrument that measures the essential foundation elements of the training content and how well it was delivered.

Hold on to your evaluations. Use the ideas and suggestions they contain to improve your future performance and program designs. The evaluations can also serve as input to your efforts to continually develop your skills and competencies.

Measuring Level 2

Level 2, or learning, can be measured using self-assessments, facilitator assessments, tests, simulations, case studies, and other exercises. This type of evaluation should be conducted at the end of the training before participants leave to measure the degree to which the content was learned. Use pre- and post-test results to compare improvement.

Measuring before and after the training session gives you the best data because you will be able to compare the knowledge level prior to the session and after the training has been completed. This is the best way to determine whether participants gained knowledge or skills during the session. How do you evaluate? Remember KSAs? That’s what you will measure: the knowledge, skills, and attitudes that the training session was designed to improve. Use tests to measure knowledge and attitude surveys to measure changing attitudes. To measure skill acquisition, use performance tests, where the participant is actually modeling the skill. Like Level 1, attempt to obtain a 100-percent response rate.

Some trainers use a control group for comparison. And Dr. Kirkpatrick recommends that you do so if it is practical. While this may be the most scientific way to gather data, it may be wasted time. For example, if participants have another way to learn between pre- and post-tests, then don’t send them to training. On the other hand, if training is the only way that participants can gain the knowledge, skills, and attitudes, then why bother with a control group? You’ll need to be your own judge about whether a control group is beneficial.

Many types of testing formats exist from which to choose, and each brings with it advantages and disadvantages. True/False tests are easy to develop and cover many questions in a short amount of time. The drawback is that if the test is too easy, having superficial knowledge may lead to an inflated score. Be sure to use a subject matter expert (SME) to assist with the design.

Other options that exist include oral tests, essay tests, multiple-choice tests, and measured simulations such as in-basket exercises, business games, case studies, or role-plays. Assessments may also include self-assessments, team assessments, or instructor assessments. A Web-based evaluation may also be created. Although Web-based assessments are cost and time efficient, they bring a couple problems, including guaranteeing that participants are who they say they are when signing in and the ability to protect the questions in the exam banks. Most organizations using Web-based assessments have addressed both of these issues. Whatever testing option you use, be sure that the results are quantifiable.

Create straightforward tests. You’re not trying to trip up the participants. You’re trying to measure the learning that occurred to ensure improved job performance.

Finally, ensure that testing conditions are optimized by limiting noise, having adequate lighting and comfortable temperature, and eliminating interruptions. Be certain that the test is only as long as it needs to be. Be sure that participants know the rules, such as whether it is acceptable to ask questions during the test.

Pilot any test prior to using it with participants. Return measurement results to employees as quickly as possible. The significance of measurement feedback decreases with time.

Measuring Level 3

Level 3, or behavior, is used to determine the successful transfer of learning to the workplace. Unlike levels one and two, you need to allow time for the changed behavior to occur. How much time? The experts differ, and with good reason. The amount of time required for the change to manifest itself will be dependent on the type of training, how soon the participant has an opportunity to practice the skill, how long it takes participants to develop a new behavioral pattern, and other aspects of the job. So, how long? Anywhere from 2 to 8 months. You will probably need to work with the subject matter expert (SME) to determine the length of the delay required to allow participants an opportunity to transfer the learning (behavior) to the job.

By the way, like Level 2, a pre- and post-testing method is recommended. And again, the question of whether you want to incorporate a control group for comparison needs to be decided.

What do you measure? Each item in an evaluation instrument should be related to an objective taught as part of the training program. Based on the training objectives, you will create a list of skills that describe the desired on-the-job performance. Use a SME to assist you with the design of the evaluation. They will understand the nuances of being able to complete a task. For example, a skill may be “uses the four-step coaching model with employees.” A SME will know that even though the four steps are essential, what truly makes the model successful is the supervisor’s “willingness to be available to respond to questions at any time. ” Knowing this, your checklist of skills will be expanded to include availability.

After you have identified the skills to measure, you select an evaluation method. Evaluation tools may include interviews, focus groups, on-site observations, follow-up questionnaires, customer surveys, and colleague or supervisory input.

Level 3 evaluations should not be taken lightly. It takes a major resource investment. Know what you will measure; know how you will measure; and most important, know how you will use the data.

All trainers should be skilled in Level 3 evaluation methods. The most important reason to conduct training is to improve performance and transfer knowledge, skills, and attitude changes to the workplace. Results cannot be expected unless a positive change in behavior occurs. Therefore, use a Level 3 evaluation to measure the extent to which a change in behavior occurs.

What do you do if the results show that performance has not been changed or skills mastered? You need to go back to the training design. Certainly the first step is to determine whether the skill is required. If yes, examine the training material to ensure that appropriate learning techniques have been used and that enough emphasis has been placed on the skill. Perhaps a job aid is required to improve performance. Sometimes, you may discover something that did not show up in the needs assessment. For example, you may discover that participants are not using the skill because it is no longer important or is not frequently used on the job. In that case you may want to remove it from the training session. This is a perfect example of how the fifth stage of the Learning Cycle feeds back into the first stage.

Getting results that demonstrate performance has not improved is not what a trainer wants to hear. However, it is good to have the knowledge to make an intelligent decision about whether to maintain the training program, overhaul it, or scrap it entirely. Without a Level 3 evaluation, you would not likely be able to make a wise decision.

Measuring Level 4

Level 4, or results, may be the most difficult to measure. Dr. Kirkpatrick frequently stated that the question he was asked most often was, “How do you evaluate Level 4?” Even though Level 4 is the most challenging, training professionals need to be able to demonstrate that training is valuable and can positively affect the bottom line.

One of the issues of evaluating at this level is that you can never be sure whether external factors affected what happened. There is always a possibility that something other than the on-the-job application contaminated the results. Can you really isolate the effects training has on business results? This is one time that using a control group can be helpful to the evaluation results. Yet even with a control group there may be other factors that impact the business, such as the loss of a good customer, the introduction of a new competitor, a new hiring practice, or the economy. Several statistical methods are available to you to consider other evidence. I don’t cover them in this book.

So what do you do when management asks you to provide tangible evidence that training is having positive results? A before-and-after measurement is relatively easy because records for the kinds of things you measure (turnover, sales, expenses, errors, grievances) are generally available. The trick is to determine which figures are meaningful.

A second difficultly is predicting the amount of time that should be allowed for the change to take effect. It may be anywhere between 9 and 18 months. Gather the data that you believe provides evidence of the impact of the training. This measurement usually extends beyond the training department and utilizes tools that measure business performance, such as sales, expenses, or rework. Remember, a key issue is to try to isolate training’s impact on results. You may not be able to prove beyond a doubt that training has had a positive business impact, but you will be able to produce enough evidence so that management can make the final decision.

You’ve read in this book and other places that training should align to the business’s requirements. How do you do that? Dr. Kirkpatrick suggests that his evaluation levels can be used. Start with Level 4. Determine, with the business managers, what needs to occur (results). What are their “expectations”? Next decide what performance (behaviors, or Level 3) is required to achieve the desired results. Then identify the knowledge, skills, and attitudes (KSAs, or Level 2) employees require to achieve performance expectations. Finally, create a plan to offer the correct training, ensuring employees will be receptive to the changes. Kirkpatrick Partners calls this Return on Expectations.

Which evaluation level to use?

You may choose to evaluate training at one or all levels. How do you decide? Base your decision on answers to the following questions:

• What is the purpose of the training?
• How will you use the results of the evaluation?
• What changes or improvements may be made as a result of the evaluations?
• Who are the stakeholders and what do they want you to measure?
• What are the stakeholders’ expectations?
• What business goals will the training impact?
• What resources will be required for the evaluation?

Just because there are four evaluation levels doesn’t mean that you should use all four. After answering the preceding questions, decide which levels will be the most beneficial for each training program. Evaluation experts agree that Level 3 and especially Level 4 should be used sparingly due to the time and cost involved. A rule of thumb seems to be use Level 4 in those situations where the results are a top organizational priority or for training that is expensive.

Objective test formats

This method measures how well trainees learn program content. A facilitator administers paper-and-pencil or computer tests in class to measure participants’ progress. The test should measure the learning specified in the objective. Tests should be valid and reliable. Valid means that an item measures what it is supposed to measure; reliable means that the test gives consistent results from one application to another.

Multiple-choice questions take time and consideration to design. However, they maximize test-item discrimination yet minimize the accuracy of guessing. They provide an easy format for the participants and an easy method for scoring.

True/False tests are more difficult to write than you may imagine. They are easy to score.

Matching tests are easy to write and to score. They require a minimum amount of writing but still offer a challenge.

Fill-in-the-blank or short-answer questions require knowledge without any memory aids. A disadvantage is that scoring may not be as objective as you may think. If the questions do not have one specific answer, the scorer may need to be more flexible than originally planned. Guessing by learners is reduced because there are no choices available.

Essays are the most difficult to score, although they do measure achievement at a higher level than any of the other paper-and-pencil tests. Scoring is the most subjective.

Attitude surveys

These question-and-answer surveys determine what changes in attitude have occurred as a result of training. Practitioners use these surveys to gather information about employees’ perceptions, work habits, motivation, values, beliefs, and working relations. Attitude surveys are more difficult to construct because they measure less tangible items. There is also the potential for participants to respond with what they perceive is the “right” answer.

Simulation and on-site observation

Instructors’ or managers’ observations of performance on the job or in a job simulation indicate whether a learner is demonstrating the desired skills as a result of the training. Facilitate this process by developing a checklist of the desired behaviors. This is sometimes the only way to determine whether skills have transferred to the workplace. Some people panic or behave differently if they think they are being observed. Observations of actual performance or simulated performance can be time-consuming. It also requires a skilled observer to decrease subjectivity.

Criteria checklists

Also called performance checklists or performance evaluation instruments, these are surveys using a list of performance objectives required to evaluate observable performance. The checklists may be used in conjunction with observations.

Productivity or performance reports

Hard production data such as sales reports and manufacturing totals can help managers and instructors determine actual performance improvement on the job. An advantage of using productivity reports is that no new evaluation tool must be developed. The data is quantifiable. Disadvantages include a lack of contact with the participant and that records may be incomplete.

Post-training surveys

Progress and proficiency assessments by both managers and participants indicate perceived performance improvement on the job. Surveys may not be as objective as necessary.

Needs/objectives/content comparison

Training managers, participants, and supervisors compare needs analysis results with course objectives and content to determine whether the program was relevant to participants’ needs. Relevancy ratings at the end of the program also contribute to the comparison.

Class evaluation forms

Sometimes called a response sheet or smiley sheet, participants respond on end-of-program evaluation forms to indicate what they liked and disliked about the training delivery, content, logistics, location, and other aspects of the training experience. The form lets participants know that their input is desired. Both quantitative and qualitative data can be gathered.

Interviews

Interviews can be used to determine the extent to which skills and knowledge are being used on the job. They may also uncover constraints to implementation. Like no other method, interviews convey interest, concern, and empathy in addition to collecting data. They are useful when it isn’t possible to observe behaviors directly. The interviewer becomes the evaluation tool, and this can be both an advantage and a disadvantage. Interviews are more costly than other methods, but they give instant feedback, and the interviewer has the ability to probe for more information.

Instructor evaluation

Professional trainers administer assessment sheets and evaluation forms to measure the instructor’s competence, effectiveness, and instructional skills. See an example at the end of this chapter.

All of these evaluation methods work. All give you the information you need. Some work better than others for each of the Four Levels. The final decision about which method to use will be yours.

It is easy to grow accustomed to your favorite evaluation method. It is much better to incorporate variety in the evaluation methods used. Gaining expertise with several makes you more skilled in knowing which method will be best for each situation.

Discussions with employees on an individual basis can give insight into the effectiveness of the training session. Are there things you’re doing differently as a result of participating in the training program? Have you noticed changes in your attitudes or relationships that may be related to the training program?