One summer, as a teenager, I worked with several other young people to help restore an old farmhouse into what would become a youth camp. Our supervisor, Mr Abuhl, was an older gentleman who had volunteered his time and talents and was intent on improving our renovation skills.
Many people were surprised at the quality of work Mr Abuhl was able to produce with a few 15–17‐year‐old teenagers. Those of us that worked for him were not surprised at all. His success came, not by cajoling more work out of us, but by constantly evaluating the work that we did. He would meticulously inspect our work to confirm that we were meeting his stringent (at least from a teenager’s perspective) standards. “Work that is expected must be inspected,” he would say. It was his way of telling us that his re‐measurements and careful scrutiny of our work were not because he didn’t trust us, but because detecting small deviations now could help prevent larger problems later on.
The same is true with training. Whenever you provide learning, you must verify that the training did what you intended it to do. If you don’t, no one may notice for a while, but soon your training will be deemed ineffective. Even if it isn’t and your training is amazingly effective, or generates tremendous revenue, few will care unless you can prove it.
In the 1950s, Donald Kirkpatrick suggested four areas that must be evaluated in any training program. They have become known as Kirkpatrick’s evaluation model.1 First, trainers should measure the student’s reaction to the training. The traditional way many do that is through what we sometimes refer to as smile sheets or surveys done right after class to see if students liked the training. Second, you should determine if the individual students can meet the objectives of the course. The traditional way to conclude that is through a written test or an exam. Third, he suggests that instructors must measure behavior, or the change in their ability to do something specific. The traditional way to do this is through follow‐up and on‐the‐job analysis after the training. Finally, trainers should validate that the class itself is getting the business results you want from the class. The traditional way to that is by demonstrating that you brought in more money than you spent.
Because proficiency training must include a change in behavior, or no real learning has occurred, I will combine levels 2 and 3. You must evaluate the course (level 1), you must assess the individual (levels 2 and 3), and you must measure the value obtained (level 4). All three are important to validate success.
It is a mistake to assume the Four‐Level Training Evaluation Model is a sequential evaluation model. There are four different things being measured, but they don’t always happen in that order. As I stated earlier, you must start with the end goal in mind. Also, you may assess the individual during a class and take a survey after the class. You may know what the business value is even before the students have time to demonstrate a change in behavior.
Earlier chapters dealt with evaluating the business goal of the class. This chapter deals with evaluating the class and assessing the individuals.
In product training, there are two main types of individual assessments that you should consider: you should assess the student’s overall knowledge and you should assess their skill level. Both are important in order to determine if you have increased your students’ proficiency on your product. This is especially true if there is going to be any kind of proficiency certification granted.
Quizzes and exams are the most common way to assess for general knowledge. Both can be helpful to determine if students were able to process the material that you covered. Asking good test questions is a skill that may require seeking additional help. There are many resources available for learning how to write good quiz and exam questions. Some tools, like Questionmark.com, offer whitepapers and other helpful tools to create effective exam questions. This chapter is not a comprehensive guide to creating a quality quiz or exam.
Quizzes should be short and generally follow the module or lesson immediately, often using the same wording used in the lesson. The goal of a quiz is to make sure that students are keeping up with the information flow. Quizzes help you, the instructor, know if you need to cover something again, if you need to slow down, or if your students are with you or not. Quizzes can serve multiple purposes, like being a means of student engagement or preparing students for a later exam. Quizzes can even be given merely to take attendance. Whatever the reason, you should know what it is.
When you ask a quiz question, you should always review the answers with the class. Make sure that all of the students know the correct answer before moving on. I rarely grade a quiz, though students may see a similar question in the exam.
One way to include regular feedback, or quizzing of the students, is by using an electronic polling tool. A good polling tool or mobile application allows for all students to answer questions. Using these types of engaging tools will increase your effectiveness by adding interaction with all of your students. Another benefit is that it will force you to slow down and ask more questions.
Exams, or tests, are a more formal method of measuring a student’s comprehension. Remember that some students are better test takers than others. Also, if you don’t carefully evaluate your questions or requirements, there may be little to no correlation between a student’s exam score and their skill level. In some cases, it may be necessary to forego a written exam altogether and concentrate on a competence or skill exam.
The best way is to use both. It is very likely that your driver’s license required both a written exam and a practical exam. There are things like road signs and traffic laws that are important for you to demonstrate knowledge of, even if you are an excellent driver. The same is often true of your products.
If you do choose to create a written exam, here are a few tips that can help.
There are a number of different types of question formats you can use to ask questions, and all have their strengths and weaknesses. There are also a number of things you should do when writing exam questions. Here is a list of a few of those:
Answers B, C, and D are the distractors. None of them are plausible. This question, at least with these answers, is not a good test of comprehension.
If possible, use a tool that allows for flexibility when writing your exams. Don’t be afraid to use an online tool just because your class is offered in a traditional setting. Online exams are not exclusive to eLearning. They can just as easily be administered as part of a classroom learning event.
A good testing tool will allow you to ask multiple types of questions. Another important feature is question pooling. Pooling means that you create multiple questions for each objective and then ask for a random selection from each of the “pools.” For example, if you have five objectives, you might create ten questions for each objective. Those questions are stored in separate folders. When you create the exam, you may require two random questions from each folder, or pool, for a total of 10 questions. This means that no two students will get exactly the same exam, which helps to eliminate cheating.
Don’t feel like you have to hand out a certificate at the end of every class. This works fine when the certificate merely awards attendance, but it rarely works in a certification setting or when not all students are guaranteed to pass the course. Many online programs allow students to print their certificates, which may be the answer. Another answer is to simply mail certificates after you have had a chance to review exam and lab scores. While this creates an extra administrative step, it allows you another opportunity to connect with your students.
If you are not going to administrate an exam, you may choose to use one or more of the activities you designed in step 5 of the 4 × 8 Proficiency Design Model (see Chapters 8 and 9) as a skill assessment. These types of assessments can add significant value to your training, but they require significantly more effort to administrate. If you use a skill assessment, it is important to carefully define the skill you are assessing. Otherwise, you will be left guessing or hoping that students actually did the required work.
Another item that is required is what educators refer to as a grading rubric. You don’t need to call it anything fancy, but you do need one. A rubric is simply a measuring guide—often in the form of a spreadsheet—that ensures consistency and impartiality in your grading. A good measuring spreadsheet will still allow for some flexibility without being too rigid. The example in Table 7.1 shows a very simple spreadsheet, where the skill being assessed is a student’s ability to upgrade a software program.
Table 7.1 Rubric.
Points | |||
Skill | 1–3 | 4–6 | 7–10 |
Software upgrade | Upgraded from v.2.3 to 3.0 with significant help or in greater than 10 minutes | Upgraded from v.2.3 to 3.0 with moderate help or between 5 and 10 minutes | Upgraded from v.2.3 to 3.0 with no help in under 5 minutes |
Notice both the flexibility and the detail—the combination of both subjectivity and objectivity. This ensures against the extremes but allows for instructors to weigh in on the nonmeasurable items as well. “Significant” and “moderate” are both subjective measurements. What the students are doing (upgrading from version 2.3 to 3.0) is very specific. The timing has elements of both, giving you a little flexibility without having a new column for each possible minute.
A grading matrix is not only a helpful tool; it may be a requirement if you decide to create a certification course. Even if an instructional designer is creating the grading matrix, you will likely need to assist them. You are the subject matter expert, and you know what should be measured and what shouldn’t. It may, however, be easy to get caught up in too many details. Keep your grading as simple and practical as possible. A tool that is too complicated to use never gets used. The end result should be a consistent determination of who is qualified to do the job and who needs more practice.
As long as you can stay consistent in your measuring of students, you can get creative in your assessments. A practical exam may simply consist of a series of troubleshooting problems that students had to complete. It may include a check‐off list of items for students to self‐grade their capabilities in certain areas. In many product training settings, these types of assessments are fine to use, as long as you don’t treat some students more favorably than others.
There are other ways, of course, to test your students. Exams don’t have to be a series of written questions. They can be delivered orally, or it can be written as a paper or thesis. Remember, however, that this is product training, not a graduate degree!
Whenever possible, don’t use only the knowledge assessment as the pass/fail indicator for the course. If only one can be used, it should be the competency exam, but generally the grades should be combined. There is not a set formula or weight that is used when combining a written exam and a practical exam. It all depends on your product. This is one of the reasons why you, the expert, are involved in the training on your product. What is important, however, is consistency. You must be able to apply the same rules to everyone.
Take the time to write out the process, and make sure it will work for your most experienced trainees as well as students new to your product. If you decide to allow past experience to qualify in place of a practical assessment, be sure to qualify the expectations. Keep the process as simple as possible, but detailed enough to maintain a fair standard.
Evaluating whether students pass a class is easy enough. Evaluating whether the class is really teaching the right thing and helping students to improve their performance and proficiency is quite another.
If you are like I was for a many years, you may administer surveys because you’re expected to, not because you see any true value in them. The results can be confusing and difficult to discern between what is good and what is bad (is a 4.2 average bad and a 4.4 average good?). You may be a better judge of whether a class went well or not than your students. When this is true, you will be tempted to throw out results or even stop doing surveys altogether. There is no need to waste your students’ time and force them to quickly fill out one more thing before they can leave.
But if you truly want to make every class better than the last one, you must know what needs to be improved. Continuous improvement doesn’t happen without a structure and plan. Eventually, you will need some evidence of improvement, or at least some ideas for areas to improve on.
The problem lies in the typical Likert‐like survey question, as shown in Table 7.2. In the training industry, this type of survey is often referred to as a “smile sheet.” It merely asks the students if they liked certain things about the class or not.
Table 7.2 Likert‐like survey.
On a scale of 1–5 (5 being best), please rate the following: | |||||
Content of the class | 1 | 2 | 3 | 4 | 5 |
Length of the class | 1 | 2 | 3 | 4 | 5 |
Instructor’s presentation skills | 1 | 2 | 3 | 4 | 5 |
Others | 1 | 2 | 3 | 4 | 5 |
Kendall Kerekes and John Mattox of CEB2 suggest ditching the useless “smile sheets” that merely ask if a student liked the training. They suggest using “SmartSheets”—evaluations that predict performance improvement—instead. They offer these alternative questions:
Those are big improvements. They begin to move out of the reactionary and toward application. In his book Performance‐Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form,3 Dr Will Thalheimer suggests, as the title indicates, an even more radical approach to gathering feedback. If you are involved in writing survey questions for your training, I strongly recommend the ideas he promotes, all backed by research.
Dr Thalheimer highlights several issues with traditional smile sheets and with the Likert‐like scales they usually use.
Regarding smile sheets:
Problem: While subjective input can be helpful, it can be inaccurate. We should, according to Thalheimer, be skeptical of the findings. (I guess I was right to be skeptical!) What one person feels they “agree” with, another will mark as “strongly agree,” even though they both feel the same way.
Solution: Instead of using subjective answers, give the student objective options, or at least more descriptive answers, as in Table 7.3.
Problem: Learners are more biased toward training that is still fresh in their minds. They will also better recall the learning because they are still in the same context they learned it in.
Solution: Survey students more than once. Ask them immediately so as to capture information that you might otherwise miss, and ask them questions after a delay so as to eliminate the bias created by the recentness of the training.
Problem: If this is true in environments run by professional trainers, it is even more true in environments where subject matter experts are leading the training.
Solution: This is the problem that Kerekes and Mattox were dealing with, and one answer is to change the questions, as they have suggested.
Table 7.3 Evaluation question (question developed by Will Thalheimer, PhD, of Work Learning Research, Inc., based on his book Performance‐Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form. SmileSheets.com.).
Now that you’ve completed the course, how able are you to TROUBLESHOOT the MOST COMMON ISSUES related to the H‐Inx Distributed Antennae System (HIDAS)?
|
Regarding Likert‐like measurement scales:
Table 7.3 offers a sample question, reprinted here with permission from Dr Thalheimer.
To be clear, measuring reactions is not always bad. In fact, from a marketing perspective, it is a good thing to do. If you are training customers, for example, you want to make sure that they enjoyed the time that they committed to you and that they will return or send others back. Maybe the coffee you serve is that important to them. But measuring reactions captures the student’s immediate impression. You must do more than just measure immediate impressions. You must measure the change you’ve been able to make in their behavior and/or performance.
Ask questions that force the student to move beyond the reactionary to the use. Even if the questions themselves are not as objective as an exam question, the data is still helpful to predict the effectiveness of the training, if you ask the right questions.
Here are a few additional tips about survey questions. These come from my own experience and are subject to change, depending on your circumstances. For professional help creating questions that will drive performance change, you should engage a company like CEB or Work Learning Research that specializes in gathering the right data.
Surveys are not exams. They are not exams for the student and they are not exams for the instructor. Too many times students get the impression that their survey is the instructor’s assessment. If the instructor has done his/her job, he/she has begun to build a professional relationship with the students. As a result, the students want to help the instructor. Now you have skewed data.
Ask questions about the instructor’s facilitation skills under two conditions:
Surveys and course evaluations are here to stay and I have a renewed faith in their ability to help improve the effectiveness of product training when administered correctly. To be effective as an instructor, you must measure results. You must measure that your students can perform the objectives you set for the class, and you must measure that the class itself was effective in making that happen.
The reason you take surveys and give exams is to make sure that the training you are offering is effective. Good surveys can also ensure that you are always improving your training courses and listening to your students.
Review of Part Two: The Strategy of Hands‐On Learning