An evaluation framework

13.1 Introduction

13.2 DECIDE: A framework to guide evaluation

13.1 Introduction

Designing useful and attractive products requires skill and creativity. As products evolve from initial ideas through conceptual design and prototypes, iterative cycles of design and evaluation help to ensure that they meet users' needs. Deciding when and how to evaluate a product requires careful consideration and may be different for different kinds of products.

The case studies in the previous chapter illustrate some of the approaches used.

The design process starts with the designers working to develop a product that meets users' requirements, but, as you have seen, understanding requirements tends to happen by a process of negotiation between designers and users. As designers understand users' needs better, their designs reflect this understanding. Similarly, as users see and experience design ideas, they are able to give better feedback that enables the designers to improve their designs further. The process is cyclical, with evaluation facilitating understanding between designers and users.

Evaluation is driven by questions about how well the design or particular aspects of it satisfy users' needs and offer appropriate user experiences. Some of these questions provide high-level goals to guide the evaluation. For example, does this product excite users so that they will buy and use it? Others are much more specific. Can users find a particular menu item? Do they interpret a particular graphic as the designers intended and do they find it attractive? Practical constraints play a big role in shaping how evaluation is done: tight schedules, low budgets, or little access to users constrain what evaluators can do. There are ethical considerations too: medical records must be private and so are certain areas of people's homes.

Experienced designers get to know what works and what doesn't. As you have seen in Chapter 12, there is a broad repertoire of evaluation methods that can be tailored for specific circumstances. Knowing and having the confidence to adapt methods is essential. The wide variety of mobile and ubiquitous systems coming onto the market challenges conventional evaluation practices, which must be adapted to provide useful feedback. Therefore, when planning evaluations, evaluators must consider the nature of each product, the kinds of users that will use it, and the contexts of use, as well as logistical issues, such as the budget, the schedule, the skills and equipment required for the evaluation. Planning evaluation studies involves asking questions about the process and anticipating potential problems. Within interaction design there are many books and websites that list different techniques and guidelines for conducting an evaluation, but there is very little overarching guidance for how to plan an evaluation. To help you, we propose the DECIDE framework, which provides a structure for planning evaluation studies.

The main aims of this chapter are to:

• Discuss the conceptual, practical, and ethical issues involved in evaluation.
• Introduce and explain the DECIDE framework.

13.2 DECIDE: A Framework to Guide Evaluation

Well-planned evaluations are driven by goals which aim to seek answers to clear questions, which may be stated explicitly, upfront, as in usability testing, or may emerge as the evaluation progresses, as in ethnographic evaluation. The way questions are stated also varies depending on the stage in the design when the evaluation occurs. Questions help to determine the kind of evaluation approach that is adopted and the methods used. Practical issues, such as the amount of time available to carry out the evaluation, the availability of participants, and suitable equipment, also impact these decisions. Ethical issues must also be considered, particularly when working with users, and evaluators must have enough time and expertise to evaluate, analyze, interpret, and present the data that they collect. The DECIDE framework provides the following checklist to help you plan your evaluation studies and to remind you about the issues you need to think about. It has the following six items:

1. Determine the goals.
2. Explore the questions.
3. Choose the evaluation approach and methods.
4. Identify the practical issues.
5. Decide how to deal with the ethical issues.
6. Evaluate, analyze, interpret, and present the data.

A list has the tendency to suggest an order in which things should be done. However, when working with the DECIDE framework, it is common to think about and deal with items iteratively, moving backwards and forwards between them. Each item in the framework is related to the other items, so it would be unusual to work in any other way.

13.2.1 Determine the Goals

What are the high-level goals of the evaluation? Who wants it and why? An evaluation (i) to help clarify that user needs have been met in an early design sketch has different goals from an evaluation (ii) to select the best representation of a metaphor for a conceptual design, or an evaluation to finetune an interface, or (iii) to examine how mobile technology changes working practices, or (iv) to inform how the next version of a product should be changed, or (v) to explore the impact of ambient technology in a social space; or (vi) to investigate what makes collaborative computer games engaging.

Goals guide the evaluation by helping to determine its scope, so identifying what these goals are is the first step in planning an evaluation. For example, we can restate the first general goal statement mentioned above more clearly as:

13.2.2 Explore the Questions

In order to make goals operational, we must clearly articulate the questions to be answered by the evaluation study. For example, the goal of finding out why some customers prefer to purchase paper airline tickets over the counter rather than e-tickets can be broken down into a number of relevant questions for investigation. What are customers' attitudes to these e-tickets? Perhaps they don't trust the system and are not sure that they will actually get on the flight without a ticket in their hand. Do customers have adequate access to computers to make bookings? Are they concerned about security? Does the electronic system have a bad reputation? Is the user interface to the ticketing system so poor that they can't use it? Maybe some people can't managed to complete the transaction. Maybe some people like the social interaction with a ticketing agent?

Questions can be broken down into very specific subquestions to make the evaluation even more finegrained. For example, what does it mean to ask, “Is the user interface poor?” Is the system difficult to navigate? Is the terminology confusing because it is inconsistent? Is the response time too slow? Is the feedback confusing or maybe insufficient? Subquestions can, in turn, be further decomposed if even more specific issues need to be addressed.

13.2.3 Choose the Approach and Methods

Having identified the goals and some questions that you want to investigate, the next step is to choose the evaluation approach and methods that you will use. As mentioned in Chapter 12, the evaluation approach influences the kinds of methods that are used. For example, when performing an analytical evaluation, methods that directly involve users will not be used. During usability testing, ethnography will not be used. Often different approaches are used in this way, depending on the questions to be answered and the resources available for performing the study. Practical and ethical issues (discussed next) have to be considered and trade-offs made. For example, the methods that seem most appropriate may be too expensive, or may take too long, or may require equipment or expertise that is not available, so compromises are needed.

As you saw in several of the case studies discussed in Chapter 12, combinations of approaches and methods are often used to obtain different perspectives. For example, the methods used in field studies tend to involve observation, interviews, or informal discussions with participants. Questionnaires may also be used and so might diary studies in which participants record aspects of their technology usage. Usability testing also involves multiple methods and, as we have already said, is often supplemented with field studies. Each type of data tells the story from a different point of view. Together these perspectives give a broad picture of how well the design meets the usability and user experience goals that were identified during requirements gathering.

13.2.4 Identify the Practical Issues

There are many practical issues to consider when doing any kind of evaluation, and it is important to identify as many of them as possible before starting the study. However, even experienced evaluators encounter surprise events, which is why it is useful to do a pilot study (discussed in Chapter 7). Some issues that should be considered include access to appropriate users, facilities and equipment, whether schedules and budgets are realistic, and whether the evaluators have the appropriate expertise to conduct the study. Depending on the availability of resources, there may have to be compromises that involve adapting or substituting methods. For example, evaluators may wish to perform usability tests using 20 users and then to run a three-week-long field study, but the budget available for the study may only cover the cost of five testers and a shorter field study. Another example is provided by the Nokia cell phone, which involved evaluating cell phones in a country where the evaluators do not speak the language fluently and are only slightly aware of cultural norms. In this situation the evaluators had to work out how to collect the data that they needed to answer their evaluation questions. Furthermore, cell phone users are highly mobile so the evaluators knew that there would be places where the cell phones would be used that they could not go, e.g. in bathrooms, bedrooms. During the study the evaluators may also have experienced surprise events that required them to take decisions there and then. For example, it may not have been possible to ride in the taxi or car with the user because there was not enough room. Of course, no evaluation is going to be perfect, and a good field study can be done without the evaluator seeing how the product is used 100% of the time, but it is helpful to be aware of the kind of compromises that may be necessary. Thinking about the kind of users who will be involved and logistical issues, such as availability of equipment, the schedule and the budget, and the kind of expertise needed to perform the study, when planning a study will help to ensure its success.

Users

It goes without saying that a key aspect of an evaluation is involving appropriate users or, in the case of analytical evaluation, focusing on the characteristics of the anticipated user population. When doing usability testing, for example, users must be found who represent the user population for which the product is targeted. This generally involves identifying users with a particular level of experience, e.g. novices or experts, or users with a range of expertise. The number of males and females within a particular age range, cultural diversity, educational experience, and personality differences may also need to be taken into account, depending on the kind of product being evaluated. Questionnaire surveys require large numbers of participants, so ways of identifying and reaching a representative sample of participants are needed. For field studies to be successful, the evaluator needs access to users who will interact with the technology in their natural setting.

Another issue to consider is how the users will be involved. The tasks used in a usability laboratory study should be representative of those for which the product is designed. However, there are no written rules about the length of time that a user should be expected to spend on an evaluation task. Ten minutes is too short for most tasks and two hours is a long time, so what is reasonable? Task times will vary according to the type of evaluation, but when tasks go on for more than 20 minutes, consider offering breaks. It is accepted that people using desktop computers should stop, move around, and change their position regularly after every 20 minutes spent at the keyboard to avoid repetitive strain injury. Evaluators also need to put users at ease so they are not anxious and will perform normally; it is important to treat them courteously. Participants should not be made to feel uncomfortable when they make mistakes. Greeting users, explaining that it is the product that is being tested and not them, and planning an activity to familiarize them with it before starting the task all help to put users at ease in test situations.

In field studies the onus is on the evaluators to fit in with the users and to cause as little disturbance to participants and their activities as possible. This requires practice, and even anthropologists who are trained in ethnographic methods may cause unforeseen changes (see Dilemma box below).

Facilities and Equipment

There are many practical issues concerned with using equipment in an evaluation. For example, when using video you need to think about how you will do the recording: how many cameras and where do you put them? Some people are disturbed by having a camera pointed at them and will not perform normally, so how can you avoid making them feel uncomfortable? How will you record data about use of a mobile device when the users move rapidly from one environment to another? Several of the case studies in Chapter 12 addressed these issues. Think back, or reread the Nokia cell phone study, the Indian auxiliary midwife data collection study, and HutchWorld.

Schedule and Budget Constraints

Time and budget constraints are important considerations to keep in mind. It might seem ideal to have 20 users test your interface, but if you need to pay them, then it could get costly. Planning evaluations that can be completed on schedule is also important, particularly in commercial settings. However, as you have seen in the interview with Sara Bly, there is rarely enough time to do the kind of studies that you would ideally like, so you have to compromise and plan to do the best job possible with the resources and time available.

Expertise

Different evaluation methods require different expertise. For example, running user tests requires knowledge of experimental design and video recording. Does the evaluation team have the expertise needed to do the evaluation? If you need to analyze your results using statistical measures and you are unsure of how to use them, then consult a statistician before starting the evaluation and then again during data collection and analysis, if needed.

13.2.5 Decide How to Deal with the Ethical Issues

The Association for Computing Machinery (ACM) and many other professional organizations provide ethical codes (Box 13.1) that they expect their members to uphold, particularly if their activities involve other human beings (ACM, 1992). All data gathering requires you to consider ethical issues (see Chapter 7), but this is particularly important for evaluation because the participants are often put into unfamiliar situations. People's privacy should be protected, which means that their name should not be associated with data collected about them or disclosed in written reports (unless they give explicit permission). Personal records containing details about health, employment, education, financial status, and where participants live should be confidential. Similarly, it should not be possible to identify individuals from comments written in reports. For example, if a focus group involves nine men and one woman, the pronoun ‘she’ should not be used in the report because it will be obvious to whom it refers.

Most professional societies, universities, government, and other research offices require researchers to provide information about activities in which human participants will be involved. This documentation is reviewed by a panel and the researchers are notified whether their plan of work, particularly the details about how human participants and data collected about them will be treated, is acceptable. Drawing up such an agreement is mandatory in most universities and major organizations. Indeed, special review boards generally prescribe the format required and many provide a detailed form which must be completed. Once the details are accepted the review board checks periodically in order to oversee compliance. In American universities they are known as Institutional Review Boards (IRB). Other countries use different names for similar processes. Over the years IRB forms have become increasingly detailed, particularly now that much research involves the Internet and people's interaction via communication technologies across the Internet. Several law suits at prominent universities have heightened attention to IRB compliance to the extent that it sometimes takes several months and multiple amendments to get IRB acceptance. IRB reviewers are not only interested in the more obvious issues of how participants will be treated and what they will be asked to do, they also want to know how the data will be analyzed and stored. For example, data about subjects must be coded and stored to prevent linking participants' names with that data. This means that names must be replaced by a code and that the code and the data must be stored separately, usually under lock and key. Figure 13.1 contains part of a completed IRB form to evaluate a Virtual Business Information Center (VBIC) at the University of Maryland.

People give their time and their trust when they agree to participate in an evaluation study and both should be respected. But what does it mean to be respectful to participants? What should participants be told about the evaluation? What are participants' rights? Many institutions and project managers require participants to read and sign an informed consent form similar to the one in Box 13.2. This form explains the aim of the study and promises participants that their personal details and performance will not be made public and will be used only for the purpose stated. It is an agreement between the evaluator and the participants that helps to confirm the professional relationship that exists between them.

Figure 13.1 Excerpt from an IRB to evaluate a Virtual Business Information Center (VBIC) at the University of Maryland

The following summary provides guidelines that will help ensure evaluations are done ethically and that adequate steps to protect users' rights have been taken.

• Tell participants the goals of the study and exactly what they should expect if they participate. The information given to them should include outlining the process, the approximate amount of time the study will take, the kind of data that will be collected, and how that data will be analyzed. The form of the final report should be described and, if possible, a copy offered to them. Any payment offered should also be clearly stated.
• Be sure to explain that demographic, financial, health, or other sensitive information that users disclose or is discovered from the tests is confidential. A coding system should be used to record each user and, if a user must be identified for a follow-up interview, the code and the person's demographic details should be stored separately from the data. Anonymity should also be promised if audio and video are used.
• Make sure participants know that they are free to stop the evaluation at any time if they feel uncomfortable with the procedure.
• Consider your relationship with the participants and decide whether it is appropriate to provide incentives such as food, book tokens, or financial payment. For example, if it is your child taking part in a colleague's study, a gift token or a toy would be more appropriate than offering payment as an incentive.
• Avoid including quotes or descriptions that inadvertently reveal a person's identity by using numbers or fictitious names to record and identify individuals. Where quotes are reported to illustrate findings then it is convention to replace words that would reveal the source with representative words, in square brackets. For example, if the study was evaluating a university's information system and one of the participants commented: “When I tried to send a message to Harry Jones about my meeting with Mary Ann Green the whole system suddenly froze,” then the comment would be quoted as: “When I tried to send a message to […] about my meeting with […] the whole system suddenly froze.”
• Ask users' permission in advance to quote them, promise them anonymity, and offer to show them a copy of the report before it is distributed.

The explosion in Internet and web usage has resulted in more research on how people use these technologies and their effects on everyday life (Jones, 1999). Consequently, there are many projects in which developers and researchers are logging users' interactions, analyzing blogs, recording web traffic, or examining conversations in chatrooms, bulletin boards, or on email. These studies can be done without users knowing that they are being studied. This raises ethical concerns, chief among which are issues of privacy, confidentiality, informed consent, and appropriation of others' personal stories (Sharf, 1999). People often say things online that they would not say face to face. Furthermore, many people are unaware that the personal information they share online can be read by someone with technical know-how years later, even after they have deleted it from their personal mailbox (Erickson et al., 1999).

13.2.6 Evaluate, Interpret, and Present the Data

Decisions are must be made about what data is needed to answer the study questions, how the data will be analyzed, and how the findings will be presented (see Chapter 8). To a great extent the method used determines the type of data collected, but there are still some choices. For example, should the data be treated statistically? Some general questions also need to be asked. Is the method reliable? Will the method measure what is intended, i.e. what is its validity? Are biases creeping in that will distort the results? Will the results be generalizable, i.e. what is their scope? Will the evaluation study be ecologically valid or is the fundamental nature of the process being changed by studying it?

Reliability

The reliability or consistency of a method is how well it produces the same results on separate occasions under the same circumstances. Another evaluator or researcher who follows exactly the same procedure should get similar results. Different evaluation methods have different degrees of reliability. For example, a carefully controlled experiment will have high reliability, whereas observing users in their natural setting will be variable. An unstructured interview will have low reliability: it would be difficult if not impossible to repeat exactly the same discussion.

Validity

Validity is concerned with whether the evaluation method measures what it is intended to measure. This encompasses both the method itself and the way it is performed. If, for example, the goal of an evaluation study is to find out how users use a new product in their homes, then it is not appropriate to plan a laboratory experiment. An ethnographic study in users' homes would be more appropriate. If the goal is to find average performance times for completing a task, then a method that only recorded the number of user errors would be invalid.

Biases

Bias occurs when the results are distorted. For example, expert evaluators performing a heuristic evaluation may be more sensitive to certain kinds of design flaws than others, and this will be reflected in the results. Evaluators collecting observational data may consistently fail to notice certain types of behavior because they do not deem them important. Put another way, they may selectively gather data that they think is important. Interviewers may unconsciously influence responses from interviewees by their tone of voice, their facial expressions, or the way questions are phrased, so it is important to be sensitive to the possibility of biases.

Scope

The scope of an evaluation study refers to how much its findings can be generalized. For example, some modeling methods, like the keystroke model, have a narrow, precise scope. The model predicts expert, error-free behavior so, for example, the results cannot be used to describe novices learning to use the system. The problems of overstating the results were discussed in more detail in Chapter 8.

Ecological Validity

Ecological validity concerns how the environment in which an evaluation is conducted influences or even distorts the results. For example, laboratory experiments are controlled and are quite different from workplace, home, or leisure environments. Laboratory experiments therefore have low ecological validity because the results are unlikely to represent what happens in the real world. In contrast, ethnographic studies do not impact the environment as much, so they have high ecological validity.

Ecological validity is also affected when participants are aware of being studied. This is sometimes called the Hawthorne effect after a series of experiments at the Western Electric Company's Hawthorne factory in the USA in the 1920s and 1930s. The studies investigated changes in length of working day, heating, lighting, etc., but eventually it was discovered that the workers were reacting positively to being given special treatment rather than just to the experimental conditions. Similar findings sometimes occur in medical trials. Patients given the placebo dose (a false dose in which no drug is administered) show improvement that is due to receiving extra attention that makes them feel good.