Without proper information it is difficult, if not impossible, to initiate a proper benchmarking effort. Information gathered in this process—called data collection by planners and requirements elicitation by software developers—will enable the organization to develop valid measures against which it should be measured.
The most common method of gathering information is by interviewing people. Interviewing can serve two purposes at the same time. The first is a fact-finding mission to discover what each person’s goals and objectives are with respect to the project; and the second is to begin a communications process that enables one to set realistic expectations for the project.
There are a wide variety of stakeholders that can and should be interviewed. Stakeholders are those that have an interest in seeing this project successfully completed—that is, they have a stake in the project. Stakeholders include employees, management, clients, and benchmarking partners.
Interviews have some major obstacles to overcome. The interviewees may resist giving information out of fear, they may relate their perception of how things should be done rather than how they really do them, or they may have difficulty in expressing themselves. On the other hand, the analyst’s own mind-set may also act as a filter too. Interviewers sometimes have to set aside their own technical orientation and make the best effort that they can to put themselves in the position that the interviewee is in. This requires that the analyst develops a certain amount of empathy.
An interview outline should contain the following information:
Recommended guidelines for handling the employee interview process include:
Customers often have experiences with other vendors or suppliers and can offer insight into the processes that other companies use or that they have experienced.
Guidelines for interviewing customers include
Another source of potentially valuable information is from other companies in the industry and consultants who specialize in the process areas being examined. While consultants can be easily located and paid for their expert advice, it is wise to tread slowly when working with other companies that are current or potential competitors.
Guidelines for interviewing other companies include
Suppliers of the products you are considering are also an important source of ideas. These suppliers know a great deal about how their products are being used in the processes you are examining.
When interviewing anyone, it is important to be aware of how to ask questions properly. Open-ended questions are the best for gaining the most information because they do not limit the individuals to predefined answers. Other benefits of using open-ended questions include: puts the interviewee at ease, provides more detail, induces spontaneity, and it is far more interesting for the interviewee. Open-ended questions require more than a yes or no answer. An example of an open-ended question is “What types of problems do you see on a daily basis with the current process?” These questions allow individuals to elaborate on the topics and potentially uncover the hidden problems at hand that might not be discoverable with a question that requires a yes or no answer.
One disadvantage of open-ended questions is that they create lengthier interviews. Another disadvantage is that it is easy for the interview to get off track, and it takes an interviewer with skill to maintain the interview in an efficient manner.
Closed-ended questions are, by far, the most common questions in interviewing. They are questions that have yes and no answers and are utilized to elicit definitive responses.
Past-performance questions can be useful to determine past experiences with similar problems and issues. An example of how a past-performance question is used is, “In your past job how did you deal with these processes?”
Reflexive questions are appropriate for closing a conversation or moving it forward to a new topic. Reflexive questions are created with a statement of confirmation and adding a phrase such as: Don’t you? Couldn’t you? Or wouldn’t you?
Mirror questions are a subtle form of probing and are useful in obtaining additional detail on a subject. After the interviewee makes a statement, pause and repeat his or her statement back with an additional or leading question: “So, when this problem occurs, you simply move on to more pressing issues?”
Often, answers do not give the interviewer enough detail, so one follows the question with additional questions to prod the interviewee to divulge more details on the subject. For example:
Another, more subtle, prodding technique can be used by merely sitting back and saying nothing. The silence will feel uncomfortable, causing the interviewee to expand on his or her last statement.
If there are large numbers of people to interview, one might start with a questionnaire and then follow up with certain individuals that present unusual ideas or issues in the questionnaires. Survey development and implementation are composed of the following tasks, according to Creative Research Systems, makers of a software solution for survey creation (surveysolutions.com):
Similar to interviews, questionnaires may contain closed-end or open-ended questions or a hybrid, which is a combination of the two.
Survey creation is quite an art form. Guidelines for the creation of a survey include
Not useful at all | Very useful | |||
1 | 2 | 3 | 4 | 5 |
Survey questions must be carefully worded. Ask yourself the following questions when reviewing each question:
Tallying the responses will provide a “score” that assists in making a decision that requires the use of quantifiable information. When using interval scales, keep in mind that not all questions will carry the same weight. Hence, it is a good idea to use a weighted average formula during calculation. To do this, assign a “weight” or level of importance to each question. For example, the aforementioned question might be assigned a weight of 5 on a scale of 1 to 5 meaning that this is a very important question. On the other hand, a question such as “Was the training center comfortable” might carry a weight of only 3. The weighted average is calculated by multiplying the weight by the score (w * s) to get the final score. Thus, the formula is snew = w * s.
There are several problems that might result in a poorly constructed questionnaire. Leniency is caused by respondents who grade nonsubjectively—in other words, too easily. Central tendency occurs when respondents rate everything as average. The halo effect occurs when the respondent carries his or her good or bad impression from one question to the next.
There are several methods that can be used to successfully deploy a survey. The easiest and most accurate is to gather all respondents in a conference room and hand out the survey. For the most part, this is not realistic, so other approaches would be more appropriate. E-mail and traditional mail are two methodologies that work well, although you often have to supply an incentive (i.e., prize) to get respondents to fill out those surveys on a timely basis. Web-based surveys (Internet and intranet) are becoming increasingly popular as they enable the inclusion of demos, audio, and video. For example, a web-based survey on what type of user interface is preferable could have hyperlinks to demos or screen shots of the choices.
Observation is an important tool that can provide a wealth of information. There are several forms of observation: silent and directed. In silent observation, the analyst merely sits on the sidelines, pen and pad, and observes what is happening. If it is suitable, a tape recorder or video recorder can record what is being observed. However, this is not recommended if the net result will be several hours of random footage.
Silent observation is best used to capture the spontaneous nature of a particular process or procedure. For example,
Directed observation provides the analyst with a chance to micro-control a process or procedure so that it is broken down into its observable parts. At one accounting firm, a tax system was being developed. The analysts requested that several senior tax accountants be coupled with a junior staff member. The group was given a problem as well as all the manuals and materials they needed. The junior accountant sat at one end of the table with the pile of manuals and forms while the senior tax accountants sat at the other end. A tough tax problem was posed. The senior tax accountants were directed to think through the process and then direct the junior member to follow through on their directions to solve this problem. The catch was that the senior members could not walk over to the junior person nor touch any of the reference guides. This whole exercise had to be verbal and use just their memories and expertise. The entire process was videotaped. The net result was that the analyst had a complete record of how to perform one of the critical functions of the new system.
The flip side of observation is participation. Actually becoming a member of the staff, and thereby learning exactly what it is that the staff does so that it might be automated, is an invaluable experience.
It is logical to assume that there will be a wide variety of documentation available to the analyst. This includes, but is not limited to the following:
In a brainstorming session, you gather together a group of people, create a stimulating and focused atmosphere, and let people come up with ideas without risk of being ridiculed. Even seemingly stupid ideas may turn out to be “golden.”
Focus groups are derived from marketing. These are structured sessions where a group of stakeholders are presented with a solution to a problem and then are closely questioned on their views about that solution.