CHAPTER 4
Ordering and Laying Out the Survey
Once you have finished writing the questions, you must think about the design of the survey. Design elements to consider are:
Appearance
The length of the survey
The order of the questions
The combination of questions.
Appearance
When a participant first starts the survey, motivate him or her by “creating interest and confidence that they can answer all the questions” (Abbey-Livingston 1982, 44). Questions at the beginning of the survey should be:
Related directly to the objectives stated in the cover letter
Relatively easy to answer
Neutral; don’t start with questions that ask for the participant’s opinion.
Consider starting with questions that ask for demographic details, which will allow you to get to know the participant, but avoid education or income questions at this point. Consider qualifying the participant at the start of the survey by using a filtering question. A filtering question in a web-based survey is used to determine if the participant is qualified to answer one or more subsequent questions. If the “wrong” answer is selected, the participant receives a message expressing appreciation for his or her interest; at that point, the survey ends.
After you have identified the initial questions, arrange the rest of the questions in a logical order. (The section “The Order of the Questions” later in this chapter provides more details.)__At the end of the survey, leave the participant with a positive feeling. Ask the participant if he or she has anything further to say with a question such as “Is there anything else you would care to add that has not been adequately addressed in this survey?” And be sure to thank the participant for taking the survey.
It is important to use transitions effectively throughout the survey. Transitions indicate that the survey is changing direction. They are frequently used as a bridge to the first set of questions, between sections, at the start of new pages, or to break up a long series of questions. For a minor change in the type of questioning, a short transition is appropriate; if there is a major change in questioning, a longer transition is suggested. Be sure not to use too many transitions—they might cause the participant to lose interest. Sample transitions you might try include:
“Now, we will change direction slightly and ask about the …”
“We would like to get your input about related topics. One topic we do not know enough about is …”
“There are only a few questions left. We want to ask about …”
“To help summarize the responses and interpret the results, we would like to learn more about you.”
Here you would include demographic questions about the participant’s team, years of experience in his or her role, what type of system user he or she is, geographic location, level in the organization, and level of technical knowledge. Here’s a sample transition addressing two of these data points: “First, I need to ask a few general questions to determine how many systems you are knowledgeable about and how long you have been on this team.”
Depending on how the survey will be administered—by email, on paper, or on the web—consider these format-related pointers:
Use plenty of white space so the survey does not look crowded.
Assign a unique identifier to each question and option.
Do not split questions between pages.
Create an appealing, easy-to-follow layout.
Ensure spelling and grammar are correct.
For open-ended questions, be sure to leave enough space for a reply. If you provide only one line, participants will assume you want a very brief answer.
If certain questions are to be answered by some participants and not others, give clear instructions.
The Length of the Survey
Time is critical in our busy world. The length of the survey can affect the response rate, which is the percentage of participants invited to participate in the survey that actually complete the survey. If the survey is too long, participants may get tired and stop. This is called abandoning the survey. If the survey has a lot of open-ended questions, it will take longer to complete. Suggested survey lengths vary. If the survey is too short—say, three questions—participants may wonder whether the survey has a real purpose. That said, a short survey should generally contain less than ten questions and take no more than five to ten minutes to complete. A longer survey should not exceed five pages.
The Order of the Questions
The order of the questions in a survey is important. Here are some things to consider:
Start with easy questions first.
Limit the number of questions.
Try to organize the questions in a way that would mirror a conversation with the participant.
The unwritten rules of conversation also apply to survey questions. In a conversation, each party gets to know the other before asking personal questions. Personal or sensitive questions, such as those about finances or personal habits or problems, should appear later in the survey.
Group similar questions or questions about the same topic together. This makes it easier for the participants; they won’t have to jump from topic to topic.
Flow questions in a logical order. Questions may go from general to specific, from easy to difficult to answer, or from most important to least important.
The questions surrounding a question, especially those immediately preceding it, can affect how that question is answered.
Combining Questions
Survey questions can be effectively paired or combined. Consider the following techniques:
Use a combination of open-ended and closed-ended questions to identify important issues and compare responses.
For example, ask a question like “What issues do you think the organization is facing?” Later, ask “In the following list of issues, which two do you think are the most important in our organization?”
If this technique is used, separate the two questions with other questions.
Ask the open-ended question first so that participants are not influenced by the options provided in the closed-ended question.
Analysis of the data can identify what issues participants talk about when they are not prompted, which are top priorities when they are forced to choose, and which appear in both situations.
Use a funneling technique in the survey. Funneling is beginning a section of the survey with an open-ended question, then moving into closed-ended questions about very specific topics.
Several funnels can be used in a single survey.
The following series of questions illustrates the funneling technique.
“How do you like the new system manual?”
“In the past three months, how often have you used this manual to complete your work?”
a)__Less than ten times
b)__11 to 20 times
c)__21 to 30 times
d)__More than 31 times.
Please respond to the following statements, where 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, and 5=strongly agree.
Now we would like to explore the importance of specific sections of the manual. Please rate the importance of the following sections, where 1=very important, 2=important, 3=average importance, 4=slightly important, and 5=not important.
When the layout of the survey is complete, look at the order of the questions to ensure that questions flow as intended. The flow of questions can also be verified during the pilot. The pilot process is discussed in the next chapter.