11

INTERVIEWING MEMBERS OF
ONLINE COMMUNITIES

A Practical Guide to Recruiting Participants

Amy Bruckman

GEORGIA INSTITUTE OF TECHNOLOGY

In an interview study of members of an online community, one of the most daunting tasks is finding people who are willing to talk to you – and doing so ethically. Since 1997, I have taught a Georgia Tech class called “Design of Online Communities,” in which students are assigned to interview at least three members of an online group and also engage in participant observation. The class began at the graduate level, and now is offered to undergraduates as well. Teams of three students study the same site and write up a detailed account of its design. The goal of the assignment is, first, to help students understand how designed features of an online site impact user behavior. Second, the project introduces them to interviewing and participant observation as research methods for understanding online behavior. As you might imagine, interesting things happen when I unleash 40 new student researchers each year.

Each semester, I repeatedly warn students that getting people to talk to you is hard work, and they should expect to spend a lot of time doing it. To further complicate matters, while you must be persistent, you may not annoy community members. And all activity takes place as governed by approval from our Institutional Review Board (IRB), the committee charged with protecting human subjects in research. In this chapter, I shall share the lessons learned from shepherding students through this process for the last 12 years. Those lessons are relevant not just for student projects but also for people working on research about this medium for publication.

In this chapter, I will focus primarily on the issue of how to get people to agree to talk with you. I will discuss issues of ethical conduct of online research, but not cover the subject in detail. For a more complete account from the ethics point of view, see (Bruckman, 2002, 2006).

Students in the class are required to read the book Interviewing as Qualitative Research by Irving Seidman (Seidman, 2006). Much of the advice relayed here builds on advice from Seidman's book, focusing on my students’ experiences applying Seidman to studying online groups in particular.

Medium for the Interview

Students are instructed to conduct interviews on the telephone/skype or in person – not in text-based interaction. Students and potential interview subjects are often puzzled by this requirement. If we're talking about people's online experiences, why not talk with them in the medium itself ? It's easier for both the researcher and subject, and seems appropriate given the topic of investigation. However, an online interview yields so few data that it is almost useless in comparison to a phone or in-person interview. When conducting an online interview, one develops a sense that you've had a profound conversation. But then looking back at the transcript later, you find there's little in it. The easiest way to demonstrate this is simply to compare a transcript of a text-based interview and a telephone one. The online interview transcript will typically fill a few pages, and the voice interview five to ten times as many. Each term I spread the pages out on the floor at the front of the classroom to make the point visually. When students see that the online/text transcript crosses the desk, and the phone/skype transcript goes across the room and back, they understand why the text-based interview is insufficient.

I will let students do text-based interviews as a backup for subjects who refuse to do a voice interview, though it doesn't “count” towards their interview assignment. As a result, for many years our informed consent form said “you are agreeing to do an interview in person, but telephone, or online.” Many subjects agreed to a phone interview, saw the wording of the consent form, and then requested to do an online interview instead. As a result, we now have two separate, approved consent forms: one that says “in person or by telephone/skype,” and another that says “online.” We distribute the forms for audio-based interviews, and only send out the text-based forms when absolutely necessary.

Students in the online communities class are required to interview three subjects simply because time is limited. For publishable research, more are needed. How many interviews are needed is a subject of much discussion in qualitative methods literature. A good rule of thumb is that you are done when you “start to hear the same thing” over and over. For many studies, you may interview different kinds of participants (e.g., new users and “regulars.”) For each category of user, you generally need 10–20 interview subjects. This is a very rough estimate, and the number, of course, differs depending on the goals of the particular study.

Although we interview people in person or on the phone, initial contact is made online. But before we do that, we first need to choose a site to study.

Challenges of recruiting subjects

The choice of site to study, of course, depends primarily on your research questions. Beyond that, researchers should also be aware that recruiting subjects is harder on some kinds of sites than others. The task is challenging for any site, but impossible if the site is poorly chosen.

To gain some insight into whether a particular site is a reasonable choice, try to imagine yourself as a participant on that site. Imagine someone asks if you'd talk to them about it. Under what circumstances will you say yes? List in your mind all the sites you personally use, and think about which ones you'd be willing or eager to discuss and which ones not. Most people will find they can immediately think of a special site they'd be willing to chat about (i.e..,”I love that site – I spend way too much time there!”) and many others they would not (i.e.., “I posted a question there once, but I can't really comment on it in detail.”)

For most people, it's a pleasure to talk about a site they're excited about. If a user is invested in a site, then telling someone about it is fun. It helps if being a member of the site makes them feel good about themselves – in other words, if being a member conveys a positive sense of social status. For example, if you are proud of the electronic music you make and share with friends, you'll be eager to talk to a researcher about the electronic music site where you post your work and get feedback from friends.

It is generally easier in to recruit subjects on a successful site (one that is growing, with users who appreciate it) than on a less successful one (one with a shrinking population, or a small site that appears to not be “taking off”). A site may also be problematic if it seems somewhat impersonal, for example, one where people participate but don't tend to form friendships.

Beyond the difficulty of getting people to talk to you, it is also more challenging to learn anything from a mediocre site. If it's not working well, it can be difficult to figure out why. There may be a dozen reasons why a site is less successful than others, or less successful than it used to be – and it's hard to figure out what the real reasons are. Furthermore, even if you can identify some of the reasons, they may be uninteresting (i.e.., a site may be less successful because it has ugly graphics, slow performance, and poor user interface design). None of those observations contributes meaningfully to our understanding of online activity. On the other hand, there is usually something to learn from a site that is thriving – something is working, and that something is usually valuable to explore.

Terms of Service

Online sites have terms of service (TOS), and for many sites this may include explicit rules on what kind of studies are allowed there. Most ethics experts believe that researchers must comply with TOS, but the issue is complicated. In many cases, TOS are selectively enforced. If the rules are not followed by most site members, do researchers need to be held to them? TOS constitute a contract, and the legal standing of some provisions of those contracts can be drawn into question. Just because a company puts a clause in their policy does not mean the provision is legal and will hold up in court. The question of what research is ethical is arguably separate from the legal one concerning whether the terms of a TOS are legal and binding. On the other hand, some will argue that it is inherently unethical to break a contract, and hence the legal and ethical issues are the same.

If TOS of a particular site prohibit research, it is often possible to get permission to do the research with permission of the company. What the general public may not do, internal staff, summer interns, and collaborators with the company may do with permission. In these situations, some members of the research community argue that the independence of research is jeopardized. If activity on a particular online site becomes an important cultural phenomenon, it is ethically problematic if research on that site can only be conducted with the company's blessing and according to their restrictions.

In most cases, abiding by TOS is possible and is the wisest course of action. Studies that deviate from TOS can potentially be rejected from conferences and journals on those grounds. Before you invest time in exploring a potential study site, it is important to review its TOS and understand its terms. Deviating from those terms is notsomething to be done lightly. Note also that the wording of many sites'TOS are ambiguous, and it may be unclear what is permitted. In such cases, your IRB may be able to help you interpret the wording and decide what is permitted.

Gatekeepers

One of the trickiest concepts to understand is that of a gatekeeper. A gatekeeper is someone you legitimately must go through before approaching members of a group. A parent is a legitimate gatekeeper for their child – you must ask the parents’ permission before approaching the child. A teacher is a gatekeeper for his or her class. For a small online group, particularly one with a strong and visible moderator, the moderator may serve as a legitimate gatekeeper for the group. Asking the gatekeeper's permission is often necessary. If the gatekeeper is enthusiastic about your study and willing to pass your request on to members, this may really get your study off to a strong start. On the other hand, it is not necessary or appropriate to ask permission of the site founder Jimmy Wales before starting to study Wikipedia. Many online sites do not really have gatekeepers. For a large site, you must comply with the TOS, but you do not need to obtain explicit permission from management. The size of the group and hands-on involvement of the group leader is a key to deciding whether a particular individual is a gatekeeper.

Sometimes a regular member of a group may see himself or herself as a gatekeeper when this is not really the case. In these situations, going through that person is not necessary, and may actually be damaging to your study. If the self-appointedgatekeeper is a busybody no one likes, then it will harm your study tohave that person introduce you to the group; see Seidman (2006) for a discussion of “illegitimate” gatekeepers.

Choosing a community where you fit

Choosing a good site to study is not only about the site, but also about your fit for the site. Before you can begin studying a site, you need to participate there. People respond better to someone who has been a part of the group (at least for a little while), and seems to understand the community. In 2008, a team of students in theonline communities class were having trouble getting interviews on bix.com, a site that presents itself as an amateur talent hour where members sing. However, once our student researchers started singing more themselves, interview subjects became much easier to recruit.

Students studying couchsurfing. com, a site where youstay on the couch at a stranger's house in another city, had no trouble at all finding interview subjects, because they went couch surfing. Their experiences staying on other people's couches and letting others stay with them built ties that made obtaining interviews trivial. The choice of the site made me nervous, because couch surfing seems risky to me. I personally would not do it, and was hesitant about the students trying. But they insisted that they had read enough to assure themselves that it was safe – couch surfers vouch for one another, so every host comes with references. Meeting people in person developed interpersonal ties, and they had one of the easiest jobs I have ever observed getting interviews. Everyone they stayed with was happy to be interviewed, and was eager to refer their couchsurfing friends.

It is easy to say, “if you're studying a talent site, start singing,” but sometimes the stated purpose of a site is not its real purpose. One year a team of students chose to study a travel site – and travelling more was not the secret to getting users to participate in their interview study. The research team consisted of two single women in their 20s and one married man in his 30s. As they started their participant observation, the women had no trouble at all finding people to interview, but the man could not get anyone to respond to him. His team mates noticed that while they were talking about travel, there was a lot of flirting going on. Our male team mate took his marital status out of his profile and changed his picture from one of him and his wife to one of just himself, and suddenly interviews were much easier to get. In this case, the site seemed like a travel site, but functioned more as a dating site, with “people who like to travel” as a criteria for selection.

Note that removing his marital information was something he did only after great thought and discussion with myself and our teaching assistant. He did not in any way lie – he just removed irrelevant information. Every piece of information in his self-presentation was accurate, and he clearly disclosed that he was a student studying the site for a class. He just did not need to have his arm around his wife in his photo.

If you wish to study a site that contains sensitive information, you need to be even more careful. Lots of fantastic research is being conducted today on sites like health support sites. However, those researchers are generally educated about the topic of health support, and are usually part of a research team that includes physicians and other medical professionals. It is one thing to march into bix.com and start singing and see if anyone will talk to you, but quite another to join a cancer site and start talking with patients and their caregivers. I generally ask my students not to study sites with vulnerable populations unless it is necessary. We make exceptions for students who themselves suffer from a particular condition or have an immediate family member who is a sufferer. Personal experience gives one a sensitivity to the norms of the community that is hard to attain otherwise. Health support studies can be studied by outsiders with training and support, but it is advisable if possible to stick with something less sensitive where there are lower consequences to disturbing site activity accidentally.

Self-presentation

To conduct an ethical study, it is critical that you clearly disclose that you are a researcher studying the site. In sites that have a user profile, this fact can be placed prominently in your profile with a link to a URL with information about the study as approved by your IRB. In sites with a user-definable visual representation, you might consider looking like a researcher. Students in my class studying Second Life made avatars wearing white lab coats (see Figure 11.1). The fact that you are a researcher may even be put into your online name.

It is imperative that researchers avoid any kind of identity deception – even if identity deception is common on the site. Early in the history of the online communities class before sites selected had to be explicitly approved by our IRB and the instructor, one female student chose to study a site that was for men only. To gain entry into the site, she created a male avatar. When her true identity was unmasked, she was kicked off the site, and had to start her project over on a new site. (The requirement for explicit approval of site choice was added for the next class.) Defending her choice of self-presentation, she argued that gender swapping on the site was common – lots of women wanted to check out the site, and made male personas. However, a researcher is held to a higher standard than other participants of the site. It is not ethical (and not in the best interests of a successful study) to use deception in any form. Our student who took his marital status out of his profile is a rare case that stayed just on the right side of the ethical boundary: he merely omitted irrelevant information, and this even was a close call.

It is possible to do deceptive research studies. IRBs generally require that the deception be well motivated, and subjects be debriefed about the real purpose of the study. This is something to do only when necessary and with the supervision of an IRB. For an example of a deceptive study with a waiver of consent and exemption from debriefing, see Hudson and Bruckman (2004).

FIGURE 11.1 Avatar of Georgia Tech Ph.D. student, Susan Wyche, used for an online communities class project about Second Life.

While we typically worry primarily about the researcher creating risks for community members, the opposite does rarely occur: sometimes the community can harass the researcher. In 2002 (a year after the 9/11 attacks), a student of Turkish descent chose to study an online site providing consumer advice. To everyone's surprise, the site turned out to be a politically charged “flame fest,” with radically politically conservative members making a sport out of attacking consumer advocates as “leftists.” Some of the more extreme members chose to attack our student with a Turkish name as being indirectly responsible for the 9/11 attacks. Their comments were racist and offensive. He used his real name on the site, and they found term papers he had written and posted online and used them as “evidence” of his status as a terrorist. While this kind of experience is rare (this is the only incident like it since 1997), it still is worth noting as a cautionary tale. It is important to get to know a site before committing to study it, and approach hate-filled sites with great caution. In special circumstances, extra precautions to protect the researcher may be advisable – with approval from your IRB. While we normally require students to use their real names and post links to Georgia Tech to show the legitimacy of their study, I would seriously consider having students studying contentious sites operate more anonymously in the future.

Our Turkish student responded politely but clearly to the ridiculous accusations against him. Having received a response from him, the racist site members provoked him more. The situation had already escalated when he first came to talk to the instructor and TA about what was going on. In general, the best advice for dealing with angry and contentious people online – sometimes referred to as “trolls” (Raymond, 1991) – is to simply not respond at all. It is hard to leave their accusations unanswered, but any reply will likely simply fan the flames. It is small consolation for the upsetting experience our Turkish student underwent, but it should be noted that all of this made for an extremely interesting (A +) study.

Dangers of being a “regular”

Every semester, at least one student asks to study a site where they have participated for an extended period of time – where they are a “regular” (Oldenburg, 1989). Knowing a site well certainly brings some benefits: the researcher has a depth of knowledge that can only be acquired over time. However, it is generally a bad idea for several reasons. First, it can be difficult to observe a site objectively when you are already familiar with it. You take things for granted, because they seem “normal.” Anthropologists travel to remote villages to study human behavior to achieve that defamiliarization. What they observe about human relations in New Guinea ultimately helps us understand human relations in western culture: you can see phenomena better when they are unusual. The Level 70 Mage may have a hard time really understanding World of Warcraft, because they have already spent too much time there. Furthermore, studying your “friends” creates ethical tensions. They can't really say no to an interview if you have a longstanding relationship with them, and if they don't like what you ultimately write about them, you may jeopardize your friendship (see Pratt & Kim, this volume).

Last but not least, a “native” often finds things interesting that an outsider will not. To someone who is a longstanding community member, the history of internal squabbles among members may seem of great importance. To a neutral observer and the broader cause of research, such observations are usually decidedly uninteresting. The worst papers I receive each term are often by people studying sites they already knew too well.

Persistence in finding research subjects

Once you have chosen a site to study, and clearly marked your researcher status in your online representation, how do you recruit people to be interviewed? First, it is important not to jump right in but to “lurk” quite a bit first. It is a good idea to participate significantly before you ask anyone to speak with you at all. There are multiple reasons for this. First, you will not appreciate the nature of the site until you have spent a nontrivial amount of time there. Our team studying the consumer advice site thought they were going to learn about peer production of content, but instead learned a whole lot about deviant behavior online. The site was something quite other than what they expected – and this necessarily shaped the questions they asked members and which members were approached for interviews. Second, people generally will not speak to you unless you have participated at least to some degree. Being an active participant creates a social context for approaching members of the group.

In sites that have synchronous communication, it is necessary to talk to someone for a while before asking if they would like to be interviewed. Then you may give them a URL with IRB-approved text describing the study. You will ideally talk with them on more than one occasion before making the request. This means you need to spend a lot of time talking with a lot of people. It is important to keep a record of who you have asked, and never ever ask the same person a second time.

In asynchronous communication, you again need to participate for a nontrivial amount before posting a message requesting study participation. It may be more effective to send individual requests than to make a public posting. Once you have made a public posting, you probably can not make a second posting to that same forum again. It is often strategic to have a private message conversation with an individual for more than one conversational turn before you make your request. You can start by complimenting someone on a post, or asking them a clarification question. The request to be interviewed will be more effective if you have already been interacting with them a few times over a period of weeks before you ask. This is a delicate business, because you do not want people to feel manipulated or used. It works better if you participate as a regular participant and then contact people with whom you have had genuine interactions.

Each term, some students fail to get any interviews at all. Typically these students did not participate or interact, but simply marched in and asked for interviews. They also typically send out one or two requests and then wait for an answer, doing nothing for a week or more. You can not simply wait – you need to assume that most requests will not be successful, and continue interacting and requesting interviews until you are successful. This is a labor-intensive process.

Even when someone agrees to be interviewed, you cannot assume that the interview will actually take place. Many people will agree to be interviewed, but then never return the consent form. Others will return the consent form and arrange a time, but then not answer the phone or show up at the time and place you have chosen. This is often passive aggressive behavior: they did not really want to be interviewed, but were reluctant to say so. Or perhaps they want to help, but are simply too busy. When a subject misses a scheduled meeting, you should follow up very lightly (i.e.., you can say something like “I missed seeing you Tuesday. Let me know if you'd like to reschedule”). If they do not reply, do not contact them further. You should take the hint and give them an easy out if they are reluctant to follow through on their agreement to participate. Researchers should not take such rejections personally. It happens to everyone and is to be expected.

Although you must be persistent in requesting interviews, it is important not to disturb the site you are trying to study, and not to annoy people. The boundary between being sufficiently persistent to get results but not so persistent as to be annoying can be quite thin, and even the most experienced researcher may need to stop and talk over details of a plan with colleagues. People are more likely to get annoyed when they see the same request more than once. Thus, you may want to avoid public postings about your study, and focus more on approaching people individually. If you do post publicly, try to make sure any given user sees the posting only once.

The details of how to recruit subjects of course depend on what kinds of subjects you want to reach. That in turn depends on your research questions.

Sampling

The hardest interview to get is your very first. Once you have one interview subject, you can use “snowball sampling:” asking your subject who else you should particularly talk to (Seidman, 2006). This is particularly effective because then you have a de facto introduction to your next subject. You can say,” Pat suggested I talk with you.” An introduction greatly increases the chance of someone agreeing to talk with you.

It is important to note that this biases your sample. If you talk only to one person and their friends and their friends’ friends, you will not get a representative picture of activity on the site. Studies of this nature typically use “purposeful sampling” – you select people to represent certain types of participants. Think about what kinds of people should be in your purposeful sample, and work to recruit accordingly. What kinds of people you need will, of course, depend on your research questions.

Note that no matter what you do, you are more likely to talk to “regulars” – people who have spent a lot of time in the community and are invested in it. People who care less are less likely to agree to be interviewed. Incentives for participation (like small gift cards) may help you to reach a more diverse group of users. Those incentives will also often attract a few people with nothing relevant to say who just want the incentive. It is often acceptable to have a nonrepresentative sample, if this is clearly discussed in your methods and findings.

Consent, waivers of documentation of consent, and minors

After you make initial contact with a participant, you will need to obtain their consent to participate in the study. This paper will not deal with this topic in detail. However, note that we generally use a web form with a click-to-accept consent form. This process requires a waiver of documentation of consent from your IRB: clicking “OK” on a web form is not legal documentation. However, in most cases, the process of documenting consent can create more risk to the subject than participating in the study itself. There is often little or no risk to your privacy from participating in the interview – the risk is created by the consent form that contains your personal information. As a result, most IRBs will approve of a waiver of documentation of consent for such a study, particularly if the subject of the online site is not sensitive. A waiver of documentation of consent is not the same thing as a waiver of consent: the latter is much more rare.

Only adults can consent for themselves. Once someone has agreed to participate in your interview study, you need to ask if they are over 18 years of age. If they are a minor, then they need to provide assent and their parents need to sign a consent form. For minors, instead of a web form, we use paper assent and consent forms that are signed and returned to us by fax or surface mail.

Researchers are often surprised to discover that a particular subject is a minor. Teenagers often interact online with adults without acknowledging their age. In fact, Casey Fiesler studied HarryPotter role-playing groups, and found that some adults, who were role-playing romantic scenes with others, were not aware that they were interacting with teens (Fiesler, 2007). Age strongly shapes whom we interactwith face to face, but these social conventions change online in waysthat are complex and not yet sufficiently studied.

In some cases, researchers may wish to study a site that is explicitly for kids. It is nearly impossible for a strange adultto approach a child online and recruit them for an experimental study. It breaks too many taboos. One effective alternate approach is to leverage face-to-face contacts. For example, students in online communitiesclass who wanted to study neopets.com found children of friends who were willing to participate, and then thosechildren referred their friends and classmates who were also active on the site.

Summary

Getting members of online sites to agree to participate in an interview study is surprisingly challenging and complex – and the task has become more complex over time, as more and more users are annoyed by legions of internet researchers. To summarize, I offer these recommendations:

• conduct interviews in person or by phone – not online;
• study a successful site;
• study a site where you fit in, and can comfortably participate;
• avoid sensitive topics, if possible;
• do not study a site where you are a “regular;”
• get to know the site before you commit to study it;
• read your site's TOS, and abide by them;
• think carefully about whether your site has a legitimate gatekeeper and, if so, ask their permission to approach the group;
• go slowly – lurk before you participate, and participate before you start requesting interviews;
• be honest in your self-presentation – avoid any identity deception even if it is common on the site;
• clearly disclose your status as a researcher in your online self-presentation;
• use your real name, and provide links to your institution and officially approved details of your study;
• take extra precautions on contentious sites or those dealing with highly charged political content (you may consider not using your real name in those circumstances);
• do not respond to “trolls;”
• be persistent in requesting interviews (but not too persistent);
• do not do anything that disturbs natural activity on the site;
• never request an interview of the same person twice, and do not post interview requests in the same forum twice;
• interact with people (ideally more than once) before requesting an interview;
• do not send out requests and wait – keep working on recruiting until you are successful;
• use snowball sampling;
• use purposeful sampling;
• expect to be stood up now and then, and do not take it personally;
• if you need to interview minors, recruit them through face-to-face contacts.

References

Bruckman, A. (2002). Studying the amateur artist: A perspective on disguising data collected in human subjects research on the internet. Ethics and Information Technology, 4(3), 217–231.

Bruckman, A. (2006). Teaching students to study online communities ethically. Journal of Information Ethics, 15(2), 82–98.

Fiesler, C. (2007). Imagined identities: Harry Potter, roleplaying, and blogs. In The Witching Hour: A Magical Compendium (pp. 147–148). Xlibris Corporation.

Hudson, J. M., & Bruckman, A. (2004). “Go away:” Participant objections to being studied. The Information Society, 20(2), 127–139.

Oldenburg, R. (1989). The great good place. New York : Paragon House.

Raymond, E. (1991). The new hacker's dictionary. Cambridge, MA : The MIT Press.

Seidman, I. (2006). Interviewing as qualitative research (3rd ed.). New York : Teachers College Press.