RESEARCH METHOD
23 Crowdsourcing
Crowdsourcing occurs when an undefined, large group of people (a “crowd”) voluntarily responds to an open call and completes tasks and microprojects.1
Experienced researchers know that planning research takes effort, time, and money to align the necessary tools, participants, and resources. When extra care is taken to properly devise remote user evaluation tasks and experiments, the method of crowdsourcing can be used to elicit a large quantity of data from real people in less time.2
Crowdsourcing leverages the “strength of weak ties”3 in a decentralized model that brings together users and testers—members of the crowd—to evaluate prototypes and submit potential solutions to problems. The “microtasks” that are assigned to volunteers are specifically structured to focus the degree and the nature of effort required from volunteers. A microtask is defined as a short task—either qualitative or quantitative—that is accessed via a common platform, and that can be completed by volunteers within just a few seconds or minutes.4 Once completed, the participants receive some sort of compensation, which can be either monetary (a micropayment) or nonmonetary (e.g., reputation points).
Like most research methods, time and care taken in the design of crowdsourcing evaluations can serve the team well when collecting and analyzing data downstream. When planning crowdsourcing evaluations and microtasks, there are some key design recommendations to consider.5 First, uncomplicated tasks seem to get the most volunteers to participate, so design tasks to be straightforward. Be sure to include questions that have a bona-fide answer as part of the task. Not only will this prevent volunteers from “gaming” the system by entering nonsense that minimizes their time investment while increasing how much they are rewarded, but also it can help teams to flag suspicious responses as potentially invalid. Devise the tests so that completing them correctly and in good faith requires as much or even less work than entering random, invalid responses.
If your stakeholders value quantitative data and require large, statistically relevant samples to take user-centered research seriously, consider using crowdsourcing as a “gateway” method to open their eyes to the potential of other user-centered research methods. Having access to a global crowdsourcing community has both benefits and drawbacks: on one hand, crowdsourcing provides an opportunity for teams to gather and generalize results to represent a more varied, diverse population. On the other hand, there is a lack of demographic information provided by testers, not to mention other unknowns regarding their expertise or intentions. Ideally, to hedge against these drawbacks, consider triangulation to increase confidence of research outcomes.
1. The term “Crowdsourcing” is a portmanteau of the word “crowd” and the business word “outsourcing.” Jeff Howe coined it in “The Rise of Crowdsourcing,” a 2006 Wired magazine article.
2. Kittur, Aniket, Ed H. Chi, and Bongwon Suh, Palo Alto Research Center. “Crowdsourcing for Usability: Using Micro-Task Markets for Rapid, Remote, and Low-Cost User Measurements,” 2007, www.clickadvisor.com
3. See note 1 above.
4. See note 2 above.
5. See note 2 above.
Further Reading
Howe, Jeff. Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business. New York: Crown Business, 2009.
Quinn, Alexander J., and Benjamin B. Bederson. “A Taxonomy of Distributed Human Computation.” University of Maryland Technical Report, 2009.
Using crowdsourcing, frog’s frogmob invites people from all over the world to upload their photographs of interesting trends to inform and inspire designers. The images come together to tell a compelling narrative of how people live in their environments, how they visualize concepts, and the ways in which artifacts create meaning in people’s every day lives.
Courtesy of frog, frogmob.frogdesign.com