1932

Abstract

Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk—many of which are common to other nonprobability samples but unfamiliar to clinical science researchers—and suggests concrete steps to avoid these issues or minimize their impact.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-clinpsy-021815-093623
2016-03-28
2024-04-22
Loading full text...

Full text loading...

/deliver/fulltext/clinpsy/12/1/annurev-clinpsy-021815-093623.html?itemId=/content/journals/10.1146/annurev-clinpsy-021815-093623&mimeType=html&fmt=ahah

Literature Cited

  1. Allan NP, Norr AM, Macatee RJ, Gajewska A, Schmidt NB. 2015. Interactive effects of anxiety sensitivity and emotion regulation on anxiety symptoms. J. Psychopathol. Behav. 37663–72
  2. Am. Psychol. Assoc. 2010 2010. Amendments to the 2002 “Ethical principles of psychologists and code of conduct.”. Am. Psychol. 65:493 [Google Scholar]
  3. Andersson G. 2016. Internet-delivered psychological treatments. Annu. Rev. Clin. Psychol. 12:157–79 [Google Scholar]
  4. Andover MS. 2014. Non-suicidal self-injury disorder in a community sample of adults. Psychiatry Res. 219:305–10 [Google Scholar]
  5. Arbisi PA, Ben-Porath YS. 1995. An MMPI-2 infrequent response scale for use with psychopathological populations: the Infrequency-Psychopathology Scale, F(p). Psychol. Assess. 7:424–31 [Google Scholar]
  6. Arch JJ. 2014. Cognitive behavioral therapy and pharmacotherapy for anxiety: treatment preferences and credibility among pregnant and non-pregnant women. Behav. Res. Ther. 52:53–60 [Google Scholar]
  7. Arch JJ, Twohig MP, Deacon BJ, Landy LN, Bluett EJ. 2015. The credibility of exposure therapy: Does the theoretical rationale matter?. Behav. Res. Ther. 72:81–92 [Google Scholar]
  8. Arditte KA, Çek D, Shaw AM, Timpano KR. 2015. The importance of assessing clinical phenomena in Mechanical Turk research. Psychol. Assess. doi:10.137/pas0000217
  9. Austin EJ. 2005. Personality correlates of the broader autism phenotype as assessed by the Autism Spectrum Quotient (AQ). Personal. Individ. Differ. 38:451–60 [Google Scholar]
  10. Azzam T, Jacobson MR. 2013. Finding a comparison group: Is online crowdsourcing a viable option?. Am. J. Eval. 34:372–84 [Google Scholar]
  11. Barnhoorn JS, Haasnoot E, Bocanegra BR, van Steenbergen H. 2014. QRTEngine: an easy solution for running online reaction time experiments using Qualtrics. Behav. Res. Methods 530:1–12 [Google Scholar]
  12. Behrend TS, Sharek DJ, Meade AW, Wiebe EN. 2011. The viability of crowdsourcing for survey research. Behav. Res. Methods 43:800–13 [Google Scholar]
  13. Benoit K, Conway D, Lauderdale BE, Laver M, Mikhaylov S. 2015. Crowd-sourced text analysis: reproducible and agile reproduction of political data. Am. Polit. Sci. Rev. In press
  14. Beramendi P, Duch RM, Matsuo A. 2014. When lab subjects meet real people: comparing different modes of experiments. Presented at Asian Polit. Methodol. Conf., Taipai
  15. Berinsky AJ, Huber GA, Lenz GS. 2012. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Polit. Anal. 20:351–68 [Google Scholar]
  16. Berinsky AJ, Margolis MF, Sances MW. 2014. Separating the shirkers from the workers? Making sure respondents pay attention on self administered surveys. Am. J. Polit. Sci. 58:739–53 [Google Scholar]
  17. Bernal DR. 2014. Acculturation, acculturative stress, social status, and well-being among English language proficient immigrants. PhD Thesis, Univ. Miami, Coral Gables
  18. Black D, Gates G, Sanders S, Taylor L. 2000. Demographics of the gay and lesbian population in the United States: evidence from available systematic data sources. Demography 37:139–54 [Google Scholar]
  19. Blumberg SJ, Luke JV. 2007. Coverage bias in traditional telephone surveys of low-income and young adults. Public Opin. Q. 71:734–49 [Google Scholar]
  20. Boynton MH, Richman LS. 2014. An online daily diary study of alcohol use using Amazon's Mechanical Turk. Drug Alcohol. Rev. 33:456–61 [Google Scholar]
  21. Brawley AM, Pury CL. 2016. Work experiences on MTurk: job satisfaction, turnover, and information sharing. Comput. Hum. Behav. 54:531–46 [Google Scholar]
  22. Buchanan T, Williams JE. 2010. Ethical issues in psychological research on the Internet. Advanced Methods for Conducting Behavioral Research SD Gosling, JA Johnson 255–71 Washington, DC: Am. Psychol. Assoc. [Google Scholar]
  23. Buhrmester M, Kwang T, Gosling SD. 2011. Amazon's Mechanical Turk: a new source of inexpensive, yet high-quality, data?. Perspect. Psychol. Sci. 6:3–5 [Google Scholar]
  24. Bui DC, Myerson J, Hale S. 2015. Age-related slowing in online samples. Psychol. Record. doi: 10.1007/s40732-015-0135-2
  25. Burke SE, Wang K, Dovidio JF. 2014. Witnessing disclosure of depression: Gender and attachment avoidance moderate interpersonal evaluations. J. Soc. Clin. Psychol. 33:536–59 [Google Scholar]
  26. Cabrera LY, Fitz NS, Reiner PB. 2014. Empirical support for the moral salience of the therapy-enhancement distinction in the debate over cognitive, affective and social enhancement. Neuroethics 8:243–56 [Google Scholar]
  27. Carr A. 2014. An exploration of Mechanical Turk as a feasible recruitment platform for cancer survivors. Undergrad. honors thesis: Univ. Colo., Boulder [Google Scholar]
  28. Casey L, Chandler J, Levine AS, Proctor A, Strolovitch D. 2015. Demographic characteristics of a large sample of US workers. Mathematica Policy Res. Rep. In press
  29. Casler K, Bickel L, Hackett E. 2013. Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing. Comput. Hum. Behav. 29:2156–60 [Google Scholar]
  30. Cavanagh TM. 2014. Cheating on online assessment tests: prevalence and impact on validity. PhD Thesis, Colo. State Univ., Ft. Collins [Google Scholar]
  31. Chandler J, Mueller P, Paolacci G. 2014. Nonnaïveté among Amazon Mechanical Turk workers: consequences and solutions for behavioral researchers. Behav. Res. Methods 46:112–30 [Google Scholar]
  32. Chandler J, Paolacci G, Mueller P. 2013. Risks and rewards of crowdsourcing marketplaces. Handbook of Human Computation P Michelucci 377–92 New York: Springer [Google Scholar]
  33. Chandler J, Paolacci G, Peer E, Mueller P, Ratliff KA. 2015. Using nonnaive participants can reduce effect sizes. Psychol. Sci. 26:1131–39 [Google Scholar]
  34. Clifford S, Jerit J. 2014. Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. J. Exp. Polit. Sci. 1:120–31 [Google Scholar]
  35. Cooper EA, Farid H. 2014. Does the Sun revolve around the Earth? A comparison between the general public and online survey respondents in basic scientific knowledge. Public Underst. Sci. doi: 10.1177/0963662514554354
  36. Corrigan PW, Bink AB, Fokuo JK, Schmidt A. 2015. The public stigma of mental illness means a difference between you and me. Psychiatry Res. 226:186–91 [Google Scholar]
  37. Cougle JR, Hawkins KA, Macatee RJ, Zvolensky MJ, Sarawgi S. 2014. Multiple facets of problematic anger in regular smokers: exploring associations with smoking motives and cessation difficulties. Nicotine Tob. Res. 16:881–85 [Google Scholar]
  38. Crump MJ, McDonnell JV, Gureckis TM. 2013. Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLOS ONE 8:e57410 [Google Scholar]
  39. Daly TM, Nataraajan R. 2015. Swapping bricks for clicks: crowdsourcing longitudinal data on Amazon Turk. J. Bus. Res. 682603–9
  40. Davenport SW, Bergman SM, Bergman JZ, Fearrington ME. 2014. Twitter versus Facebook: exploring the role of narcissism in the motives and usage of different social media platforms. Comput. Hum. Behav. 32:212–20 [Google Scholar]
  41. de Leeuw JR. 2014. jsPsych: a JavaScript library for creating behavioral experiments in a Web browser. Behav. Res. Methods 47:1–12 [Google Scholar]
  42. Devlin HC, Johnson SL, Gruber J. 2015. Feeling good and taking a chance? Associations of hypomania risk with cognitive and behavioral risk taking. Cogn. Ther. Res. 39:473–79 [Google Scholar]
  43. DeWall CN, Pond RS Jr, Carter EC, McCullough ME, Lambert NM. et al. 2014. Explaining the relationship between religiousness and substance use: Self-control matters. J. Personal. Soc. Psychol. 107:339–51 [Google Scholar]
  44. Downes-Le Guin T, Mechling J, Baker R. 2006. Great results from ambiguous sources: cleaning Internet panel data. ESOMAR World Res. Conf. Panel Res. Amsterdam: ESOMAR [Google Scholar]
  45. Eriksson K. 2013. Autism-spectrum traits predict humor styles in the general population. Humor 26:461–75 [Google Scholar]
  46. Farrell AM, Grenier JH, Leiby J. 2014. Scoundrels or stars? Theory and evidence on the quality of workers in online labor markets. SSRN. doi: 10.2139/ssrn.2424718
  47. Feitosa J, Joseph DL, Newman DA. 2015. Crowdsourcing and personality measurement equivalence: a warning about countries whose primary language is not English. Personal. Individ. Differ. 75:47–52 [Google Scholar]
  48. Fergus TA. 2014a. The Cyberchondria Severity Scale (CSS): an examination of structure and relations with health anxiety in a community sample. J. Anxiety Disord. 28:504–10 [Google Scholar]
  49. Fergus TA. 2014b. Are “not just right experiences” (NJREs) specific to obsessive-compulsive symptoms? Evidence that NJREs span across symptoms of emotional disorders. J. Clin. Psychol. 70:353–63 [Google Scholar]
  50. Fergus TA, Bardeen JR. 2014. Emotion regulation and obsessive-compulsive symptoms: a further examination of associations. J. Obsessive-Compuls. Relat. Disord. 3:243–48 [Google Scholar]
  51. Fergus TA, Rowatt WC. 2015. Uncertainty, God, and scrupulosity: Uncertainty salience and priming God concepts interact to cause greater fears of sin. J. Behav. Ther. Exp. Psychiatry 46:93–98 [Google Scholar]
  52. Fergus TA, Valentiner DP, McGrath PB, Gier-Lonsway SL, Kim HS. 2012. Short forms of the Social Interaction Anxiety Scale and the Social Phobia Scale. J. Personal. Assess. 94:310–20 [Google Scholar]
  53. Fort K, Adda G, Cohen KB. 2011. Amazon Mechanical Turk: gold mine or coal mine?. Comput. Linguist 37:413–20 [Google Scholar]
  54. Frederick S. 2005. Cognitive reflection and decision making. J. Econ. Perspect. 19:25–42 [Google Scholar]
  55. Gates GJ. 2014. LGBT demographics: comparisons among population-based surveys. Williams Inst., UCLA Sch. Law
  56. Gerber AS, Green DP. 2012. Field Experiments: Design, Analysis, and Interpretation New York: Norton
  57. Goldin G, Darlow A. 2013. TurkGate, Version 0.4.0. Software. http://www.gideongoldin.github.com/TurkGate/
  58. Goodman JK, Cryder CE, Cheema A. 2013. Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples. J. Behav. Decis. Mak. 26:213–24 [Google Scholar]
  59. Gootzeit JH. 2014. ACT process measures: specificity and incremental value PhD Thesis, Univ. Iowa, Iowa City
  60. Gosling SD, Mason W. 2015. Internet research in psychology. Annu. Rev. Psychol. 66:877–902 [Google Scholar]
  61. Green AE, Kenworthy L, Mosner MG, Gallagher NM, Fearon EW. et al. 2014. Abstract analogical reasoning in high-functioning children with autism spectrum disorders. Autism Res. 7:677–86 [Google Scholar]
  62. Greenblatt JB. 2013. US residential consumer product information: validation of methods for post-stratification weighting of Amazon Mechanical Turk surveys. Lawrence Berkeley Natl. Lab., LBNL Pap. LBNL-6163E, Berkeley, CA
  63. Greenblatt JB. 2014. US residential miscellaneous refrigeration products: results from Amazon Mechanical Turk surveys. Lawrence Berkeley Natl. Lab., LBNL Pap. LBNL-6537E, Berkeley, CA
  64. Groves RM. 2004. Survey Errors and Survey Costs 536 New York: Wiley
  65. Gupta N, Martin D, Hanrahan BV, O'Neill J. 2014. Turk-life in India. Proc. Int. Conf. Support. Group Work, 18th, Sanibel Island1–11 New York: ACM [Google Scholar]
  66. Hauser DJ, Schwarz N. 2015a. Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behav. Res. Methods. In press
  67. Hauser DJ, Schwarz N. 2015b. It's a trap! Instructional manipulation checks prompt systematic thinking on “tricky” tasks. SAGE Open 5:1–5 [Google Scholar]
  68. Hawkins RXD. 2014. Conducting real-time multiplayer experiments on the web. Behav. Res. Methods. doi: 10.3758/s13428-014-0515-6
  69. Heer J, Bostock M. 2010. Crowdsourcing graphical perception: using Mechanical Turk to assess visualization design. Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 30th, Atlanta203–12 New York: ACM [Google Scholar]
  70. Higgins C, McGrath E, Moretto L. 2010. MTurk crowdsourcing: a viable method for rapid discovery of Arabic nicknames. Proc. NAACL HLT 2010 Workshop on Creat. Speech and Language Data with Amazon's Mechanical Turk, Los Angeles89–92 Stroudburg, PA: Assoc. Comput. Linguist. [Google Scholar]
  71. Hillygus DS, Jackson N, Young M. 2014. Professional respondents in non-probability online panels. Online Panel Research: A Data Quality Perspective M Callegaro, R Baker, J Bethlehem, AS Göritz, JA Krosnick, PJ Lavrakas 219–37 Chichester, UK: Wiley [Google Scholar]
  72. Horton JJ, Chilton LB. 2010. The labor economics of paid crowdsourcing. Proc. ACM Conf. Electron. Commer., 11th, Cambridge, MA209–18 New York: ACM [Google Scholar]
  73. Horton JJ, Rand DG, Zeckhauser RJ. 2011. The online laboratory: conducting experiments in a real labor market. Exp. Econ. 14:399–425 [Google Scholar]
  74. Howard A, MacCalla J. 2014. Pilot study to evaluate the effectiveness of a mobile-based therapy and educational app for children. Proc. Workshop on Mob. Med. Appl., 1st, Memphis12–15 New York: ACM [Google Scholar]
  75. Huang JL, Curran PG, Keeney J, Poposki EM, DeShon RP. 2012. Detecting and deterring insufficient effort responding to surveys. J. Bus. Psychol. 27:99–114 [Google Scholar]
  76. Huff C, Tingley D. 2014. “Who are these people?”: evaluating the demographic characteristics and political preferences of MTurk survey respondents Work. Pap., Dep. Gov., Harvard Univ.
  77. Ipeirotis P. 2010. Demographics of Mechanical Turk Work. Pap., Stern Sch. Bus., New York Univ.
  78. Irani LC, Silberman M. 2013, April. Turkopticon: interrupting worker invisibility in Amazon Mechanical Turk. Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 31st, Paris611–20 New York: ACM [Google Scholar]
  79. Jahnke S, Imhoff R, Hoyer J. 2015. Stigmatization of people with pedophilia: two comparative surveys. Arch. Sex. Behav. 44:21–34 [Google Scholar]
  80. Jiang L, Wagner C. 2014. Participation in micro-task crowdsourcing markets as work and leisure: the impact of motivation and micro-time structuring. Presented at Collect. Intell. Conf., June 10–12, MIT, Cambridge, MA
  81. Johnson DR, Borden LA. 2012. Participants at your fingertips using Amazon's Mechanical Turk to increase student-faculty collaborative research. Teach. Psychol. 39:245–51 [Google Scholar]
  82. Johnson PS, Herrmann ES, Johnson MW. 2015. Opportunity costs of reward delays and the discounting of hypothetical money and cigarettes. J. Exp. Anal. Behav. 103:87–107 [Google Scholar]
  83. Jones DN, Olderbak SG. 2014. The associations among dark personalities and sexual tactics across different scenarios. J. Interpers. Violence 29:1050–107 [Google Scholar]
  84. Jones DN, Paulhus DL. 2011. The role of impulsivity in the dark triad of personality. Personal. Individ. Differ. 51:679–82 [Google Scholar]
  85. Jurs SG, Glass GV. 1971. The effect of experimental mortality on the internal and external validity of the randomized comparative experiment. J. Exp. Educ. 40:62–66 [Google Scholar]
  86. Kang R, Brown S, Dabbish L, Kiesler S. 2014. Privacy attitudes of Mechanical Turk workers and the US public. Presented at SOUPS Workshop on Privacy Personas and Segmentation, 10th, Menlo Park, CA, July 9–11 37–47 Berkeley, CA: USENIX Assoc.
  87. Kazai G, Kamps J, Milic-Frayling N. 2012. The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy. Proc. Int. Conf. Inf. Knowl., 21st, Maui2583–86 New York: ACM [Google Scholar]
  88. Khanna S, Ratan A, Davis J, Thies W. 2010. Evaluating and improving the usability of Mechanical Turk for low-income workers in India. Proc. Symp. Comput. Dev., 1st, London453–56 New York: ACM [Google Scholar]
  89. Kim J. 2014. Does stigma against smokers really motivate cessation? A moderated mediation model on the effect of anti-smoking campaigns promoting smoker-related stigma on cessation intentions. Master's thesis, Univ. Wis., Milwaukee
  90. Kittur A, Chi EH, Suh B. 2008. Crowdsourcing user studies with Mechanical Turk. Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 26th, Florence453–56 New York: ACM [Google Scholar]
  91. Klein RA, Ratliff KA, Vianello M, Adams RB Jr, Bahník Š. et al. 2014. Investigating variation in replicability: a “many labs” replication project. Soc. Psychol. 45:142–52 [Google Scholar]
  92. Komarov S, Reinecke K, Gajos KZ. 2013. Crowdsourcing performance evaluations of user interfaces. Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 31st, Paris207–16 New York: ACM [Google Scholar]
  93. Konstam V, Tomek S, Celen-Demirtas S, Sweeney K. 2015. Volunteering and reemployment status in unemployed emerging adults a time-worthy investment?. J. Career Assess. 23:152–65 [Google Scholar]
  94. Kosara R, Ziemkiewicz C. 2010. Do Mechanical Turks dream of square pie charts?. Proc. BELIV'10 Workshop. Beyond Time and Errors: Novel Eval. Methods Inf. Vis., 3rd, Atlanta63–70 New York: ACM [Google Scholar]
  95. Krische SD. 2014. Who is the average individual investor? Numerical skills and implications for accounting research. SSRN. http://www.dx.doi.org/10.2139/ssrn.2426570
  96. Lanaj K, Johnson RE, Barnes CM. 2014. Beginning the workday yet already depleted? Consequences of late-night smartphone use and sleep. Organ. Behav. Hum. Dec. 124:11–23 [Google Scholar]
  97. Lang ASID, Rio-Ross J. 2011. Using Amazon Mechanical Turk to transcribe historical handwritten documents. Code4Lib J. 15:1 [Google Scholar]
  98. Lease M, Hullman J, Bigham JP, Bernstein M, Kim J. et al. 2013. Mechanical Turk is not anonymous. SSRN. doi: 10.2139/ssrn.2228728
  99. Lebowitz MS, Ahn WK, Oltman K. 2015. Sometimes more competent, but always less warm: perceptions of biologically oriented mental-health clinicians. Int. J. Soc. Psychiatry 61:668–76 [Google Scholar]
  100. Lebowitz MS, Pyun JJ, Ahn WK. 2014. Biological explanations of generalized anxiety disorder: effects on beliefs about prognosis and responsibility. Psychiatr. Serv. 65:498–503 [Google Scholar]
  101. Lebreton P, Mäki T, Skodras E, Hupont I, Hirth M. 2015. Bridging the gap between eye tracking and crowdsourcing. Proc. SPIE Conf. Hum. Vis. Electron. Imaging, 20th, San Francisco Bellingham, WA: Int. Soc. Opt. Photonics doi: 10.1117/12.2076745 [Google Scholar]
  102. Lintott C, Reed J. 2013. Human computation in citizen science. Handbook of Human Computation P Michelucci 153–62 New York: Springer [Google Scholar]
  103. Litman L, Robinson J, Rosenzweig C. 2015. The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behav. Res. Methods 47:519–28 [Google Scholar]
  104. Lynn BMD. 2014. Shared sense of purpose and well-being among veterans and non-veterans PhD Thesis, North Carolina State Univ., Raleigh
  105. Marge M, Banerjee S, Rudnicky A. 2010. Using the Amazon Mechanical Turk for transcription of spoken language. Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2010 IEEE, Dallas5270–73 New York: IEEE [Google Scholar]
  106. Marietta G, Viola J, Ibekwe N, Claremon J, Gehlbach H. 2014. Improving relationships through virtual environments: how seeing the world through victims' eyes may prevent bullying Work. Pap., Grad. Sch. Educ., Harvard Univ.
  107. Martin D, Hanrahan BV, O'Neill J, Gupta N. 2014. Being a Turker. Proc. ACM Conf. Comput. Support. Coop. Work Social Comput., 17th, San Francisco224–35 New York: ACM [Google Scholar]
  108. Mason W, Suri S. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behav. Res. Methods 44:1–23 [Google Scholar]
  109. Mason W, Suri S, Watts DJ. 2014. Long-run learning in games of cooperation. Proc. ACM Conf. Econ. Comput., 15th, Palo Alto821–38 New York: ACM [Google Scholar]
  110. Mason W, Watts DJ. 2010. Financial incentives and the performance of crowds. ACM SigKDD Explor. Newsl. 11:100–8 [Google Scholar]
  111. Mason W, Watts DJ. 2012. Collaborative learning in networks. PNAS 109:764–69 [Google Scholar]
  112. McAllister Byun TM, Halpin PF, Szeredi D. 2014. Online crowdsourcing for efficient rating of speech: a validation study. J. Commun. Disord. 53:70–83 [Google Scholar]
  113. Miller J. 2006. Online marketing research. The Handbook of Marketing Research R Grover, M Vriens 110–32 Thousand Oaks, CA: SAGE [Google Scholar]
  114. Miller JD, Few LR, Wilson L, Gentile B, Widiger TA. et al. 2013. The Five-Factor Narcissism Inventory (FFNI): a test of the convergent, discriminant, and incremental validity of FFNI scores in clinical and community samples. Psychol. Assess. 25:748–58 [Google Scholar]
  115. Mitchell GE, Locke KD. 2015. Lay beliefs about autism spectrum disorder among the general public and childcare providers. Autism 19:553–61 [Google Scholar]
  116. Morris RR, Picard R. 2014. Crowd-powered positive psychological interventions. J. Posit. Psychol. 9:509–16 [Google Scholar]
  117. Mullinix K, Drukman J, Freese J. 2014. The generalizability of survey experiments. Inst. Policy Res., Work. Pap Ser., WP-14–19, Northwest. Univ., Evanston, IL
  118. Nichols AL, Webster GD. 2015. Designing a brief measure of social anxiety: psychometric support for a three-item version of the Interaction Anxiousness Scale (IAS-3). Personal. Individ. Differ. 79:110–15 [Google Scholar]
  119. Norr AM, Allan NP, Boffa JW, Raines AM, Schmidt NB. 2015. Validation of the Cyberchondria Severity Scale (CSS): replication and extension with bifactor modeling. J. Anxiety Disord. 31:58–64 [Google Scholar]
  120. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD. et al. 2015. Promoting an open research culture. Science 348:62421422–25 [Google Scholar]
  121. Nouri E, Traaum D. 2014. Cultural differences in playing repeated ultimatum game online with virtual humans. Hawaii Int. Conf. System Sci. (HICSS) 47th, Waikoloa, Hawaii 1213–20 New York: IEEE [Google Scholar]
  122. O'Connell BH, O'Shea D, Gallagher S. 2016. Enhancing social relationships through positive psychology activities: a randomised controlled trial. J. Posit. Psychol. 11149–62
  123. Oppenheimer DM, Meyvis T, Davidenko N. 2009. Instructional manipulation checks: detecting satisficing to increase statistical power. J. Exp. Soc. Psychol. 45:867–72 [Google Scholar]
  124. Palmer CJ, Paton B, Enticott PG, Hohwy J. 2015. “Subtypes” in the presentation of autistic traits in the general adult population. J. Autism Dev. Disord. 45:1291–301 [Google Scholar]
  125. Paolacci G, Chandler J. 2014. Inside the Turk: understanding Mechanical Turk as a participant pool. Curr. Dir. Psychol. Sci 23:184–88 [Google Scholar]
  126. Paolacci G, Chandler J, Ipeirotis PG. 2010. Running experiments on Amazon Mechanical Turk. Judgm. Decis. Mak. 5:411–19 [Google Scholar]
  127. Papa A, Lancaster NG, Kahler J. 2014. Commonalities in grief responding across bereavement and non-bereavement losses. J. Affect. Disord. 161:136–43 [Google Scholar]
  128. Parent J, McKee LG, Rough JN, Forehand R. 2015. The association of parent mindfulness with parenting and youth psychopathology across three developmental stages. J. Abnorm. Child Psychol. doi: 10.1007/s10802-015-9978-x
  129. Pearl RL, Puhl RM, Dovidio JF. 2014. Differential effects of weight bias experiences and internalization on exercise among women with overweight and obesity. J. Health Psychol. doi: 10.1177/1359105313520338
  130. Peer E, Paolacci G, Chandler J, Mueller P. 2012. Selectively recruiting participants from Amazon Mechanical Turk using Qualtrics. SSRN. http://www.ssrn.com/abstract=092100631
  131. Peer E, Vosgerau J, Acquisti A. 2014. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behav. Res. Methods 46:1023–31 [Google Scholar]
  132. Pew Res. Cent 2010. Blacks upbeat about black progress, prospects: a year after Obama's election. http://www.pewsocialtrends.org/2010/01/12/blacks-upbeat-about-black-progress-prospects/
  133. Raihani NJ, Bshary R. 2012. A positive effect of flowers rather than eye images in a large-scale, cross-cultural dictator game. Proc. R. Soc. B 279:3556–64 [Google Scholar]
  134. Raines AM, Boffa JW, Allan NP, Short NA, Schmidt NB. 2015. Hoarding and eating pathology: the mediating role of emotion regulation. Compr. Psychiatry 57:29–35 [Google Scholar]
  135. Ranard BL, Ha YP, Meisel ZF, Asch DA, Hill SS. et al. 2014. Crowdsourcing—harnessing the masses to advance health and medicine, a systematic review. J. Gen. Intern. Med. 29:187–203 [Google Scholar]
  136. Rand DG. 2012. The promise of Mechanical Turk: how online labor markets can help theorists run behavioral experiments. J. Theor. Biol. 299:172–79 [Google Scholar]
  137. Rand DG, Peysakhovich A, Kraft-Todd GT, Newman GE, Wurzbacher O. et al. 2014. Social heuristics shape intuitive cooperation. Nat. Commun. 5:3677 [Google Scholar]
  138. Reese ED, Veilleux JC. 2016. Relationships between craving beliefs and abstinence self-efficacy are mediated by smoking motives and moderated by nicotine dependence. Nicotine Tob. Res. 1848–55
  139. Reidy DE, Berke DS, Gentile B, Zeichner A. 2014. Man enough? Masculine discrepancy stress and intimate partner violence. Personal. Individ. Differ. 68:160–64 [Google Scholar]
  140. Reips UD. 2002. Standards for Internet-based experimenting. Exp. Psychol. 49:243–56 [Google Scholar]
  141. Rojas SL, Widiger TA. 2014. Convergent and discriminant validity of the Five Factor Form. Assessment 21:143–57 [Google Scholar]
  142. Rose P, Segrist DJ. 2012. Difficulty identifying feelings, distress tolerance and compulsive buying: analyzing the associations to inform therapeutic strategies. J. Ment. Health Addict. 10:927–35 [Google Scholar]
  143. Rose P, Segrist DJ. 2014. Negative and positive urgency may both be risk factors for compulsive buying. J. Behav. Addict. 3:128–32 [Google Scholar]
  144. Ruzich E, Allison C, Smith P, Watson P, Auyeung B. et al. 2015. Measuring autistic traits in the general population: a systematic review of the Autism-Spectrum Quotient (AQ) in a nonclinical population sample of 6,900 typical adult males and females. Mol. Autism 6:2 [Google Scholar]
  145. Schleider JL, Weisz JR. 2015. Using Mechanical Turk to study family processes and youth mental health: a test of feasibility. J. Child Fam. Stud. 24:3235–46 [Google Scholar]
  146. Schnorf S, Sedley A, Ortlieb M, Woodruff A. 2014. A comparison of six sample providers regarding online privacy benchmarks. Presented at SOUPS Workshop on Privacy Personas and Segmentation, 10th, Menlo Park, CA, July 9–11 Berkeley, CA: USENIX Assoc.
  147. Shao W, Guan W, Liu T, Clark MA, Merchant RC. et al. 2015. Variations in recruitment yield, costs, speed and participant diversity across Internet platforms in a global study examining the efficacy of an HIV/AIDs and HIV testing animated and live-action video among English- or Spanish-speaking Internet or social media users. Digit. Cult. Educ. 7:2 [Google Scholar]
  148. Shapiro DN, Chandler J, Mueller PA. 2013. Using Mechanical Turk to study clinical populations. Clin. Psychol. Sci. 1:213–20 [Google Scholar]
  149. Shaw AD, Horton JJ, Chen DL. 2011. Designing incentives for inexpert human raters. Proc. ACM 2011 Conf., Comput. Support. Coop. Work, Hangzhou275–84 New York: ACM [Google Scholar]
  150. Silberman M, Irani L, Ross J. 2010. Ethics and tactics of professional crowdwork. XRDS Crossroads ACM Mag. Stud. 17:39–43 [Google Scholar]
  151. Simcox T, Fiez JA. 2014. Collecting response times using Amazon Mechanical Turk and Adobe Flash. Behav. Res. Methods 46:95–111 [Google Scholar]
  152. Simons DJ, Chabris CF. 2012. Common (mis) beliefs about memory: a replication and comparison of telephone and Mechanical Turk survey methods. PLOS ONE 7:e51876 [Google Scholar]
  153. Sina S, Kraus S, Rosenfeld A. 2014. Using the crowd to generate content for scenario-based serious-games. arXiv:1402.5034 [cs.AI]
  154. Standish AJ, Benfield JA, Bernstein MJ, Tragesser S. 2014. Characteristics of borderline personality disorder and disgust sensitivity. Psychol. Rec. 64:869–77 [Google Scholar]
  155. Stewart N, Ungemach C, Harris AJL, Bartels DM, Newell BR. et al. 2015. The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgm. Decis. Mak. 10:479–91 [Google Scholar]
  156. Sturgis P, Allum N, Brunton-Smith I. 2009. Attitudes over time: the psychology of panel conditioning. Methodology of Longitudinal Surveys P Lynn 113–26 Hoboken, NJ: Wiley [Google Scholar]
  157. Suri S, Goldstein DG, Mason WA. 2011. Honesty in an online labor market. . Presented at AAAI Workshops, 25th, San Francisco61–66 Menlo Park, CA: AAAI Press [Google Scholar]
  158. Suri S, Watts DJ. 2011. Cooperation and contagion in web-based, networked public goods experiments. PLOS ONE 6:e16836 [Google Scholar]
  159. ten Brinke L, Black PJ, Porter S, Carney DR. 2015. Psychopathic personality traits predict competitive wins and cooperative losses in negotiation. Personal. Individ. Differ. 79:116–22 [Google Scholar]
  160. Tobin SJ, Loxton NJ, Neighbors C. 2014. Coping with causal uncertainty through alcohol use. Addict. Behav. 39:580–85 [Google Scholar]
  161. Tylka TL, Wood-Barcalow NL. 2015. The Body Appreciation Scale-2: item refinement and psychometric evaluation. Body Image 12:53–67 [Google Scholar]
  162. Usinger T. 2014. Effect of internet administered mindfulness training on anxiety and sleep quality. Undergraduate thesis, Univ. Colo., Boulder
  163. Veilleux JC, Salomaa AC, Shaver JA, Zielinski MJ, Pollert GA. 2015. Multidimensional assessment of beliefs about emotion: development and validation of the Emotion and Regulation Beliefs Scale. Assessment 22:86–100 [Google Scholar]
  164. Veilleux JC, Skinner KD, Reese ED, Shaver JA. 2014. Negative affect intensity influences drinking to cope through facets of emotion dysregulation. Personal. Individ. Differ. 59:96–101 [Google Scholar]
  165. Victor SE, Klonsky ED. 2014. Daily emotion in non-suicidal self-injury. J. Clin. Psychol. 70:364–75 [Google Scholar]
  166. Vlahovic TA, Wang YC, Kraut RE, Levine JM. 2014. Support matching and satisfaction in an online breast cancer support community. Proc. ACM Conf. Hum. Factors Comput. Syst., 32nd, Toronto1625–34 New York: ACM [Google Scholar]
  167. Voracek M, Stieger S, Gindl A. 2001. Online replication of evolutionary psychology evidence: sex differences in sexual jealousy in imagined scenarios of mate's sexual versus emotional infidelity. Dimensions of Internet Science U Reips, M Bosnjak 91–112 Lengerich: Pabst Sci. [Google Scholar]
  168. Weatherly JN, Cookman ML. 2014. Investigating several factors potentially related to endorsing gambling as an escape. Curr. Psychol. 33:422–33 [Google Scholar]
  169. Weinberg JD, Freese J, McElhattan D. 2014. Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociol. Sci. 1:292–310 [Google Scholar]
  170. Wickham RE, Reed DE, Williamson RE. 2015. Establishing the psychometric properties of the Self and Perceived-Partner Authenticity in Relationships Scale-Short Form (AIRS-SF): measurement invariance, reliability, and incremental validity. Personal. Individ. Differ. 77:62–67 [Google Scholar]
  171. Wiens TK, Walker LJ. 2015. The chronic disease concept of addiction: helpful or harmful?. Addict. Res. Theory 23:309–21 [Google Scholar]
  172. Winer ES, Veilleux JC, Ginger EJ. 2014. Development and validation of the Specific Loss of Interest and Pleasure Scale (SLIPS). J. Affect. Disord. 152:193–201 [Google Scholar]
  173. Wong CM. 2015. 50 Percent of millennials believe gender is a spectrum, Fusion's Massive Millennial Poll finds. The Huffington Post. http://www.huffingtonpost.com/2015/02/05/fusion-millennial-poll-gender_n_6624200.html
  174. Wurtele SK, Simons D, Moreno T. 2014. Sexual interest in children among an online sample of men and women: prevalence and correlates. Sex. Abuse 26:546–68 [Google Scholar]
  175. Wymbs BT, Dawson AE. 2015. Screening Amazon's Mechanical Turk for adults with ADHD. J. Attention Disord. doi: 10.1177/1087054715597471
  176. Yang K, Friedman-Wheeler DG, Pronin E. 2014. Thought acceleration boosts positive mood among individuals with minimal to moderate depressive symptoms. Cogn. Ther. Res. 38:261–69 [Google Scholar]
/content/journals/10.1146/annurev-clinpsy-021815-093623
Loading
/content/journals/10.1146/annurev-clinpsy-021815-093623
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error