1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Psychology
  4. Volume 55, 2004
  5. Article

Annual Review of Psychology

Volume 55, 2004

Human Research and Data Collection via the Internet

Abstract

Advantages and disadvantages of Web and lab research are reviewed. Via the World Wide Web, one can efficiently recruit large, heterogeneous samples quickly, recruit specialized samples (people with rare characteristics), and standardize procedures, making studies easy to replicate. Alternative programming techniques (procedures for data collection) are compared, including client-side as opposed to server-side programming. Web studies have methodological problems; for example, higher rates of drop out and of repeated participation. Web studies must be thoroughly analyzed and tested before launching on-line. Many studies compared data obtained in Web versus lab. These two methods usually reach the same conclusions; however, there are significant differences between college students tested in the lab and people recruited and tested via the Internet. Reasons that Web researchers are enthusiastic about the potential of the new methods are discussed.

Keyword(s): anonymity, browser-based research, CGI, computer programming, computers in psychology, experimental dropouts, experimenter bias, HTML, HTML forms, Internet research, Java, JavaScript, methodology, missing data, on-line research, Perl, survey, Web experiment, WWW
Loading

Article metrics loading...

/content/journals/10.1146/annurev.psych.55.090902.141601
2004-02-04
2025-03-21
Loading full text...

Full text loading...

/deliver/fulltext/ps/55/1/annurev.psych.55.090902.141601.html?itemId=/content/journals/10.1146/annurev.psych.55.090902.141601&mimeType=html&fmt=ahah

Literature Cited

  1. Bailey RD, Foote WE, Throckmorton B. 2000. Human sexual behavior: a comparison of college and Internet surveys. See Birnbaum 2000b pp. 141–68 [Google Scholar]
  2. Bandilla W. 2002. Web surveys—an appropriate mode of data collection for the social sciences?. See Batinic et al. 2002 pp. 1–6 [Google Scholar]
  3. Bargh JA, McKenna KYA. 2004. The Internet and social life. Annu. Rev. Psychol. 55:573–90 [Google Scholar]
  4. Baron J, Siepmann M. 2000. Techniques for creating and using web questionnaires in research and teaching. See Birnbaum 2000b pp. 235–65 [Google Scholar]
  5. Batinic B. ed 2000. Internet für Psychologen Göttingen, Ger.: Hogrefe [Google Scholar]
  6. Batinic B, Reips U-D, Bosnjak M. eds 2002. Online Social Sciences Göttingen, Ger.: Hogrefe & Huber [Google Scholar]
  7. Batinic B, Werner A, Gräf L, Bandilla W. eds 1999. Online Research: Methoden, Anwendungen, Ergebnisse Göttingen, Ger.: Hogrefe [Google Scholar]
  8. Birnbaum M. 1998. SurveyWiz and FactorWiz http://psych.fullerton.edu/mbirnbaum/programs/ [Google Scholar]
  9. Birnbaum MH. 1999a. How to show that 9 > 221: collect judgments in a between-subjects design. Psychol. Methods 4:243–49 [Google Scholar]
  10. Birnbaum MH. 1999b. Testing critical properties of decision making on the Internet. Psychol. Sci. 10:399–407 [Google Scholar]
  11. Birnbaum MH. 1999c WWW Research and Decision Making Keynote address to German Online Res. Soc. Meet.Nürnberg, Ger [Google Scholar]
  12. Birnbaum MH. 2000a. Decision making in the lab and on the Web. See Birnbaum 2000b pp. 3–34 [Google Scholar]
  13. Birnbaum MH. ed 2000b. Psychological Experiments on the Internet San Diego: Academic [Google Scholar]
  14. Birnbaum MH. 2000c. SurveyWiz and FactorWiz: JavaScript Web pages that make HTML forms for research on the Internet. Behav. Res. Methods Instrum. Comput. 32:339–46 [Google Scholar]
  15. Birnbaum MH. 2001a. Introduction to Behavioral Research on the Internet Upper Saddle River, NJ: Prentice Hall [Google Scholar]
  16. Birnbaum MH. 2001b. A Web-based program of research on decision making. See Reips & Bosnjak 2001 pp. 23–55 [Google Scholar]
  17. Birnbaum MH. 2002. Wahrscheinlichkeitslernen. In Das Experimental-psychologische Praktikum im Labor und WWW ed. D Janetzko, M Hildebrand, HA Meyer pp. 141–51 Göttingen, Ger.: Hogrefe [Google Scholar]
  18. Birnbaum MH. 2004. Methodological and ethical issues in conducting social psychology research via the Internet. In Handbook of Methods in Social Psychology ed. C Sansone, CC Morf, AT Panter pp. 359–82 Thousand Oaks, CA: Sage [Google Scholar]
  19. Birnbaum MH, Martin T. 2003. Generalization across people, procedures, and predictions: violations of stochastic dominance and coalescing. In Emerging Perspectives on Decision Research ed. SL Schneider, J Shanteau pp. 84–107 New York: Cambridge Univ. Press [Google Scholar]
  20. Birnbaum MH, Mellers BA. 1989. Mediated models for the analysis of confounded variables and self-selected samples. J. Educ. Stat. 14:146–58 [Google Scholar]
  21. Birnbaum MH, Wakcher SV. 2002. Web-based experiments controlled by JavaScript: an example from probability learning. Behav. Res. Methods Instrum. Comput. 34:189–99 [Google Scholar]
  22. Bosnjak M. 2001. Participation in non-restricted Web surveys: A typology and explanatory model for item non-response. See Reips & Bosnjak 2001 pp. 193–207 [Google Scholar]
  23. Bosnjak M, Batinic B. 2002. Understanding the willingness to participate in online surveys—the case of e-mail questionnaires. See Batinic et al. 2002 pp. 81–92 [Google Scholar]
  24. Breinholt G, Krueger H. 1999. A tutorial on creating logfiles for event-driven applications. Behav. Res. Methods Instrum. Comput. 31:410–15 [Google Scholar]
  25. Brenner V. 2002. Generalizability issues in Internet-based survey research: implications for the Internet addiction controversy. See Batinic et al. 2002 pp. 93–113 [Google Scholar]
  26. Buchanan T. 2000. Potential of the Internet for personality research. See Birnbaum 2000b, pp. 121–40 [Google Scholar]
  27. Buchanan T. 2001. Online personality assessment. See Reips & Bosnjak 2001 pp. 57–74 [Google Scholar]
  28. Buchanan T, Smith JL. 1999a. Research on the Internet: validation of a World Wide Web mediated personality scale. Behav. Res. Methods Instrum. Comput. 31:565–71 [Google Scholar]
  29. Buchanan T, Smith JL. 1999b. Using the Internet for psychological research: personality testing on the World-Wide Web. Br. J. Psychol. 90:125–44 [Google Scholar]
  30. Corkrey R, Parkinson L. 2002. A comparison of four computer-based telephone interviewing methods: getting answers to sensitive questions. Behav. Res. Methods Instrum. Comput. 34:354–63 [Google Scholar]
  31. Dillman DA. 2000. Mail and Internet Surveys: The Tailored Design Method New York: Wiley [Google Scholar]
  32. Dillman DA, Bowker DK. 2001. The Web questionnaire challenge to survey methodologists. See Reips & Bosnjak 2001 pp. 159–78 [Google Scholar]
  33. Döring N. 2002a. Sozialpsychologie des Internet Göttingen, Ger.: Hogrefe [Google Scholar]
  34. Döring N. 2002b. Studying online love and cyber romance. See Batinic et al. 2002 pp. 333–56 [Google Scholar]
  35. Eichstaedt J. 2001. An inaccurate timing filter for reaction time measurement by JAVA applets implementing Internet-based experiments. Behav. Res. Methods Instrum. Comput. 33:179–86 [Google Scholar]
  36. Eichstaedt J. 2002. Measuring differences in preactivation on the Internet: the content category superiority effect. Exp. Psychol. 49:283–91 [Google Scholar]
  37. Francis G, Neath I, Surprenant AM. 2000. The cognitive psychology online laboratory. See Birnbaum 2000b pp. 267–83 [Google Scholar]
  38. Frick A, Bächtiger MT, Reips U-D. 2001. Financial incentives, personal information, and drop-out in online studies. See Reips & Bosnjak 2001 pp. 209–19 [Google Scholar]
  39. Gackenbach J. ed 1998. Psychology and the Internet: Intrapersonal, Interpersonal, and Transpersonal Implications San Diego: Academic [Google Scholar]
  40. Gordon MS, Rosenblum LD. 2001. Audiovisual speech Web-lab: an Internet teaching and research laboratory. Behav. Res. Methods Instrum. Comput. 33:267–69 [Google Scholar]
  41. Göritz A, Reinhold N, Batinic B. 2002. Online panels. See Batinic et al. 2002 pp. 27–47 [Google Scholar]
  42. Gräf L. 2002. Assessing Internet questionnaires: the online pretest lab. See Batinic et al. 2002 pp. 49–68 [Google Scholar]
  43. Hecht H, Oesker M, Kaiser A, Civelek H, Stecker T. 1999. A perception experiment with time-critical graphics animation on the World Wide Web. Behav. Res. Methods Instrum. Comput. 31:439–45 [Google Scholar]
  44. Hertel G, Naumann S, Konradt U, Batinic B. 2002. Personality assessment via Internet: comparing Online and paper-and-pencil questionnaires. See Batinic et al. 2002 pp. 115–33 [Google Scholar]
  45. Hewson CM, Laurent D, Vogel CM. 1996. Proper methodologies for psychological and sociological studies conducted via the Internet. Behav. Res. Methods Instrum. Comput. 28:186–91 [Google Scholar]
  46. Horswill MS, Coster ME. 2001. User-controlled photographic animations, photograph-based questions, and questionnaires: three Internet-based instruments for measuring drivers' risk-taking behavior. Behav. Res. Methods Instrum. Comput. 33:46–58 [Google Scholar]
  47. Huff D. 1954. How to Lie with Statistics New York: Norton [Google Scholar]
  48. Janetzko D. 1999. Statistische Anwendungen im Internet Munich: Addison-Wesley [Google Scholar]
  49. Janetzko D. 2002. Artificial dialogues—dialogue and interview bots for the World Wide Web. See Batinic et al. 2002 pp. 357–64 [Google Scholar]
  50. Janetzko D, Meyer HA, Hildebrand M. eds 2002. Das Experimental-psychologische Praktikum im Labor und WWW [A practical course on psychological experimenting in the laboratory and WWW] Göttingen, Ger.: Hogrefe [Google Scholar]
  51. Joinson AN. 1999. Social desirability, anonymity, and Internet-based questionnaires. Behav. Res. Methods Instrum. Comput. 31:433–38 [Google Scholar]
  52. Joinson AN. 2002. Understanding the Psychology of Internet Behaviour: Virtual Worlds, Real Lives Basingstoke, Hampshire, UK: Palgrave Macmillian [Google Scholar]
  53. Knapp F, Heidingsfelder M. 2001. Drop-out analysis: effects of the survey design. See Reips & Bosnjak 2001 pp. 221–30 [Google Scholar]
  54. Krantz JH. 1998. Psychological Research on the Net http://psych.hanover.edu/research/exponnet.html [Google Scholar]
  55. Krantz JH. 2001. Stimulus delivery on the Web: What can be presented when calibration isn't possible?. See Reips & Bosnjak 2001 pp. 113–30 [Google Scholar]
  56. Krantz JH, Ballard J, Scher J. 1997. Comparing the results of laboratory and world-wide web samples on the determinants of female attractiveness. Behav. Res. Methods Instrum. Comput. 29:264–69 [Google Scholar]
  57. Krantz JH, Dalal R. 2000. Validity of Web-based psychological research. See Birnbaum 2000b pp. 35–60 [Google Scholar]
  58. Kraut R, Patterson M, Lundmark V, Kiesler S, Mukopadhyay T, Scherlis W. 1998. Internet paradox: a social technology that reduces social involvement and psychological well-being?. Am. Psychol. 53:1017–31 [Google Scholar]
  59. Lange M. 1999. Museum of perception and cognition website: using JavaScript to increase interactivity in web-based presentations. Behav. Res. Methods Instrum. Comput. 31:34–45 [Google Scholar]
  60. Laugwitz B. 2001. A Web-experiment on colour harmony principles applied to computer user interface design. See Reips & Bosnjak 2001 pp. 131–45 [Google Scholar]
  61. Lukawetz G. 2002. Empirically quantifying unit-nonresponse-errors in online surveys and suggestions for computational correction methods. See Batinic et al. 2002 pp. 403–15 [Google Scholar]
  62. McClelland G. 2000. Seeing Statistics Florence, KY: Duxbury Press [Google Scholar]
  63. McGraw KO, Tew MD, Williams JE. 2000a. The integrity of Web-based experiments: Can you trust the data?. Psychol. Sci. 11:502–6 [Google Scholar]
  64. McGraw KO, Tew MD, Williams JE. 2000b. PsychExps: an on-line psychology laboratory. See Birnbaum 2000b pp. 219–33 [Google Scholar]
  65. McKenna KYA, Bargh JA. 2000. Plan 9 from cyberspace: the implications of the Internet for personality and social psychology. Personal. Soc. Psychol. Rev. 4:57–75 [Google Scholar]
  66. Morrow RH, McKee AJ. 1998. CGI scripts: a strategy for between-subjects experimental group assignment on the World-Wide Web. Behav. Res. Methods Instrum. Comput. 30:306–8 [Google Scholar]
  67. Mueller JH, Jacobsen DM, Schwarzer R. 2000. What are computing experiences good for: a case study in on-line research. See Birnbaum 2000b pp. 195–216 [Google Scholar]
  68. Musch J. 2000. Die Geschichte des Netzes: Ein historischer Abriss. See Batinic 2000 pp. 15–37 [Google Scholar]
  69. Musch J, Broeder A, Klauer KC. 2001. Improving survey research on the World-Wide Web using the randomized response technique. See Reips & Bosnjak 2001 pp. 179–92 [Google Scholar]
  70. Musch J, Klauer KC. 2002. Psychological experimenting on the World Wide Web: investigating context effects in syllogistic reasoning. See Batinic et al. 2002 pp. 181–212 [Google Scholar]
  71. Musch J, Reips U-D. 2000. A brief history of Web experimenting. See Birnbaum 2000b pp. 61–87 [Google Scholar]
  72. O'Neil KM, Penrod SD. 2001. Methodological variables in Web-based research that may affect results: sample type, monetary incentives, and personal information. Behav. Res. Methods Instrum. Comput. 33:226–33 [Google Scholar]
  73. Pagani D, Lombardi L. 2000. An intercultural examination of facial features communicating surprise. See Birnbaum 2000b pp. 169–94 [Google Scholar]
  74. Pasveer KA, Ellard JH. 1998. The making of a personality inventory: help from the WWW. Behav. Res. Methods Instrum. Comput. 30:309–13 [Google Scholar]
  75. Piper AI. 1998. Conducting social science laboratory experiments on the World Wide Web. Libr. Inf. Sci. Res. 20:5–21 [Google Scholar]
  76. Pohl RF, Bender M, Lachmann G. 2002. Hindsight bias around the world. Exp. Psychol. 49:270–82 [Google Scholar]
  77. Reips U-D. 1995. Web Experimental Psychology Lab. http://www.psychologie.unizh.ch/genpsy/Ulf/Lab/WebExpPsyLab.html [Google Scholar]
  78. Reips U-D. 1997. Das psychologische Experimentieren im Internet. In Internet für Psychologen ed. B Batinic pp. 245–65 Göttingen, Ger.: Hogrefe [Google Scholar]
  79. Reips U-D. 2000. The Web experiment method: advantages, disadvantages, and solutions. See Birnbaum 2000b pp. 89–117 [Google Scholar]
  80. Reips U-D. 2001a. Merging field and institution: running a Web laboratory. See Reips & Bosnjak 2001 pp. 1–22 [Google Scholar]
  81. Reips U-D. 2001b. The Web experimental psychology lab: five years of data collection on the Internet. Behav. Res. Methods Instrum. Comput. 33:201–11 [Google Scholar]
  82. Reips U-D. 2002a. Context effects in Web surveys. See Batinic et al. 2002 pp. 69–79 [Google Scholar]
  83. Reips U-D. 2002b. Standards for Internet experimenting. Exp. Psychol. 49:243–56 [Google Scholar]
  84. Reips U-D. 2002c. Theory and techniques of conducting Web experiments. See Batinic et al. 2002 pp. 219–49 [Google Scholar]
  85. Reips U-D. 2002d. Internet-based psychological experimenting: five dos and five don'ts. Soc. Sci. Comput. Rev. 20:241–49 [Google Scholar]
  86. Reips U-D, Bosnjak M. eds 2001. Dimensions of Internet Science Lengerich, Ger.: Pabst Sci. [Google Scholar]
  87. Reips U-D, Neuhaus C. 2002. WEXTOR: a Web-based tool for generating and visualizing experimental designs and procedures. Behav. Res. Methods Instrum. Comput. 34:234–40 [Google Scholar]
  88. Ruppertsberg AI, Givaty G, Van Veen HAHC, Bülthoff H. 2001. Games as research tools for visual perception over the Internet. See Reips & Bosnjak 2001 pp. 147–58 [Google Scholar]
  89. Sassenberg K, Kreutz S. 2002. Online research and anonymity. See Batinic et al. 2002 pp. 213–27 [Google Scholar]
  90. Schiano DJ. 1997. Convergent methodologies in cyber-psychology: a case study. Behav. Res. Methods Instrum. Comput. 29:270–73 [Google Scholar]
  91. Schillewaert N, Langerak F, Duhamel T. 1998. Non-probability sampling for WWW surveys: a comparison of methods. J. Mark. Res. Soc. 40:307–22 [Google Scholar]
  92. Schmidt WC. 1997a. World-Wide Web survey research: benefits, potential problems, and solutions. Behav. Res. Methods Instrum. Comput. 29:274–79 [Google Scholar]
  93. Schmidt WC. 1997b. World-Wide Web survey research made easy with WWW Survey Assistant. Behav. Res. Methods Instrum. Comput. 29:303–4 [Google Scholar]
  94. Schmidt WC. 2000. The server-side of psychology web experiments. See Birnbaum 2000b pp. 285–310 [Google Scholar]
  95. Schmidt WC. 2001. Presentation accuracy of Web animation methods. Behav. Res. Methods Instrum. Comput. 33:187–200 [Google Scholar]
  96. Schmidt WC, Hoffman R, MacDonald J. 1997. Operate your own World-Wide Web server. Behav. Res. Methods Instrum. Comput. 29:189–93 [Google Scholar]
  97. Schwartz A. 1998. Tutorial: PERL, a psychologically efficient reformatting language. Behav. Res. Methods Instrum. Comput. 30:605–9 [Google Scholar]
  98. Schwarz S, Reips U-D. 2001. CGI versus JavaScript: a Web experiment on the reversed hindsight bias. See Reips & Bosnjak 2001 pp. 75–90 [Google Scholar]
  99. Smith MA, Leigh B. 1997. Virtual subjects: using the Internet as an alternative source of subjects and research environment. Behav. Res. Methods Instrum. Comput. 29:496–505 [Google Scholar]
  100. Stanton JM. 1998. An empirical assessment of data collection using the Internet. Pers. Psychol. 51:709–25 [Google Scholar]
  101. Stevenson AK, Francis G, Kim H. 1999. Java experiments for introductory cognitive psychology courses. Behav. Res. Methods Instrum. Comput. 31:99–106 [Google Scholar]
  102. Suler J. 1996/2003. The Psychology of Cyberspace http://www.rider.edu/~suler/psycyber/psycyber.html [Google Scholar]
  103. Tourangeau R. 2004. Survey research and societal change. Annu. Rev. Psychol. 55:775–801 [Google Scholar]
  104. Tuten TL, Urban DJ, Bosnjak M. 2002. Internet surveys and data quality: a review. See Batinic et al. 2002 pp. 7–26 [Google Scholar]
  105. Tversky A, Kahneman D. 1992. Advances in prospect theory: cumulative representation of uncertainty. J. Risk Uncertain. 5:297–323 [Google Scholar]
  106. Van Veen HAHC, Bülthoff HH, Givaty G. 1998. Psychophysics tests on the Internet: an evaluation. Perception S27:179 [Google Scholar]
  107. Voracek M, Stieger S, Gindl A. 2001. Online replication of evolutionary psychological evidence: sex differences in sexual jealousy in imagined scenarios of mate's sexual versus emotional infidelity. See Reips & Bosnjak 2001 pp. 91–112 [Google Scholar]
  108. Wallace PM. 2001. The Psychology of the Internet Cambridge, UK: Cambridge Univ. Press [Google Scholar]
  109. Welch N, Krantz JH. 1996. The World-Wide Web as a medium for psychoacoustical demonstrations and experiments: experience and results. Behav. Res. Methods Instrum. Comput. 28:192–96 [Google Scholar]
  110. Welker M. 2001. E-mail surveys: non-response figures reflected. See Reips & Bosnjak 2001 pp. 231–38 [Google Scholar]
  111. White RJ, Hammer CA. 2000. Quiz-o-matic: a free Web-based tool for construction of self-scoring on-line quizzes. Behav. Res. Methods Instrum. Comput. 32:250–53 [Google Scholar]
  112. Wilhelm O, McKnight PE. 2002. Ability and achievement testing on the World Wide Web. See Batinic et al. 2002 pp. 151–80 [Google Scholar]
  113. Williams JE, McGraw KO, Tew MD. 1999. Undergraduate labs and computers: the case for PsychExps. Behav. Res. Methods Instrum. Comput. 31:287–91 [Google Scholar]
/content/journals/10.1146/annurev.psych.55.090902.141601
Loading
Human Research and Data Collection via the Internet
Annual Review of Psychology 55, 803 (2004); https://doi.org/10.1146/annurev.psych.55.090902.141601
/content/journals/10.1146/annurev.psych.55.090902.141601
/content/journals/10.1146/annurev.psych.55.090902.141601
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error