1932

Abstract

Surveys are an essential approach for eliciting otherwise invisible factors such as perceptions, knowledge and beliefs, attitudes, and reasoning. These factors are critical determinants of social, economic, and political outcomes. Surveys are not merely a research tool. They are also not only a way of collecting data. Instead, they involve creating the process that will generate the data. This allows the researcher to create their own identifying and controlled variation. Thanks to the rise of mobile technologies and platforms, surveys offer valuable opportunities either to study broadly representative samples or to focus on specific groups. This article offers guidance on the complete survey process, from the design of the questions and experiments to the recruitment of respondents and the collection of data to the analysis of survey responses. It covers issues related to the sampling process, selection and attrition, attention and carelessness, survey question design and measurement, response biases, and survey experiments.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-economics-091622-010157
2023-08-02
2024-06-06
Loading full text...

Full text loading...

/deliver/fulltext/economics/15/1/annurev-economics-091622-010157.html?itemId=/content/journals/10.1146/annurev-economics-091622-010157&mimeType=html&fmt=ahah

Literature Cited

  1. Alesina A, Ferroni MF, Stantcheva S. 2021. Perceptions of racial gaps, their causes, and ways to reduce them NBER Work. Pap. 29245
  2. Alesina A, Miano A, Stantcheva S. 2023. Immigration and redistribution. Rev. Econ. Stud. 90:1–39
    [Google Scholar]
  3. Alesina A, Stantcheva S, Teso E. 2018. Intergenerational mobility and preferences for redistribution. Am. Econ. Rev. 108:2521–54
    [Google Scholar]
  4. Allcott H, Taubinsky D. 2015. Evaluating behaviorally motivated policy: experimental evidence from the lightbulb market. Am. Econ. Rev. 105:82501–38
    [Google Scholar]
  5. Althaus SL, Kim YM. 2006. Priming effects in complex information environments: reassessing the impact of news discourse on presidential approval. J. Politics 68:4960–76
    [Google Scholar]
  6. Bansak K, Hainmueller J, Hopkins DJ, Yamamoto T. 2018. The number of choice tasks and survey satisficing in conjoint experiments. Political Anal 26:1112–19
    [Google Scholar]
  7. Bargh JA, Chartrand TL 2014. The mind in the middle: a practical guide to priming and automaticity research. Handbook of Research Methods in Social Psychology H Reis, C Judd 311–44. Cambridge, UK: Cambridge Univ. Press. , 2nd ed..
    [Google Scholar]
  8. Berinsky AJ, Huber GA, Lenz GS. 2012. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Anal 20:3351–68
    [Google Scholar]
  9. Billiet J, Davidov E. 2008. Testing the stability of an acquiescence style factor behind two interrelated substantive variables in a panel design. Sociol. Methods Res. 36:542–62
    [Google Scholar]
  10. Billiet JB, McClendon MJ. 2000. Modeling acquiescence in measurement models for two balanced sets of items. Struct. Equ. Model. 7:4608–28
    [Google Scholar]
  11. Bogner K, Landrock U. 2016. Response biases in standardised surveys GESIS Surv. Guidel., Leibniz Inst. Soc. Sci. Mannheim, Ger.: https://www.gesis.org/fileadmin/upload/SDMwiki/BognerLandrock_Response_Biases_in_Standardised_Surveys.pdf
  12. Bottan NL, Perez-Truglia R. 2022. Choosing your pond: location choices and relative income. Rev. Econ. Stat. 104:51010–27
    [Google Scholar]
  13. Brader T, Valentino NA, Suhay E. 2008. What triggers public opposition to immigration? Anxiety, group cues, and immigration threat. Am. J. Political Sci. 52:4959–78
    [Google Scholar]
  14. Brehm J. 1993. The Phantom Respondents: Opinion Surveys and Political Representation Ann Arbor: Univ. Mich. Press
  15. Byrne BM. 1989. Multigroup comparisons and the assumption of equivalent construct validity across groups: methodological and substantive issues. Multivar. Behav. Res. 24:4503–23
    [Google Scholar]
  16. Byrne BM, Shavelson RJ, Muthén B. 1989. Testing for the equivalence of factor covariance and mean structures: the issue of partial measurement invariance. Psychol. Bull. 105:3456–66
    [Google Scholar]
  17. Cheung GW, Rensvold RB. 2000. Assessing extreme and acquiescence response sets in cross-cultural research using structural equations modeling. J. Cross-Cult. Psychol. 31:2187–212
    [Google Scholar]
  18. Clifford S, Sheagley G, Piston S. 2021. Increasing precision without altering treatment effects: repeated measures designs in survey experiments. Am. Political Sci. Rev. 115:31048–65
    [Google Scholar]
  19. Cohn A, Engelmann J, Fehr E, Maréchal MA. 2015. Evidence for countercyclical risk aversion: an experiment with financial professionals. Am. Econ. Rev. 105:2860–85
    [Google Scholar]
  20. Cohn A, Maréchal MA. 2016. Priming in economics. Curr. Opin. Psychol. 12:17–21
    [Google Scholar]
  21. Dafoe A, Zhang B, Caughey D. 2018. Information equivalence in survey experiments. Political Anal 26:4399–416
    [Google Scholar]
  22. De Jong MG, Steenkamp JBE, Fox JP, Baumgartner H. 2008. Using item response theory to measure extreme response style in marketing research: a global investigation. J. Market. Res. 45:1104–15
    [Google Scholar]
  23. Dechezleprêtre A, Fabre A, Kruse T, Planterose B, Sanchez Chico A, Stantcheva S. 2022. Fighting climate change: international attitudes toward climate policies NBER Work. Pap. 30265
  24. Dillman D, Smyth J, Christian L. 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method Hoboken, NJ: Wiley
  25. Epper T, Fehr E, Fehr-Duda H, Kreiner CT, Lassen DD et al. 2020. Time discounting and wealth inequality. Am. Econ. Rev. 110:41177–205
    [Google Scholar]
  26. Guiso L, Sapienza P, Zingales L. 2018. Time varying risk aversion. J. Financ. Econ. 128:3403–21
    [Google Scholar]
  27. Haaland I, Roth C. 2020. Labor market concerns and support for immigration. J. Public Econ. 191:104256
    [Google Scholar]
  28. Haaland I, Roth C. 2023. Beliefs about racial discrimination and support for pro-Black policies. Rev. Econ. Stat. 105:140–53
    [Google Scholar]
  29. Haaland I, Roth C, Wohlfart J. 2020. Designing information provision experiments CESifo Work. Pap. 8406 CESifo Munich, Ger.:
  30. Hainmueller J, Hangartner D, Yamamoto T. 2015. Validating vignette and conjoint survey experiments against real-world behavior. PNAS 112:82395–400
    [Google Scholar]
  31. Hainmueller J, Hopkins DJ, Yamamoto T. 2014. Causal inference in conjoint analysis: understanding multidimensional choices via stated preference experiments. Political Anal 22:11–30
    [Google Scholar]
  32. Heen M, Lieberman JD, Meithe TD. 2020. A comparison of different online sampling approaches for generating national samples Res. Brief, Cent. Crime Justice Policy, Univ. Nev. Las Vegas:
  33. Hvidberg KB, Kreiner C, Stantcheva S. 2021. Social position and fairness views NBER Work. Pap. 28099
  34. Israel A, Rosenboim M, Shavit T. 2014. Using priming manipulations to affect time preferences and risk aversion: an experimental study. J. Behav. Exp. Econ. 53:36–43
    [Google Scholar]
  35. Johnson T, Kulesa P, Cho YI, Shavitt S. 2005. The relation between culture and response styles: evidence from 19 countries. J. Cross-Cult. Psychol. 36:2264–77
    [Google Scholar]
  36. Jöreskog KG. 2005. Structural equation modeling with ordinal variables using LISREL Unpublished manuscript Upssala Univ. Upssala, Swed:.
  37. Karadja M, Mollerstrom J, Seim D. 2017. Richer (and holier) than thou? The effect of relative income improvements on demand for redistribution. Rev. Econ. Stat. 99:2201–12
    [Google Scholar]
  38. Krosnick JA. 1999. Survey research. Annu. Rev. Psychol. 50:537–67
    [Google Scholar]
  39. Krosnick JA, Narayan S, Smith WR. 1996. Satisficing in surveys: initial evidence. New Dir. Eval. 70:29–44
    [Google Scholar]
  40. Kuziemko I, Norton M, Saez E, Stantcheva S. 2015. How elastic are preferences for redistribution? Evidence from randomized survey experiments. Am. Econ. Rev. 105:41478–508
    [Google Scholar]
  41. Link S, Peichl A, Roth C, Wohlfart J. 2022. Information frictions among firms and households CESifo Work. Pap. 8969 CESifo Munich, Ger.:
  42. Merolla J, Ramakrishnan SK, Haynes C. 2013.. “ Illegal,” “undocumented,” or “unauthorized”: equivalency frames, issue frames, and public opinion on immigration. Perspect. Politics 11:3789–807
    [Google Scholar]
  43. Morren M, Gelissen JP, Vermunt JK. 2011. Dealing with extreme response style in cross-cultural research: a restricted latent class factor analysis approach. Sociol. Methodol. 41:113–47
    [Google Scholar]
  44. Pasek J, Krosnick J. 2010. Optimizing survey questionnaire design in political science: insights from psychology. The Oxford Handbook of American Elections and Political Behavior JE Leighley 27–50. Oxford, UK: Oxford Univ. Press
    [Google Scholar]
  45. Pew Res. Cent 2022. Writing survey questions. Pew Research Center Accessed Sept. 7, 2022. https://www.pewresearch.org/our-methods/u-s-surveys/writing-survey-questions/
    [Google Scholar]
  46. Scheufele DA. 2000. Agenda-setting, priming, and framing revisited: another look at cognitive effects of political communication. Mass Commun. Soc. 3:2–3297–316
    [Google Scholar]
  47. Small ML. 2009. “How many cases do I need?” On science and the logic of case selection in field-based research. Ethnography 10:15–38
    [Google Scholar]
  48. Stantcheva S. 2021. Understanding tax policy: How do people reason?. Q. J. Econ. 136:42309–69
    [Google Scholar]
  49. Stantcheva S. 2022. Understanding of trade NBER Work. Pap. 30040
  50. Steiner P, Atzmüller C, Su D. 2017. Designing valid and reliable vignette experiments for survey research: a case study on the fair gender income gap. J. Methods Measur. Soc. Sci. 7:252–94
    [Google Scholar]
  51. US Census Bur 2019. Counting the hard to count in a census: select topics in international censuses Tech. Note, US Census Bur. Washington, DC:
  52. Weber M, D'Acunto F, Gorodnichenko Y, Coibion O 2022. The subjective inflation expectations of households and firms: measurement, determinants, and implications. J. Econ. Perspect. 36:3157–84
    [Google Scholar]
  53. Welkenhuysen-Gybels J, Billiet J, Cambré B. 2003. Adjustment for acquiescence in the assessment of the construct equivalence of Likert-type score items. J. Cross-Cult. Psychol. 34:6702–22
    [Google Scholar]
/content/journals/10.1146/annurev-economics-091622-010157
Loading
/content/journals/10.1146/annurev-economics-091622-010157
Loading

Data & Media loading...

Supplementary Data

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error