1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Resource Economics
  4. Volume 12, 2020
  5. Article

Annual Review of Resource Economics

Volume 12, 2020

What Can We Learn from Experimenting with Survey Methods?

Abstract

This review covers a nascent literature that experiments with survey design to measure whether the way in which we collect socio-economic data in developing countries influences the data and affects the results of subsequent analyses. We start by showing that survey methods matter and the size of the survey design effects can be nothing short of staggering, affecting basic stylized facts of development (such as country rankings by poverty levels) and conclusions drawn from econometric analyses (such as what the returns to education are or whether small farm plots are more productive than large ones). We describe some of the emerging best practices for conducting survey experiments, including benchmarking against the truth, delving into the error-generating mechanisms, and documenting the costs of different survey approaches.

Keyword(s): experiments, measurement error, socio-economic surveys
Loading

Article metrics loading...

/content/journals/10.1146/annurev-resource-103019-105958
2020-10-06
2024-04-20
Loading full text...

Full text loading...

/deliver/fulltext/resource/12/1/annurev-resource-103019-105958.html?itemId=/content/journals/10.1146/annurev-resource-103019-105958&mimeType=html&fmt=ahah

Literature Cited

  1. Abay KA, Abate GT, Barrett CB, Bernard T 2019. Correlated non-classical measurement errors, ‘second best’ policy inference, and the inverse size-productivity relationship in agriculture. J. Dev. Econ. 139:171–84
     [Google Scholar]
  2. Arthi V, Beegle K, De Weerdt J, Palacios-López A 2018. Not your average job: measuring farm labor in Tanzania. J. Dev. Econ. 130:160–72
     [Google Scholar]
  3. Backiny-Yetna P, Steele D, Djima IY 2017. The impact of household food consumption data collection methods on poverty and inequality measures in Niger. Food Policy 72:7–19
     [Google Scholar]
  4. Bardasi E, Beegle K, Dillon A, Serneels P 2011. Do labor statistics depend on how and to whom the questions are asked? Results from a survey experiment in Tanzania. World Bank Econ. Rev. 25:3418–47
     [Google Scholar]
  5. Battistin E, De Nadai M, Krishnan N 2019. The insights and illusions of consumption measurement: evidence from a large scale randomization Paper presented at the IARIW-World Bank Conference: New Approaches to Defining and Measuring Poverty in a Growing World Washington DC: November
  6. Beaman L, Dillon A. 2012. Do household definitions matter in survey design? Results from a randomized survey experiment in Mali. J. Dev. Econ. 98:1124–35
     [Google Scholar]
  7. Beegle K, Carletto G, Himelein K 2012a. Reliability of recall in agricultural data. J. Dev. Econ. 98:134–41
     [Google Scholar]
  8. Beegle K, De Weerdt J, Friedman J, Gibson J 2012b. Methods of household consumption measurement through surveys: experimental results from Tanzania. J. Dev. Econ. 98:13–18
     [Google Scholar]
  9. Blair E, Burton S. 1987. Cognitive processes used by survey respondents to answer behavioral frequency questions. J. Cons. Res. 14:2280–88
     [Google Scholar]
  10. Bound J, Brown C, Mathiowetz N 2001. Measurement error in survey data. Handbook of Econometrics 5: J Heckman, E Leamer 3705–843 Amsterdam: Elsevier Sci.
     [Google Scholar]
  11. Brown N. 1995. Estimation strategies and the judgement of event frequency. J. Exp. Psych. 21:61539–53
     [Google Scholar]
  12. Browning M, Crossley T, Weber G 2003. Asking consumption questions in general purpose surveys. Econ. J. 113:491540–67
     [Google Scholar]
  13. Caeyers B, De Weerdt J, Chalmers N 2012. Improving consumption measurement and other survey data through CAPI: evidence from a randomized experiment. J. Dev. Econ. 98:119–33
     [Google Scholar]
  14. Card D. 1999. The causal effect of education on earnings. Handbook of Labor Economics O Ashenfelter, D Card 1801–63 Amsterdam: Elsevier Sci.
     [Google Scholar]
  15. Carletto G, Gourlay S, Winters P 2015. From guesstimates to GPStimates: land area measurement and implications for agricultural analysis. J. Afr. Econ. 24:5593–628
     [Google Scholar]
  16. Chang LC, Krosnick J. 2003. Measuring the frequency of regular behaviors: comparing the “typical week” to the “past week. .” Soc. Meth. 33:155–80
     [Google Scholar]
  17. Choumert‐Nkolo J, Cust H, Taylor C 2019. Using paradata to collect better survey data: evidence from a household survey in Tanzania. Rev. Dev. Econ. 23:2598–618
     [Google Scholar]
  18. Das J, Hammer J, Sánchez-Páramo C 2012. The impact of recall periods on reported morbidity and health seeking behavior. J. Dev. Econ. 98:176–88
     [Google Scholar]
  19. Deaton A. 2003. Adjusted Indian poverty estimates for 1999–2000. Econ. Political Wkly. 38:4322–26
     [Google Scholar]
  20. Deaton A, Grosh M. 2000. Consumption. Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 Years of the Living Standards Measurement Study M Grosh, P Glewwe 91–133 Washington, DC: World Bank
     [Google Scholar]
  21. de Mel S, McKenzie D, Woodruff C 2009. Measuring microenterprise profits: Must we ask how the sausage is made. ? J. Dev. Econ. 88:119–31
     [Google Scholar]
  22. de Nicola F, Giné X 2014. How accurate are recall data? Evidence from coastal India. J. Dev. Econ. 106:152–65
     [Google Scholar]
  23. Desiere S, Jolliffe D. 2018. Land productivity and plot size: Is measurement error driving the inverse relationship. ? J. Dev. Econ. 130:184–98
     [Google Scholar]
  24. De Weerdt J, Beegle K, Friedman J, Gibson J 2016. The challenge of measuring hunger through survey. Econ. Dev. Cult. Change 64:4727–58
     [Google Scholar]
  25. Dillon A, Gourlay S, McGee K, Oseni G 2019. Land measurement bias and its empirical implications: evidence from a validation exercise. Econ. Dev. Cult. Change 67:3595–624
     [Google Scholar]
  26. Dillon A, Karlan D, Udry C, Zinman J 2020. Good identification, meet good data. World Dev 127:104976
     [Google Scholar]
  27. Dillon B. 2012. Using mobile phones to collect panel data in developing countries. J. Int. Dev. 24:4518–27
     [Google Scholar]
  28. Di Maio M, Fiala N 2019. Be wary of those who ask: a randomized experiment on the size and determinants of the enumerator effect. World Bank Econ. Rev. https://doi.org/10.1093/wber/lhy024
    [Crossref] [Google Scholar]
  29. Doss C, Baah-Boateng W, Boakye-Yiadom L, Catanzarite Z, Deere CD et al. 2013. Measuring personal wealth in developing countries: interviewing men and women about asset values Work. Pap. 15 Cent. Public Policy Indian Inst. Manag:.
  30. Escobal J, Laszlo S. 2008. Measurement error in access to markets. Oxf. Bull. Econ. Stat. 70:2209–43
     [Google Scholar]
  31. FAO (Food Agric. Organ.) 1982. Estimation of Crop Areas and Yields in Agricultural Statistics Rome: FAO
  32. Fermont A, Benson T. 2011. Estimating yield of food crops grown by smallholder farmers: a review in the Uganda context Discuss. Pap. 01097 IFPRI Washington, DC:
  33. Friedman J, Beegle K, De Weerdt J, Gibson J 2017. Decomposing response errors in food consumption measurement: implications for survey design from a survey experiment in Tanzania. Food Policy 72:194–111
     [Google Scholar]
  34. Gaddis I, Oseni-Siwatu G, Palacios-Lopez A, Pieters J 2018. Measuring farm labor: survey experimental evidence from Ghana Work. Pap. 8717 World Bank Washington, DC:
  35. Garlick R, Orkin K, Quinn S 2020. Call me maybe: experimental evidence on frequency and medium effects in microenterprise surveys. World Bank Econ. Rev. 34:2418–43
     [Google Scholar]
  36. Gazeaud J. 2018. Proxy means testing vulnerability to measurement errors? Work. Pap CERDI, Univ. Clermont Auvergne Clermont-Ferrand, Fr:.
  37. Gibson J. 2019. Are you estimating the right thing? An editor reflects. Appl. Econ. Perspect. Policy 41:3329–50
     [Google Scholar]
  38. Gibson J, Beegle K, De Weerdt J, Friedman J 2015. What does variation in household survey methods reveal about the nature of measurement errors in consumption estimates. ? Oxf. Bull. Econ. Stat. 77:3466–74
     [Google Scholar]
  39. Gibson J, Kim B. 2007. Measurement error in recall surveys and the relationship between household size and food demand. Am. J. Agric. Econ. 89:2473–89
     [Google Scholar]
  40. Gibson J, Kim B. 2010. Non‐classical measurement error in long‐term retrospective recall surveys. Oxf. Bull. Econ. Stat. 72:5687–95
     [Google Scholar]
  41. Gibson J, McKenzie D. 2007. Using global positioning systems in household surveys for better economics and better policy. World Bank Res. Obs. 22:2217–41
     [Google Scholar]
  42. Gillen B, Snowberg E, Yariv L 2019. Experimenting with measurement error: techniques with applications to the Caltech cohort study. J. Political Econ. 127:41826–63
     [Google Scholar]
  43. Godlonton S, Hernandez M, Murphy M 2017. Anchoring bias in recall data: evidence from Central America. Am. J. Agric. Econ. 100:2479–501
     [Google Scholar]
  44. Gourlay S, Kilic T, Lobell DB 2019. A new spin on an old debate: errors in farmer-reported production and their implications for inverse scale–productivity relationship in Uganda. J. Dev. Econ. 141:102376
     [Google Scholar]
  45. Jerven M, Johnston D. 2015. Statistical tragedy in Africa? Evaluating the database for African economic development. J. Dev. Stud. 51:2111–15
     [Google Scholar]
  46. Keita N, Carfagna E. 2009. Use of modern geo-positioning devices in agricultural censuses and surveys: use of GPS for crop area measurement Presented at the 57th session of the International Statistical Institute Aug 16–22 Durban, S. Afr:.
  47. Kilic T, Moylan H. 2016. Methodological experiment on measuring asset ownership from a gender perspective Tech. Rep., World Bank Group Washington, DC: https://openknowledge.worldbank.org/handle/10986/33653
  48. Kilic T, Sohnesen T. 2019. Same question but different answer: experimental evidence on questionnaire design's impact on poverty measured by proxies. Rev. Inc. Wealth 65:1144–65
     [Google Scholar]
  49. Lajaaj R, Macours K. 2019. Measuring skills in developing countries. J. Human Res. https://doi.org/10.3368/jhr.56.4.1018-9805R1
    [Crossref] [Google Scholar]
  50. Larsen AF, Headey D, Masters W 2019. Misreporting month of birth: diagnosis and implications for research on nutrition and early childhood in developing countries. Demography 56:2707–28
     [Google Scholar]
  51. Menzel A, Woodruff C. 2019. Gender wage gaps and worker mobility: evidence from the garment sector NBER Work Pap 25982
  52. Niederle M, Vesterlund L. 2007. Do women shy away from competition? Do men compete too much. ? Q. J. Econ. 122:31067–101
     [Google Scholar]
  53. Rom A, Günther I, Borofsky Y 2019. Using sensors to measure technology adoption in the social sciences Work. Pap ETH Zurich:
  54. Sana M, Stecklov G, Weinreb A 2016. A test of the stranger-interviewer norm in the Dominican Republic. Pop. Stud. 70:173–92
     [Google Scholar]
  55. Schündeln M. 2018. Multiple visits and data quality in household surveys. Oxf. Bull. Econ. Stat. 80:2380–405
     [Google Scholar]
  56. Schwarz N. 1999. Self-reports: how the questions shape the answers. Am. Psych. 54:293–105
     [Google Scholar]
  57. Scott C, Amenuvegbe B. 1991. Recall loss and recall duration: an experimental study in Ghana. Inter-Stat 4:131–55
     [Google Scholar]
  58. Serneels P, Beegle K, Dillon A 2017. Do returns to education depend on how and whom you ask. ? Econ. Educ. Rev. 60:15–19
     [Google Scholar]
  59. Sharp M, Buffière B, Himelein K, Gibson J 2019. Effects of data collection methods on estimated household consumption and poverty, and on survey costs: evidence from an experiment in the Marshall Islands Paper presented at the IARIW-World Bank Conference, New Approaches to Defining and Measuring Poverty in a Growing World Washington, DC: November
  60. Stecklov G, Weinreb A, Carletto G 2018. Can incentives improve survey data quality in developing countries? Results from a field experiment in India. J. R. Stat. Soc. A 181:41033–56
     [Google Scholar]
  61. Strack F, Martin L, Schwarz N 1988. Priming and communication: social determinants of information use in judgments of life satisfaction. Eur. J. Soc. Psych. 18:3429–42
     [Google Scholar]
  62. Sudman S, Bradburn N. 1973. Effects of time and memory factors on response in surveys. J. Am. Stat. Assoc. 68:344805–15
     [Google Scholar]
  63. Sudman S, Bradburn NM. 1974. Response Effects in Surveys: A Review and Synthesis Chicago: Aldine
  64. Tourangeau R, Rips L, Rasinski K 2000. The Psychology of Survey Response New York: Cambridge Univ. Press
  65. Tversky A, Kahneman D. 1974. Judgment under uncertainty: heuristics and biases. Science 185:1124–31
     [Google Scholar]
  66. United Nations 2019. Guidelines for Producing Statistics on Asset Ownership from a Gender Perspective New York: United Nations
  67. Visaria P. 2000. Poverty in India during 1994–98: alternative estimates Work. Pap Inst. Econ. Growth New Delhi:
  68. Winkielman P, Knäuper B, Schwarz N 1998. Looking back at anger: reference periods change the interpretation of emotional frequency questions. J. Pers. Soc. Psych. 75:3719–28
     [Google Scholar]
  69. World Bank 2018. Poverty and Shared Prosperity 2018: Piecing Together the Poverty Puzzle Washington, DC: World Bank
  70. Zanella G, Bellani M. 2019. The volatility of survey measures of culture and its consequences Discuss. Pap. 12730 IZA Bonn:
/content/journals/10.1146/annurev-resource-103019-105958
Loading
  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error