1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Statistics and Its Application
  4. Volume 6, 2019
  5. Article

Annual Review of Statistics and Its Application

Volume 6, 2019

Nonprobability Sampling and Causal Analysis

Abstract

The long-standing approach of using probability samples in social science research has come under pressure through eroding survey response rates, advanced methodology, and easier access to large amounts of data. These factors, along with an increased awareness of the pitfalls of the nonequivalent comparison group design for the estimation of causal effects, have moved the attention of applied researchers away from issues of sampling and toward issues of identification. This article discusses the usability of samples with unknown selection probabilities for various research questions. In doing so, we review assumptions necessary for descriptive and causal inference and discuss research strategies developed to overcome sampling limitations.

Keyword(s): big data, causal inference, generalizability, heterogeneous treatment effects, measurement error, nonprobability sampling, self-selection, validity
Loading

Article metrics loading...

/content/journals/10.1146/annurev-statistics-030718-104951
2019-03-07
2024-04-20
Loading full text...

Full text loading...

/deliver/fulltext/statistics/6/1/annurev-statistics-030718-104951.html?itemId=/content/journals/10.1146/annurev-statistics-030718-104951&mimeType=html&fmt=ahah

Literature Cited

  1. Angrist JD, Pischke JS 2009. Mostly Harmless Econometrics Princeton, NJ: Princeton Univ. Press
  2. ASA (Am. Stat. Assoc.). 2016. ASA statement on statistical significance and p-values. Am. Stat. 70:131–33
     [Google Scholar]
  3. Athey S, Imbens GW. 2017. The state of applied econometrics: causality and policy evaluation. J. Econ. Perspect. 31:3–32
     [Google Scholar]
  4. Baker R, Brick JM, Bates NA, Battaglia M, Couper MP et al. 2013. Report of the AAPOR task force on non-probability sampling. Rep., Am. Assoc. Public Opin. Res., Oakbrook Terrace, IL. https://www.aapor.org/Education-Resources/Reports/Non-Probability-Sampling.aspx
    [Google Scholar]
  5. Berkson J. 1946. Limitations of the application of fourfold table analysis to hospital data. Epidemiology 2:47–53
     [Google Scholar]
  6. Bethlehem J. 2015. Essay: Sunday shopping—the case of three surveys. Surv. Res. Methods 9:221–30
     [Google Scholar]
  7. Bethlehem J 2017. The perils of non-probability sampling Presentation at Inference from Non Probability Samples, Paris, March 16–17. https://www.europeansurveyresearch.org/conference/non-probability
  8. Bethlehem J, Cobben F, Schouten B 2011. Handbook of Nonresponse in Household Surveys New York: Wiley
  9. Bia M, Mattei A. 2008. A Stata package for the estimation of the dose-response function through adjustment for the generalized propensity score. Stata J. 8:354–73
     [Google Scholar]
  10. Biemer PA. 2010. Total survey error: design, implementation, and evaluation. Public Opin. Q. 74:817–48
     [Google Scholar]
  11. Biemer PP, de Leeuw E, Eckman S, Edwards B, Kreuter F et al. 2017. Total Survey Error in Practice New York: Wiley
  12. Caliendo M, Kühn S. 2011. Start-up subsidies for the unemployed: long-term evidence and effect heterogeneity. J. Public Econ. 95:311–31
     [Google Scholar]
  13. Callegaro M, Baker RP, Bethlehem J, Goritz AS, Krosnick JA, Lavrakas PJ 2014. Online Panel Research: A Data Quality Perspective New York: Wiley
  14. Cattaneo M. 2010. Efficient semiparametric estimation of multi-valued treatment effects under ignorability. J. Econom. 155:138–54
     [Google Scholar]
  15. Cattaneo M, Farrel MH. 2011. Efficient estimation of the dose-response function under ignorability using subclassification on the covariates. Missing Data Methods: Cross-sectional Methods and Applications DM Druker93–127 Bingley, UK: Emerald
     [Google Scholar]
  16. Dafoe A. 2014. Science deserves better: the imperative to share complete replication files. PS Political Sci. Politics 47:60–66
     [Google Scholar]
  17. Dawid AP. 2015. Statistical causality from a decision-theoretic perspective. Annu. Rev. Stat. Appl. 2:273–303
     [Google Scholar]
  18. Desenclos J, Klontz K, Wilder M, Gunn R. 1992. The protective effect of alcohol on the occurrence of epidemic oyster-borne hepatitis A. Epidemiology 3:371–74
     [Google Scholar]
  19. Dever J, Rafferty A, Valliant R. 2008. Internet surveys: Can statistical adjustments eliminate coverage bias?. Surv. Res. Methods 2:47–62
     [Google Scholar]
  20. Dewald WG, Thursby JG, Anderson RG. 1986. Replication in empirical economics: the Journal of Money, Credit and Banking project. Am. Econ. Rev. 76:587–603
     [Google Scholar]
  21. Dutwin D, Buskirk TD. 2017. Apples to oranges or gala versus golden delicious? Comparing data quality of nonprobability internet samples to low response rate probability samples. Public Opin. Q. 81:213–49
     [Google Scholar]
  22. Eckstein H. 1975. Case study and theory in political science. Handbook of Political Science Vol. 1: Political Science: Scope and Theory FI Greenstein, NW Polsby117–76 Boston: Addison-Wesley
     [Google Scholar]
  23. Elliott MR, Valliant R. 2017. Inference for nonprobability samples. Stat. Sci. 32:249–64
     [Google Scholar]
  24. Elwert F. 2013. Graphical causal model. Handbook of Causal Analysis for Social Research S Morgan245–73 Dordrecht, Neth.: SpringerA friendly introduction to graphical causal models.
     [Google Scholar]
  25. Elwert F, Winship C. 2014. Endogenous selection bias: The problem of conditioning on a collider variable. Annu. Rev. Sociol. 40:31–53
     [Google Scholar]
  26. Fisher R. 1935. The logic of inductive inference. J. R. Stat. Soc. A 98:39–54
     [Google Scholar]
  27. Gelman A. 2014. The statistical crisis in science. Am. Sci. 102:460–65
     [Google Scholar]
  28. Gelman A, Stern H. 2006. The difference between “significant” and “not significant” is not itself statistically significant. Am. Stat. 60:328–31
     [Google Scholar]
  29. Greenland S. 2003. Quantifying biases in causal models: classical confounding versus collider-stratification bias. Epidemiology 14:300–5
     [Google Scholar]
  30. Groves RM. 2006. Nonresponse rates and nonresponse bias in household surveys. Public Opin. Q. 70:646–75
     [Google Scholar]
  31. Groves RM, Fowler FJ Jr., Couper MP, Lepkowski JM, Singer E, Tourangeau R 2011. Survey Methodology New York: Wiley
  32. Hernán M, Hernández-Diaz S, Robins J. 2004. A structural approach to selection bias. Epidemiology 155:174–84
     [Google Scholar]
  33. Hirano K, Imbens GW. 2004. The propensity score with continuous treatment. Applied Bayesian Modelling and Causal Inference from Missing Data Perspectives A Gelman, X Meng73–84 New York: Wiley
     [Google Scholar]
  34. Holland P. 1986. Statistics and causal inference. J. Am. Stat. Assoc. 81:945–60
     [Google Scholar]
  35. Imai K, King G, Stuart EA. 2008. Misunderstandings between experimentalists and observationalists about causal inference. J. R. Stat. Soc. 171:481–502Discusses similar problems to the present article.
     [Google Scholar]
  36. Imai K, van Dyk D. 2004. Causal treatment with general treatment regimes: generalizing the propensity score. J. Am. Stat. Assoc. 99:854–66
     [Google Scholar]
  37. Imbens GW. 2000. The role of the propensity score in estimating dose-response functions. Biometrika 3:706–10
     [Google Scholar]
  38. Kaizar EE 2018. Combining data in a single analysis Paper presented at AAAS Annual Meeting, Austin, TX, Feb. 18
  39. Keiding N, Louis TA. 2016. Perils and potentials of self-selected entry to epidemiological studies and surveys. J. R. Stat. Soc. A 179:319–76
     [Google Scholar]
  40. Keiding N, Louis TA. 2018. Web-based enrollment and other types of self-selection in surveys and studies: consequences for generalizability. Annu. Rev. Stat. Appl. 5:25–47Can be read as a refutation of Rothman et al. (2013).
     [Google Scholar]
  41. Kennedy C, Mercer A, Keeter S, Hatley N, McGeeney K, Gimenez A 2016. Evaluating Online Nonprobability Surveys Washington, DC: Pew Res. Cent.
  42. Kim J, Pearl J 1983. A computational model for causal and diagnostic reasoning in inference systems. Proceedings of the Eighth International Joint Conference on Artificial Intelligence 1:190–93 San Francisco: Morgan Kaufmann
     [Google Scholar]
  43. King G. 1991. Stochastic variation: a comment on Lewis-Beck and Skalaban's “The R-square.”. Political Anal. 2:158–200
     [Google Scholar]
  44. King G. 1995. Replication, replication. PS Political Sci. Politics 18:443–99
     [Google Scholar]
  45. King G, Keohane RO, Verba S 1994. Designing Social Inquiry Princeton, NJ: Princeton Univ. Press
  46. Kohler U. 2015. Editorial: maintaining quality. Surv. Res. Methods 9:139–40
     [Google Scholar]
  47. Kreuter F, Valliant R. 2007. A survey on survey statistics: what is done and can be done in Stata. Stata J. 7:1–21
     [Google Scholar]
  48. Lakens D. 2017. Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Soc. Psychol. Personal. Sci. 8:355–62
     [Google Scholar]
  49. Lazarsfeld PF, Berelson B, Gaudet H 1948. The People's Choice: How the Voter Makes Up His Mind in a Presidential Campaign New York: Columbia Univ. Press
  50. Lee BK, Lessler J, Stuart EA. 2010. Improving propensity score weighting using machine learning. Stat. Med. 29:337–46
     [Google Scholar]
  51. Leibenstein H. 1950. Bandwagon, snob, and Veblen effects in the theory of consumers’ demand. J. Econ. 64:183–207
     [Google Scholar]
  52. Little RJ, West BT, Boonstra PS, Hu J 2018. Measures of the degree of departure from ignorable sample selection Paper presented at 73rd Annual Conference of the American Association for Public Opinion Research, Denver, CO, May 16–19
  53. Lohr SL, Raghunathan TE. 2017. Combining survey data with other data sources. Stat. Sci. 32:293–312
     [Google Scholar]
  54. McShane BB, Gal D, Gelman A, Robert C, Tackett JL 2018. Abandon statistical significance. arXiv:1709.07588 [stat.ME]
    [Google Scholar]
  55. Mercer A 2018. Selection bias in nonprobability surveys. A causal inference approach PhD thesis, Univ. Maryland Book-length treatise on nonprobability surveys.
  56. Miettinen O 1985. Theoretical Epidemiology New York: Wiley
  57. Mill JS 1843. A System of Logic, Ratiocinative and Inductive Vol. 1. London: John W. Parker
  58. Morgan S, Winship C 2007. Counterfactuals and Causal Inference: Methods and Principles for Social Research Cambridge, UK: Cambridge Univ. Press
  59. Mutz DC 2011. Population-Based Survey Experiments Princeton, NJ: Princeton Univ. Press
  60. National Academies of Sciences, Engineering, and Medicine 2017a. Federal Statistics, Multiple Data Sources, and Privacy Protection: Next Steps Washington, DC: Natl. Acad. Press
  61. National Academies of Sciences, Engineering, and Medicine 2017b. Innovations in Federal Statistics: Combining Data Sources while Protecting Privacy Washington, DC: Natl. Acad. Press
  62. Neyman JS. 1934. On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. J. R. Stat. Soc. 97:558–625
     [Google Scholar]
  63. Neyman JS, Iwaszkiewicz K, Kolodziejczyk S. 1935. Statistical problems in agricultural experimentation. Suppl. J. R. Stat. Soc. 2:107–80
     [Google Scholar]
  64. Nuzzo R. 2015. How scientists fool themselves—and how they can stop. Nature 526:182–85
     [Google Scholar]
  65. O'Muircheartaigh C, Hedges L. 2014. Generalizing from unrepresentative experiments: a stratified propensity score approach. J. R. Stat. Soc. C 63:195–210
     [Google Scholar]
  66. Pearl J. 1993. Comment: graphical models, causality, and interventions. Stat. Sci. 8:266–69
     [Google Scholar]
  67. Pearl J. 1995. Causal diagrams for empirical research. Biometrika 82:669–710
     [Google Scholar]
  68. Pearl J 2009. Causality: Models, Reasoning, and Inference Cambridge, UK: Cambridge Univ. Press. 2nd ed.
  69. Pearl J, Bareinboim E. 2014. External validity: from do-calculus to transportability across populations. Stat. Sci. 29:579–95
     [Google Scholar]
  70. Popper KR 1962. Conjectures and Refutations: the Growth of Scientific Knowledge London: Routledge
  71. Popper KR 1982. The Open Universe: An Argument for Indeterminism. Postscript to The Logic of Scientific Discovery 2 London: Hutchinson
  72. Rivers D 2007. Sampling for web surveys Paper presented at the 2007 Joint Statistical Meetings, Salt Lake City, UT
  73. Rivers D, Bailey D 2009. Inference from matched samples in the 2008 US national elections Paper presented at the American Association for Public Opinion Research Annual Conference, Hollywood, Florida, March 14–17
  74. Robbins M, Ghosh-Dastidar B, R. R 2017. Blending of probability and convenience samples as applied to a survey of military caregivers Presentation at Inference from Nonprobability Samples, Washington, DC, September 25
  75. Rosenbaum PR. 2015. How to see more in observational studies: Some new quasi-experimental devices. Annu. Rev. Stat. Appl. 2:21–48
     [Google Scholar]
  76. Rothman KJ, Gallacher JE, Hatch EE. 2013. Why representativeness should be avoided. Int. J. Epidemiol. 42:1012–14Argues very strongly against probability samples for causal inference.
     [Google Scholar]
  77. Rubin DB. 1974. Estimating causal effects of treatments in randomized and nonrandomized studies. J. Educ. Psychol. 66:688–701
     [Google Scholar]
  78. Rubin DB. 2001. Estimating the causal effects of smoking. Stat. Med. 20:1395–414
     [Google Scholar]
  79. Selvin HC. 1958. Durkheim's Suicide and problems of empirical research. Am. J. Sociol. 63:607–19
     [Google Scholar]
  80. Shpitser I, VanderWeele TJ, Robins JM 2010. On the validity of covariate adjustment for estimating causal effects. Proceedings of the 26th Conference on Uncertainty and Artificial Intelligence Corvallis, OR: AUAI Press
     [Google Scholar]
  81. Statistics Canada 2017. Data quality toolkit. Statistics Canada https://www.statcan.gc.ca/eng/data-quality-toolkit
    [Google Scholar]
  82. Stuart EA, Bradshaw CP, Leaf PJ. 2015. Assessing the generalizability of randomized trial results to target populations. Prev. Sci. 16:475–85
     [Google Scholar]
  83. Stuart EA, Cole SR, Bradshaw CA, Leaf PJ. 2001. The use of propensity scores to assess the generalizability of results from randomized trials. J. R. Stat. Soc. A 174:369–86
     [Google Scholar]
  84. Tourangeau R, Brick JM, Li J. 2017. Adaptive and responsive survey designs: a review and assessment. J. R. Stat. Soc. A 180:202–23
     [Google Scholar]
  85. Valliant R, Dever J. 2011. Estimating propensity adjustments for volunteer web surveys. Sociol. Methods Res. 40:105–37
     [Google Scholar]
  86. Valliant R, Dever J, Kreuter F 2018. A Practical Guide to Designing and Weighting Survey Samples New York: Springer. 2nd ed.
  87. VanderWeele T, Shpister I. 2011. A new criterion for confounder selection. Biometrics 67:1406–13
     [Google Scholar]
  88. Wang W, Rothschild D, Goel S, Gelman A. 2015. Forecasting elections with non-representative polls. Int. J. Forecast. 31:980–91
     [Google Scholar]
  89. Wasserstein RL, Lazar NA. 2017. The ASA's statement on p-values: context, process, and purpose. Am. Stat. 70:129–31
     [Google Scholar]
  90. Winship C, Morgan S. 1999. The estimation of causal effects from observational data. Annu. Rev. Sociol. 25:659–707Introduces various techniques to estimate causal effects from observational data.
     [Google Scholar]
  91. Wooldridge JM 2009. Introductory Econometrics: A Modern Approach Mason, OH: South-Western. 4th ed.
  92. Yeager DS, Krosnick JA, Chang L, Javitz HS, Levendusky MS et al. 2011. Comparing the accuracy of RDD telephone surveys and Internet surveys conducted with probability and non-probability samples. Public Opin. Q. 75:709–47
     [Google Scholar]
  93. Zubizarreta JR, Small DS, Rosenbaum PR. 2014. Isolation in the construction of natural experiments. Ann. Appl. Stat. 8:2096–121
     [Google Scholar]
/content/journals/10.1146/annurev-statistics-030718-104951
Loading
Nonprobability Sampling and Causal Analysis
Annual Review of Statistics and Its Application 6, 149 (2019); https://doi.org/10.1146/annurev-statistics-030718-104951
/content/journals/10.1146/annurev-statistics-030718-104951
/content/journals/10.1146/annurev-statistics-030718-104951
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error