Inspired by the success of evidence-based medicine, environmental scholars and practitioners have grown enthusiastic about applying a similar evidence-based approach to solve some of the world's most pressing environmental problems. An important component of the evidence-based movement is the empirical evaluation of program and policy impacts. Impact evaluations draw heavily from recent advances in the empirical study of causal relationships—the effect of one thing on another. This review highlights the key components of these advances and characterizes the way in which they contribute to better evaluations of the environmental and social impacts of environmental programs. The review emphasizes that a solid understanding of these advances is required before environmental scholars and practitioners can begin to collect the relevant data, analyze them within credible research designs, and generate reliable evidence about the effectiveness of the myriad proposed solutions to the world's environmental and social problems.


Article metrics loading...

Loading full text...

Full text loading...


Literature Cited

  1. Pearl J. 1.  2009. Causality: Models, Reasoning, and Inference Cambridge, UK: Cambridge Univ. Press
  2. Rosenbaum PR. 2.  2002. Design of Observational Studies New York: Springer
  3. Rubin DB. 3.  2007. The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials. Stat. Med. 26:120–36 [Google Scholar]
  4. Angrist JD, Pischke JS. 4.  2009. Mostly Harmless Econometrics: An Empiricist's Companion Princeton, NJ: Princeton Univ. Press
  5. Morgan SL, Winship C. 5.  2007. Counterfactuals and Causal Inference: Methods and Principles for Social Research Cambridge, UK: Cambridge Univ. Press
  6. Ferraro PJ. 6.  2009. Counterfactual thinking and impact evaluation in environmental policy. New Dir. Eval. 122:75–84 [Google Scholar]
  7. Heckman JJ, Vytlacil E. 7.  2005. Structural equations, treatment effects, and econometric policy evaluation. Econometrica 73:3669–738 [Google Scholar]
  8. 8. Millenn. Ecosyst. Assess 2005. Ecosystems and Human Well-Being: A Framework for Assessment. Washington, DC: Island
  9. Joppa LN, Pfaff A. 9.  2009. High and far: biases in the location of protected areas. PLOS ONE 4:12e82739 [Google Scholar]
  10. Pressey RL, Bottrill MC. 10.  2008. Opportunism, threats, and the evolution of systematic conservation planning. Conserv. Biol. 22:51340–45 [Google Scholar]
  11. Aronow PM, Samii C. 11.  2013. Estimating average causal effects under interference between units. arXiv1305.6156
  12. Andam KS, Ferraro PJ, Pfaff A, Sanchez-Azofeifa GA, Robalino JA. 12.  2008. Measuring the effectiveness of protected area networks in reducing deforestation. Proc. Natl. Acad. Sci. USA 105:4216089–89 [Google Scholar]
  13. Gaveau DLA, Epting J, Lyne O, Linkie M, Kumara I. 13.  et al. 2009. Evaluating whether protected areas reduce tropical deforestation in Sumatra. J. Biogeogr. 36:112165–75 [Google Scholar]
  14. Alix-Garcia JM, Shapiro EN, Sims KRE. 14.  2012. Forest conservation and slippage: evidence from Mexico's national payments for ecosystem services program. Land Econ. 88:4613–38 [Google Scholar]
  15. Ferraro PJ. 15.  2012. Experimental project designs in the Global Environment Facility: designing projects to create evidence and catalyze investments to secure global environmental benefits STAP Advis. Doc., Glob. Environ. Facil., Washington, DC
  16. Ferraro PJ, Miranda JJ, Price MK. 16.  2011. The persistence of treatment effects with norm-based policy instruments: evidence from a randomized environmental policy experiment. Am. Econ. Rev. Pap. Proc. 101:3318–22 [Google Scholar]
  17. Allcott H. 17.  2011. Social norms and energy conservation. J. Public Econ. 95:91082–95 [Google Scholar]
  18. Allcott H. 18.  2011. Rethinking real-time electricity pricing. Resour. Energy Econ. 33:4820–42 [Google Scholar]
  19. Wolak FA. 19.  2011. Do residential customers respond to hourly prices? Evidence from a dynamic pricing experiment. Am. Econ. Rev. 101:383–87 [Google Scholar]
  20. Jessoe K, Rapson D. 20.  2014. Knowledge is (less) power: experimental evidence from residential energy use. Am. Econ. Rev. 104:41417–38 [Google Scholar]
  21. Ferraro PJ, Price MK. 21.  2013. Using nonpecuniary strategies to influence behavior: evidence from a large-scale field experiment. Rev. Econ. Stat. 95:164–73 [Google Scholar]
  22. Duo E, Greenstone M, Pande R, Ryan R. 22.  2013. Truth-telling by third-party auditors and the response of polluting firms: experimental evidence from India. Q. J. Econ. 28:41499–545 [Google Scholar]
  23. Bennear L, Tarozzi A, Pfaff A, Balasubramanya S, Ahmed M, van Geen A. 23.  2013. Impact of a randomized controlled trial in arsenic risk communication on household water-source choices in Bangladesh. J. Environ. Econ. Man. 65:2225–40 [Google Scholar]
  24. Fowlie M, Greenstone M, Wolfram C. 24.  2014. Reducing energy consumption and greenhouse gas emissions through energy efficient retrofits: evidence from low-income households Rep., Poverty Action Lab, Cambridge, MA. http://www.povertyactionlab.org/evaluation/reducing-energy-consumption-and-greenhouse-gas-emissions-through-energy-efficient-retrofi
  25. Imbens GW. 25.  2010. Better late than nothing: some comments on Deaton 2009 and Heckman and Urzua 2009. J. Econ. Lit. 48:399–423 [Google Scholar]
  26. Gerber AS, Green DP. 26.  2012. Field Experiments: Design, Analysis, and Interpretation New York: WW Norton
  27. Ludwig J, Kling JR, Mullainathan S. 27.  2011. Mechanism experiments and policy evaluations. J. Econ. Perspect. 25:317–38 [Google Scholar]
  28. Heckman J, Ichimura H, Todd P. 28.  1997. Matching as an econometric evaluation estimator: evidence from evaluating a job training program. Rev. Econ. Stud. 64:4605–54 [Google Scholar]
  29. Heckman J, Ichimura H, Todd P. 29.  1998. Matching as an econometric evaluation estimator. Rev. Econ. Stud. 65:2261–94 [Google Scholar]
  30. Smith J, Todd P. 30.  2005. Does matching overcome Lalonde's critique of nonexperimental estimators?. J. Econom. 125:305–53 [Google Scholar]
  31. Cook T, Shadish W, Wong V. 31.  2008. Three conditions under which observational studies produce the same results as experiments. J. Policy Anal. Manag. 274:724–50 [Google Scholar]
  32. Ferraro PJ, Miranda JJ. 32.  2014. The performance of non-experimental designs in the evaluation of environmental programs: a design-replication study using a large-scale randomized experiment as a benchmark. J. Econ. Behav. Organ. In press
  33. Stuart EA. 33.  2010. Matching methods for causal inference: a review and a look forward. Stat. Sci. 25:11 [Google Scholar]
  34. Imbens GW, Wooldridge JM. 34.  2009. Recent developments in the econometrics of program evaluation. J. Econ. Lit. 47:15–86 [Google Scholar]
  35. Diamond A, Sekhon JS. 35.  2013. Genetic matching for estimating causal effects: a general multivariate matching method for achieving balance in observational studies. Rev. Econ. Stat. 95:3932–45 [Google Scholar]
  36. Hainmueller J. 36.  2012. Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Polit. Anal. 20:125–46 [Google Scholar]
  37. Andam KS, Ferraro PJ, Sims KRE, Healy A, Holland MB. 37.  2010. Protected areas reduced poverty in Costa Rica and Thailand. Proc. Natl. Acad. Sci. USA 107:229996–10001 [Google Scholar]
  38. Lee WS. 38.  2013. Propensity score matching and variations on the balancing test. Empir. Econ. 44:147–80 [Google Scholar]
  39. List JA, Millimet DL, Fredriksson PG, McHone WW. 39.  2003. Effects of environmental regulations on manufacturing plant births: evidence from a propensity score matching estimator. Rev. Econ. Stat. 85:4944–52 [Google Scholar]
  40. Greenstone M. 40.  2004. Did the clean air act cause the remarkable decline in sulfur dioxide concentrations?. J. Environ. Econ. Manag. 47:3585–611 [Google Scholar]
  41. Blackman A, Rivera J. 41.  2010. The evidence base for environmental and socioeconomic impacts of sustainable certification RFF Discuss. Pap., Resour. Futur., Washington, DC. http://www.rff.org/RFF/Documents/EfD-DP-10-10.pdf
  42. Mezzatesta M, Newburn DA, Woodward RT. 42.  2013. Additionality and the adoption of farm conservation practices. Land Econ. 89:4722–42 [Google Scholar]
  43. Blackman A, Naranjo A. 43.  2012. Does eco-certification have environmental benefits? Organic coffee in Costa Rica. Ecol. Econ. 83:58–66 [Google Scholar]
  44. Weber JG, Sills EO, Bauch S, Pattanayak SK. 44.  2011. Do ICDPs work? An empirical evaluation of forest-based microenterprises in the Brazilian Amazon. Land Econ. 87:4661–81 [Google Scholar]
  45. Arriagada RA, Ferraro PJ, Sills EO, Pattanayak SK, Cordero-Sancho S. 45.  2012. Do payments for environmental services affect forest cover? A farm-level evaluation from Costa Rica. Land Econ. 88:2382–99 [Google Scholar]
  46. Joppa L, Pfaff A. 46.  2012. Reassessing the forest impacts of protection: the challenge of nonrandom location and a corrective measure. Ann. N. Y. Acad. Sci. 1185:135–39 [Google Scholar]
  47. Honey-Roses J, Baylis K, Ramírez MI. 47.  2011. A spatially explicit estimate of avoided forest loss. Conserv. Biol. 25:51032–43 [Google Scholar]
  48. Gaveau DLA, Curran LM, Paoli GD, Carlson KM, Wells P. 48.  et al. 2012. Examining protected area effectiveness in Sumatra: importance of regulations governing unprotected lands. Conserv. Lett. 5:2142–48 [Google Scholar]
  49. Andam KS, Ferraro PJ, Hanauer MM. 49.  2013. The effects of protected area systems on ecosystem restoration: a quasi-experimental design to estimate the impact of Costa Rica's protected area system on forest regrowth. Conserv. Lett. 6:5317–23 [Google Scholar]
  50. Ferraro PJ, Hanauer MM, Miteva DA, Canavire-Bacarezza G, Pattanayak SK, Sims KRE. 50.  2013. More strictly protected areas are not necessarily more protective: evidence from Bolivia, Costa Rica, Indonesia, and Thailand. Environ. Res. Lett. 8:025011 [Google Scholar]
  51. Nolte C, Agrawal A. 51.  2013. Linking management effectiveness indicators to observed effects of protected areas on fire occurrence in the Amazon rainforest. Conserv. Biol. 27:1155–65 [Google Scholar]
  52. Ho DE, Imai K, King G, Stuart EA. 52.  2007. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Polit. Anal. 15:199–236 [Google Scholar]
  53. Abadie A, Imbens GW. 53.  2006. Large sample properties of matching estimators for average treatment effects. Econometrica 74:1235–67 [Google Scholar]
  54. Bennear LS, Olmstead SM. 54.  2008. The impacts of the right to know: information disclosure and the violation of drinking water standards. J. Environ. Econ. Manag. 56:2117–30 [Google Scholar]
  55. Sims KRE, Schuetz J. 55.  2009. Local regulation and land-use change: the effects of wetlands bylaws in Massachusetts. Reg. Sci. Urban Econ. 39:4409–21 [Google Scholar]
  56. Burgess R, Hansen M, Olken BA, Potapov P, Sieber S. 56.  2012. The political economy of deforestation in the tropics. Q. J. Econ. 127:41707–54 [Google Scholar]
  57. Davis L, Fuchs A, Gertler P. 57.  2014. Cash for coolers: evaluating a large-scale appliance replacement program in Mexico. Am. Econ. J. Econ. Policy In press
  58. Abadie A, Diamond A, Hainmueller J. 58.  2010. Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. J. Am. Stat. Assoc. 105:490493–505 [Google Scholar]
  59. Sims KRE. 59.  2010. Conservation and development: evidence from Thai protected areas. J. Environ. Econ. Manag. 60:294–114 [Google Scholar]
  60. Chay K, Dobkin C, Greenstone M. 60.  2003. The Clean Air Act of 1970 and adult mortality. J. Risk Uncertain. 27:3279–300 [Google Scholar]
  61. Busch J, Cullen R. 61.  2009. Effectiveness and cost-effectiveness of yellow-eyed penguin recovery. Ecol. Econ. 68:3762–76 [Google Scholar]
  62. Millimet DL. 62.  2011. The Elephant in the Corner: A Cautionary Tale About Measurement Error in Treatment Effects Models 27 Bingley, UK: Emerald Group
  63. Chay K, Greenstone M. 63.  2005. Does air quality matter? Evidence from the housing market. J. Polit. Econ. 113:2376–424 [Google Scholar]
  64. Cutter WB, Neidell M. 64.  2009. Voluntary information programs and environmental regulation: evidence from spare the air. J. Environ. Econ. Manag. 58:3253–65 [Google Scholar]
  65. Alix-Garcia J, McIntosh C, Sims KRE, Welch JR. 65.  2013. The ecological footprint of poverty alleviation: evidence from Mexico's Oportunidades program. Rev. Econ. Stat. 95:2417–35 [Google Scholar]
  66. Liscow ZD. 66.  2013. Do property rights promote investment but cause deforestation? Quasi-experimental evidence from Nicaragua. J. Environ. Econ. Manag. 65:2241–61 [Google Scholar]
  67. Deaton A. 67.  2010. Instruments, randomization and learning about development. J. Econ. Lit. 48:424–55 [Google Scholar]
  68. Altonji J, Elder TE, Taber CR. 68.  2005. Selection on observed and unobserved variables: assessing the effectiveness of Catholic school. J. Polit. Econ. 105:151–84 [Google Scholar]
  69. Ichino A, Mealli F, Nannicini T. 69.  2008. From temporary help jobs to permanent employment: What can we learn from matching estimators and their sensitivity?. J. Appl. Econom. 23:3305–27 [Google Scholar]
  70. Manski CF. 70.  2011. Policy analysis with incredible certitude. Econ. J. 121:554F261–89 [Google Scholar]
  71. Canavire-Bacarezza G, Hanauer MM. 71.  2013. Estimating the impacts of Bolivia's protected areas on poverty. World Dev. 41:265–85 [Google Scholar]
  72. Ferraro PJ, McIntosh C, Ospina M. 72.  2007. The effectiveness of the US Endangered Species Act: an econometric analysis using matching methods. J. Environ. Econ. Manag. 54:3245–61 [Google Scholar]
  73. Butler JS, Moser C. 73.  2007. Cloud cover and satellite images of deforestation. Land Econ. 83:2166–73 [Google Scholar]
  74. Ferraro PJ, Hanauer MM. 74.  2011. Protecting ecosystems and alleviating poverty with parks and reserves: ‘win-win’ or tradeoffs?. Environ. Resour. Econ. 48:2269–86 [Google Scholar]
  75. Ferraro PJ, Hanauer MM, Sims KRE. 75.  2011. Conditions associated with protected area success in conservation and poverty reduction. Proc. Natl. Acad. Sci. USA 108:3413913–18 [Google Scholar]
  76. Pfaff A, Robalino J, Sanchez-Azofeifa GA, Andam KS, Ferraro PJ. 76.  2009. Park location affects forest protection: land characteristics cause differences in park impacts across Costa Rica. B.E. J. Econ. Anal. Policy 9:25 [Google Scholar]
  77. Ferraro PJ, Miranda JJ. 77.  2013. Heterogeneous treatment effects and causal mechanisms in non-pecuniary, information-based environmental policies: evidence from a large-scale field experiment. Res. Energy Econ. 35:356–79 [Google Scholar]
  78. Sun X, Briel M, Busse JW, You JJ, Akl EA. 78.  et al. 2012. Credibility of claims of subgroup effects in randomised controlled trials: systematic review. BMJ 344:e1553 [Google Scholar]
  79. Crump RK, Hotz VJ, Imbens GW, Mitnik OA. 79.  2008. Nonparametric tests for treatment effect heterogeneity. Rev. Econ. Stat. 90:3389–405 [Google Scholar]
  80. Ferraro PJ, Hanauer MM. 80.  2014. Quantifying causal mechanisms to determine how protected areas affect poverty through changes in ecosystem services and infrastructure. Proc. Natl. Acad. Sci. USA 111:114332–37 [Google Scholar]
  81. Pullin AS, Knight T. 81.  2013. Time to build capacity for evidence synthesis in environmental management. Environ. Evid. 2:121 [Google Scholar]

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error