1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Resource Economics
  4. Volume 15, 2023
  5. Article

Annual Review of Resource Economics

Volume 15, 2023

The Rigor Revolution: New Standards of Evidence for Impact Assessment of International Agricultural Research

Abstract

We take stock of the major changes in methodology for studying the impacts of international agricultural research, focusing on the period 2006–2020. Impact assessment of agricultural research has a long and recognized tradition. Until the mid-2000s, such assessments were dominated by a model of demand for and supply of agricultural products in partial equilibrium. The basic ideas for this approach were sketched out by Griliches more than half a century ago. We describe the implications of heightened standards of evidence for good practice in three domains of research design: causal inference, valid measurement, and statistical representativeness. We document advances in each of these domains and review recent evidence that demonstrates the lessons that can be learned from adopting these practices, emphasizing the importance of evidence at-scale, the need to consider portfolios of innovations at a national level, and the challenges of accounting for innovations that are promoted as bundles.

Keyword(s): agricultural research for development, causality, impact assessment, JEL O12, JEL O13, JEL O32, JEL O33, JEL Q16, measurement, scale
Loading

Article metrics loading...

/content/journals/10.1146/annurev-resource-101722-082519
2023-10-05
2024-04-29
Loading full text...

Full text loading...

/deliver/fulltext/resource/15/1/annurev-resource-101722-082519.html?itemId=/content/journals/10.1146/annurev-resource-101722-082519&mimeType=html&fmt=ahah

Literature Cited

  1. Abay K, Wossen T, Abate GA, Stevenson JR, Michelson H, Barrett CB. 2023. Inferential and behavioral implications of measurement error in agricultural data. Annu. Rev. Resour. Econ. 15:63–83
     [Google Scholar]
  2. Aker JC, Jack K. 2021. Harvesting the rain: the adoption of environmental technologies in the Sahel NBER Work. Pap. 29518
  3. Alix-Garcia J, Millimet DL 2023. Remotely incorrect? Accounting for nonclassical measurement error in satellite data on deforestation. J. Assoc. Environ. Resour. Econ.
     [Google Scholar]
  4. Alston JM, Norton G, Pardey PG. 1995. Science Under Scarcity: Principles and Practice for Agricultural Research Evaluation and Priority Setting Ithaca, NY: Cornell Univ. Press
  5. Andrews I, Kasy M. 2019. Identification of and correction for publication bias. Am. Econ. Rev. 109:82766–94
     [Google Scholar]
  6. Angrist JD, Imbens GW, Rubin DB. 1996. Identification of causal effects using instrumental variables. J. Am. Stat. Assoc. 91:434444–55
     [Google Scholar]
  7. Aragón FM, Restuccia D, Rud JP. 2022. Assessing misallocation in agriculture: Plots versus farms NBER Work. Pap. 29749
  8. Baird S, Bohren A, McIntosh C, Özler B. 2018. Optimal designs of experiments in the presence of interference. Rev. Econ. Stat. 100:5844–60
     [Google Scholar]
  9. Banerjee A, Duflo E, Finkelstein A, Katz LF, Olken BA, Sautmann A. 2020. In praise of moderation: suggestions for the scope and use of preanalysis plans for RCTs in economics NBER Work. Pap. 26993
  10. Banerjee A, Duflo E, Goldberg N, Karlan D, Osei R et al. 2015. A multifaceted program causes lasting progress for the very poor: evidence from six countries. Science 348:1260799
     [Google Scholar]
  11. Barrett CB. 2003. Natural resources management research in CGIAR: a meta-evaluation Themat. Work. Pap. 27799 World Bank Washington, DC: https://documents1.worldbank.org/curated/en/171131468135928440/pdf/11050277990CGIAR.pdf
  12. Barrett CB, Agrawal A, Coomes OT, Platteau J-P. 2009. Stripe review of social sciences in the CGIAR Work. Pap. CGIAR Sci. Counc. Rome:
  13. Barrett CB, Benton TG, Cooper KA, Fanzo J, Gandhi R et al. 2020. Bundling innovations to transform agrifood systems. Nat. Sustain. 3:12974–76
     [Google Scholar]
  14. Barrett CB, Carter MR. 2010. The power and pitfalls of experiments in development economics: some nonrandom reflections. Appl. Econ. Perspect. Policy 32:515–48
     [Google Scholar]
  15. Barrett CB, Upton JB. 2013. Food security and sociopolitical stability in Sub-Saharan Africa. Food Security and Sociopolitical Stability CB Barrett 323–56. Oxford, UK: Oxford Univ. Press
     [Google Scholar]
  16. Beaman L, BenYishay A, Magruder J, Mobarak AM. 2021. Can network theory-based targeting increase technology adoption?. Am. Econ. Rev. 11:61918–43
     [Google Scholar]
  17. Beaman L, Dillon A. 2018. Diffusion of agricultural information within social networks: evidence on gender inequalities from Mali. J. Dev. Econ. 133:147–61
     [Google Scholar]
  18. Beegle K, Carletto C, Himelein K. 2012. Reliability of recall in agricultural data. J. Dev. Econ. 98:134–41
     [Google Scholar]
  19. Behaghel L, Gignoux J, Macours K. 2020. Social learning in agriculture: does smallholder heterogeneity impede technology diffusion in Sub-Saharan Africa? CEPR Discuss. Pap. 15220 Cent. Econ. Policy Res. London:
  20. BenYishay A, Mobarak M. 2018. Social learning and incentives for experimentation and communication. Rev. Econ. Stud. 86:3976–1009
     [Google Scholar]
  21. Bernard T, Lambert S, Macours K, Vinez M. 2023. Impact of small farmers’ access to improved seeds and deforestation in DR Congo. Nat. Commun. 14:1603
     [Google Scholar]
  22. Bulte E, Beekman G, Di Falco S, Hella J, Lei P. 2014. Behavioral responses and the impact of new agricultural technologies: evidence from a double-blind field experiment in Tanzania. Am. J. Agric. Econ. 96:3813–30
     [Google Scholar]
  23. Calvi R, Penglase J, Tommasi D. 2022. Measuring women's empowerment in collective households. AEA Pap. Proc. 112:566–60
     [Google Scholar]
  24. Carletto C, Dillon A, Zezza A. 2021. Agricultural data collection to minimize measurement error and maximize coverage. Handbook of Agricultural Economics CB Barrett, DR Just , Vol. 54407–80. Amsterdam: Elsevier
     [Google Scholar]
  25. Carletto C, Gourlay S, Murray S, Zezza A. 2016. Land area measurement in household surveys: Empirical evidence and practical guidance for effective data collection Work. Pap. 147692 LSMS Guidebook, World Bank Washington, DC:
  26. Christensen G, Miguel E 2018. Transparency, reproducibility, and the credibility of economics research. J. Econ. Lit. 56:3920–80
     [Google Scholar]
  27. Corong EL, Hertel TW, McDougall R, Tsigas ME, Van Der Mensbrugghe D. 2017. The standard GTAP model, version 7. J. Glob. Econ. Anal. 2:11–119
     [Google Scholar]
  28. Dalrymple DG. 1978. Development and Spread of High-Yielding Varieties of Wheat and Rice in the Less Developed Nations Foreign Agric. Econ. Rep. 95 Washington, DC: USAID
  29. Dar M, de Janvry A, Emerick K, Sadoulet E, Wiseman E. 2020. Private input suppliers as information agents for technology adoption in agriculture Work. Pap. AFC India Ltd. Kerala:
  30. Deaton A, Cartwright N. 2017. Understanding and misunderstanding randomized controlled trials NBER Work. Pap. 22595
  31. de Janvry A, Dustan A, Sadoulet E. 2011. Recent advances in impact analysis methods for ex-post impact assessments of agricultural technology: options for the CGIAR Rep. Indep. Sci. Partnersh. Counc. Secr. Rome:
     [Google Scholar]
  32. de Janvry A, Sadoulet E, Suri T 2017. Field experiments in developing country agriculture. Handbook of Economic Field Experiments AV Banerjee, E Duflo , Vol. 2427–66. Amsterdam: North-Holland
     [Google Scholar]
  33. De Pinto A, Wiebe K, Pacheco P. 2017. Help bigger palm oil yields to save land. Nature 544:416
     [Google Scholar]
  34. De Weerdt J, Gibson J, Beegle K. 2020. What can we learn from experimenting with survey methods?. Annu. Rev. Resour. Econ. 12:431–47
     [Google Scholar]
  35. Devarajan S, Robinson S. 2002. The influence of computable general equilibrium models on policy. TMD Discuss. Pap. 98 Trade Macroecon. Div., Int. Food Policy Res. Inst. Washington, DC:
  36. Dillon A, Karlan D, Udry C, Zinman J. 2020. Good identification, meet good data. World Dev. 127:104796
     [Google Scholar]
  37. Doss C, Kieran C, Kilic T. 2020. Measuring ownership, control, and use of assets. Fem. Econ. 26:3144–68
     [Google Scholar]
  38. Doss CR. 2006. Analyzing technology adoption using microstudies: Limitations, challenges, and opportunities for improvement. Agric. Econ. 34:3207–19
     [Google Scholar]
  39. Emerick K, Dar M. 2021. Farmer field days and demonstrator selection for increasing technology adoption. Rev. Econ. Stat. 103:4680–93
     [Google Scholar]
  40. Emerick K, de Janvry A, Sadoulet E, Dar M. 2016. Optimizing social learning about agricultural technology: experiments in India and Bangladesh FERDI Policy Brief 158 Fond. Étud. Rech. Dev. Int. Clermont-Ferrand, Fr.:
  41. Erenstein O, Laxmi V. 2008. Zero tillage impacts in India's rice–wheat systems: a review. Soil Tillage Res. 100:1–21–14
     [Google Scholar]
  42. Fuglie K. 2018. R&D capital, R&D spillovers, and productivity growth in world agriculture. Appl. Econ. Perspect. Policy 40:3421–44
     [Google Scholar]
  43. Fulginiti L. 2010. Estimating Griliches’ k-shifts. Am. J. Agric. Econ. 92:186–101
     [Google Scholar]
  44. Garbero A, Marion P, Brailovskaya V. 2018. The Impact of the Adoption of CGIAR's Improved Varieties on Poverty and Welfare Outcomes: A Systematic Review IFAD Res. Ser. 33 Rome: IFAD
     [Google Scholar]
  45. Gechter M, Meager R. 2022. Combining experimental and observational studies in meta-analysis: a debiasing approach Work. Pap. Penn State Univ./London Sch. Econ.
  46. Gollin D, Probst LT, Brower E. 2018. Assessing Poverty Impacts of Agricultural Research: Methods and Challenges for CGIAR Rome: Indep. Sci. Partnersh. Counc.
  47. Gollin D, Udry C. 2021. Heterogeneity, measurement error, and misallocation: evidence from African agriculture. J. Political Econ. 129:1 https://doi.org/10.1086/711369
    [Google Scholar]
  48. Griliches Z. 1957. Hybrid corn: an exploration in the economics of technological change. Econometrica 25:4501–22
     [Google Scholar]
  49. Griliches Z. 1958. Research costs and social returns: hybrid corn and related innovations. J. Political Econ. 66:5419–431
     [Google Scholar]
  50. Herdt RW, Mine S. 2017. Does modern technology increase agricultural productivity? Revisiting the evidence from Loevinsohn et al. Tech. Note, Indep. Sci. Partnersh. Counc. Secr. Rome:
  51. Hurley TM, Pardey PG, Rao X, Andrade RS. 2016. Returns to food and agricultural R&D investments worldwide, 1958–2015 Brief., InSTePP St. Paul, MN:
  52. Imbens G. 2018. Understanding and misunderstanding randomized controlled trials: a commentary on Deaton and Cartwright. Soc. Sci. Med. 210:50–52
     [Google Scholar]
  53. Jaleta M, Tesfaye K, Kilian A, Yirga C, Habte E et al. 2020. Misidentification by farmers of the crop varieties they grow: lessons from DNA fingerprinting of wheat in Ethiopia. PLOS ONE 15:7e0235484
     [Google Scholar]
  54. Jayachandran S, Biradavolu M, Cooper J. 2023. Using machine learning and qualitative interviews to design a five-question women's agency index. World Dev. 161:106076
     [Google Scholar]
  55. Jayachandran S, De Laat J, Lambin EF, Stanton CY, Audy R, Thomas NE 2017. Cash for carbon: a randomized trial of payments for ecosystem services to reduce deforestation. Science 357:267–73
     [Google Scholar]
  56. Kilic T, Sohnesen T. 2015. Same question but different answer: experimental evidence on questionnaire design's impact on poverty measured by proxies Policy Res. Work. Pap. 7182 World Bank Washington, DC: https://ideas.repec.org/p/wbk/wbrwps/7182.html
  57. Kosmowski F, Alemu S, Mallia P, Stevenson J, Macours K. 2020. Shining a brighter light: Comprehensive evidence on adoption and diffusion of CGIAR-related innovations in Ethiopia Synth. Rep. Stand. Panel Impact Assess. Rome, It.:
  58. Kosmowski F, Aragaw A, Kilian A, Ambel A, Ilukor J et al. 2019. Varietal identification in household surveys: results from three household-based methods against the benchmark of DNA fingerprinting in southern Ethiopia. Exp. Agric. 55:3371–85
     [Google Scholar]
  59. Kosmowski F, Stevenson J, Campbell J, Ambel A, Tsegay AH. 2017. On the ground or in the air? A methodological experiment on crop residue cover measurement in Ethiopia. Environ. Manag. 60:705–16
     [Google Scholar]
  60. Laajaj R, Macours K. 2016. Learning-by-doing and learning-from-others: evidence from agronomical trials in Kenya Policy Brief, Fond. Étud. Rech. Dev. Int. Clermont-Ferrand, Fr.:
  61. Laajaj R, Macours K. 2021. Measuring skills in developing countries. J. Hum. Resour. 56:41254–95
     [Google Scholar]
  62. Laajaj R, Macours K, Masso C, Thuita M, Vanlauwe B. 2020. Reconciling yield gains in agronomic trials with returns under African smallholder conditions. Sci. Rep. 10:114286
     [Google Scholar]
  63. Laborde D, Robichaud V, Tokgoz S. 2013. MIRAGRODEP 1.0: documentation Tech. Note, AGRODEP Washington, DC: https://www.agrodep.org/resource/no-20-miragrodep-10-documentation
  64. Loevinsohn M, Sumberg J, Diagne A, Whitfield S. 2013. Under what circumstances and conditions does adoption of technology result in increased agricultural productivity? Rep. Inst. Dev. Stud. Brighton, UK:
  65. MacAskill W. 2015. Doing Good Better: How Effective Altruism Can Help You Help Others, Do Work That Matters, and Make Smarter Choices About Giving Back New York: Penguin
  66. Macours K. 2019. Farmers’ demand and the traits and diffusion of agricultural innovations in developing countries. Annu. Rev. Resour. Econ. 11:483–99
     [Google Scholar]
  67. Magnan N, Spielman DJ, Lybbert TJ, Gulati K. 2015. Leveling with friends: social networks and Indian farmers’ demand for a technology with heterogeneous benefits. J. Dev. Econ. 116:223–51
     [Google Scholar]
  68. Mallia P. 2022. You reap what (you think) you sow? Evidence on farmers’ behavioral adjustments in the case of correct crop varietal identification Work. Pap. 2022-08 Paris Sch. Econ.
  69. Maredia M, Reyes B, Manu-Aduening J, Dankyi A, Hamazakaza P et al. 2016. Testing alternative methods of varietal identification using DNA fingerprinting: results of pilot studies in Ghana and Zambia. Food Secur. Int. Dev. Work. Pap. 149 Mich. State. Univ. East Lansing:
  70. Mason-D'Croz D, Bogard JR, Herrero M, Robinson S, Sulser T et al. 2020. Modelling the global economic consequences of a major African swine fever outbreak in China. Nat. Food 1:4221–28
     [Google Scholar]
  71. Meenakshi JV, Johnson N, Karasalo M. 2021. Designing quasi-experimental impact studies of agricultural research at scale Tech. Note 10 Stand. Panel Impact Assess. Rome:
  72. Oakes JM. 2018. The tribulations of trials: a commentary on Deaton and Cartwright. Soc. Sci. Med. 210:57–59
     [Google Scholar]
  73. OECD (Organ. Econ. Co-op. Dev.) 2005. Paris Declaration on Aid Effectiveness Paris: OECD Publ http://dx.doi.org/10.1787/9789264098084-en
  74. OECD (Organ. Econ. Co-op. Dev.) 2008. Accra Agenda for Action Paris: OECD Publ http://dx.doi.org/10.1787/9789264098107-en
  75. Olken B. 2015. Promises and perils of pre-analysis plans. J. Econ. Perspect. 29:361–80
     [Google Scholar]
  76. Parker SW, Todd PE. 2017. Conditional cash transfers: the case of Progresa/Oportunidades. J. Econ. Lit. 55:3866–915
     [Google Scholar]
  77. Pingali P. 2012. Green revolution: impacts, limits, and the path ahead. PNAS 109:3112302–8
     [Google Scholar]
  78. Poets A, Silverstein K, Pardey P, Hearne S, Stevenson J. 2020. DNA Fingerprinting for Crop Varietal Identification: Fit-for-Purpose Protocols, Their Costs and Analytical Implications Rome: SPIA
  79. Pritchett L. 2002. It pays to be ignorant: a simple political economy of rigorous program evaluation. J. Policy Reform 5:251–69
     [Google Scholar]
  80. Quisumbing AR, Ahmed A, Gilligan DO, Hoddinott J, Kumar N et al. 2020. Randomized controlled trials of multi-sectoral programs: lessons from development research. World Dev. 127:104822
     [Google Scholar]
  81. Quisumbing AR, Sproule K, Martinez EM, Malapit H. 2021. Do tradeoffs among dimensions of women's empowerment and nutrition outcomes exist? Evidence from six countries in Africa and Asia. Food Policy 100:102001
     [Google Scholar]
  82. Raitzer DA, Kelley TG. 2008. Assessing the contribution of impact assessment to donor decisions for international agricultural research. Res. Eval. 17:187–99
     [Google Scholar]
  83. Rao X, Hurley TM, Pardey PG. 2020. Recalibrating the reported returns to agricultural R&D: What if we all heeded Griliches?. Aust. J. Agric. Resour. Econ. 64:3977–1001
     [Google Scholar]
  84. Renkow M, Byerlee D. 2010. The impacts of CGIAR research: a review of recent evidence. Food Policy 35:5391–402
     [Google Scholar]
  85. Robinson S, Mason-D'Croz D, Islam S, Sulser T, Robertson RD et al. 2015. The international model for policy analysis of agricultural commodities and trade (IMPACT): model description for version 3 IFPRI Discuss. Pap. 1483 Int. Food Policy Res. Inst. Washington, DC:
  86. Rosenzweig MR, Udry C. 2020. External validity in a stochastic world: evidence from low-income countries. Rev. Econ. Stud. 87:1343–81
     [Google Scholar]
  87. Savedoff WD, Levine R, Birdsall N. 2006. When will we ever learn? Improving lives through impact evaluation Rep. Cent. Glob. Dev. Washington, DC:
  88. Skoufias E. 2005. PROGRESA and its impacts on the welfare of rural households in Mexico Res. Rep. 139 Int. Food Policy Res. Inst. Washington, DC:
  89. Stern E, Stame N, Mayne J, Forss K, Davies R, Befani B. 2012. Broadening the range of designs and methods for impact evaluations: report of a study commissioned by the Department for International Development Rep. 38 Dep. Int. Dev. London:
  90. Stevenson J, Gantier M, Traxler G, Kosmowski F, Macours K. 2023. The challenge of tracking the reach of post-green revolution crop breeding. Preprint CGIAR Standing Panel on Impact Assessment Rome: https://doi.org/10.21203/rs.3.rs-3028333/v1
  91. Stevenson J, Vanlauwe B, Macours K, Johnson N, Krishnan L et al. 2019. Farmer adoption of plot-and farm-level natural resource management practices: between rhetoric and reality. Glob. Food Secur. 20:101–4
     [Google Scholar]
  92. Stevenson J, Vlek P. 2018. Assessing the adoption and diffusion of sustainable agricultural practices: synthesis of a new set of empirical studies Rep. Stand. Panel Impact Assess. Paris:
  93. Stewart R, Langer L, Rebelo da Silva N, Muchiri E, Zaranika H et al. 2015. The effects of training, innovation and new technology on African smallholder farmers’ economic outcomes and food security: a systematic review. Campbell Syst. Rev. 11:1–224
     [Google Scholar]
  94. Valenzuela E, Hertel TW, Keeney R, Reimer JJ. 2007. Assessing global computable general equilibrium model validity using agricultural price volatility. Am. J. Agric. Econ. 89:2383–97
     [Google Scholar]
  95. Walker TS, Crissman CC. 1996. Case studies of the economic impact of CIP related technologies Rep. Int. Potato Cent. Lima, Peru:
  96. Walker TS, Maredia M, Kelley T, La Rovere R, Templeton D et al. 2008. Strategic guidance for ex post impact assessment of agricultural research Rep. CGIAR Sci. Counc. Rome:
     [Google Scholar]
  97. White H. 2009. Theory-based impact evaluation: principles and practice. J. Dev. Effect. 1:3271–84
     [Google Scholar]
  98. Wiebe K, Sulser TB, Dunston S, Rosegrant MW, Fuglie K et al. 2021. Modeling impacts of faster productivity growth to inform the CGIAR initiative on Crops to End Hunger. PLOS ONE 16:4e0249994
     [Google Scholar]
  99. Wossen T, Girma G, Abdoulaye T, Rabbi I, Olanrewaju A, Alene A et al. 2017. The cassava monitoring survey in Nigeria Rep. Int. Inst. Trop. Agric. Ibadan, Niger.:
/content/journals/10.1146/annurev-resource-101722-082519
Loading
  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error