1932

Abstract

Field experiments are randomized experiments that take place under naturalistic conditions. This research method is experiencing rapid growth throughout the social sciences and especially in legal studies, where it is used to rigorously evaluate policies and programs. We begin by charting the growth of field experimentation in law and legal studies, describing the statistical properties of experiments and discussing the practical threats that may undermine experiments conducted in field settings. Next, we review the experimental research literature in a variety of domains: legal institutions, including the judiciary, legislature, and legal profession; incentives, especially as they apply to tax compliance and business law; and laws and obligations, including legal code, policy, and legal theory. We conclude by highlighting some of the challenges that the experimental literature must confront if it is to speak convincingly to issues of law and policy.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-lawsocsci-110413-030936
2014-11-03
2024-04-15
Loading full text...

Full text loading...

/deliver/fulltext/lawsocsci/10/1/annurev-lawsocsci-110413-030936.html?itemId=/content/journals/10.1146/annurev-lawsocsci-110413-030936&mimeType=html&fmt=ahah

Literature Cited

  1. Abramowicz M, Ayres I, Listokin Y. 2011. Randomizing law. Univ. Pa. Law Rev. 159:4929–1005 [Google Scholar]
  2. Abrams DS, Bertrand M, Mullainathan S. 2012. Do judges vary in their treatment of race?. J. Leg. Stud. 41:2347–83 [Google Scholar]
  3. Abrams DS, Rohlfs C. 2011. Optimal bail and the value of freedom: evidence from the Philadelphia Bail Experiment. Econ. Inq. 49:3750–70 [Google Scholar]
  4. Abrams DS, Yoon AH. 2007. The luck of the draw: using random case assignment to investigate attorney ability. Univ. Chic. Law Rev. 74:41145–77 [Google Scholar]
  5. Acquisti A, John LK, Loewenstein G. 2013. What is privacy worth?. J. Leg. Stud. 42:2249–74 [Google Scholar]
  6. Anderson JM, Heaton P. 2012. How much difference does the lawyer make? The effect of defense counsel on murder case outcomes. Yale Law J. 122:1154–217 [Google Scholar]
  7. Angrist JD. 1990. Lifetime earnings and the Vietnam era draft lottery: evidence from Social Security administrative records. Am. Econ. Rev. 80:3313–36 [Google Scholar]
  8. Angrist JD, Imbens GW, Rubin DB. 1996. Identification of causal effects using instrumental variables. J. Am. Stat. Assoc. 91:434444–55 [Google Scholar]
  9. Angrist JD, Pischke JS. 2009. Mostly Harmless Econometrics: An Empiricist's Companion Princeton, NJ: Princeton Univ. Press
  10. Angrist JD, Pischke JS. 2010. The credibility revolution in empirical economics: how better research design is taking the con out of econometrics. J. Econ. Perspect. 24:23–30 [Google Scholar]
  11. Argersinger v. Hamlin 405 U.S. 25 1972.
  12. Aronow PM, Middleton J. 2013. A class of unbiased estimators of the average treatment effect in randomized experiments. J. Causal Inference 1:1135–44 [Google Scholar]
  13. Ashenfelter O, Eisenberg T, Schwab SJ. 1995. Politics and the judiciary: the influence of judicial background on case outcomes. J. Leg. Stud. 24:2257–81 [Google Scholar]
  14. Ayres I. 1991. Fair driving: gender and race discrimination in retail car negotiations. Harvard Law Rev. 104:4817–72 [Google Scholar]
  15. Baradaran S, Findley M, Nielson D, Sharman JC. 2013. Does international law matter?. Minn. Law Rev. 97:3743–837 [Google Scholar]
  16. Baradaran S, Findley M, Nielson D, Sharman JC. 2014. Funding terror. Univ. Pa. Law Rev. 162:3477–536 [Google Scholar]
  17. Bazelon DL. 1973. Defective assistance of counsel. Univ. Cincinnati Law Rev. 42:11–46 [Google Scholar]
  18. Beath A, Christia F, Egorov G, Enikolopov R. 2013. Electoral rules and the quality of politicians: theory and evidence from a field experiment in Afghanistan Work. Pap. No. 2013–27, Dep. Polit. Sci., MIT, Cambridge, MA
  19. Boutron I, John P, Torgerson DJ. 2010. Reporting methodological items in randomized experiments in political science. Ann. Am. Acad. Polit. Soc. Sci. 628:1112–31 [Google Scholar]
  20. Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P. 2008. Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration. Ann. Intern. Med. 148:4295–309 [Google Scholar]
  21. Broockman DE. 2013. Black politicians are more intrinsically motivated to advance blacks' interests: a field experiment manipulating political incentives. Am. J. Polit. Sci. 57:3521–36 [Google Scholar]
  22. Broockman DE. 2014. Distorted communication, unequal representation: constituents communicate less to representatives not of their race. Am. J. Polit. Sci. 58:2307–21 [Google Scholar]
  23. Burruss GW, Kempf-Leonard K. 2002. The questionable advantage of defense counsel in juvenile court. Justice Q. 19:137–68 [Google Scholar]
  24. Butler DM, Broockman DE. 2011. Do politicians racially discriminate against constituents? A field experiment on state legislators. Am. J. Polit. Sci. 55:3463–77 [Google Scholar]
  25. Butler DM, Karpowitz CF, Pope JC. 2012. A field experiment on legislators' home styles: service versus policy. J. Polit. 74:2474–86 [Google Scholar]
  26. Camerer C, Talley E. 2004. Experimental law and economics. Handbook of Law and Economics 2 AM Polinsky, S Shavell 1619–50 Amsterdam: North Holland [Google Scholar]
  27. Campbell DT. 1969. Reforms as experiments. Am. Psychol. 24:4409–29 [Google Scholar]
  28. Casey K, Glennerster R, Miguel E. 2012. Reshaping institutions: evidence on aid impacts using a preanalysis plan. Q. J. Econ. 127:41755–812 [Google Scholar]
  29. Castro L, Scartascini C. 2013. Tax compliance and enforcement in the Pampas: evidence from a field experiment Work. Pap. No. IDB-WP-472, Dep. Res. Chief Econ., Inter-Am. Dev. Bank, Washington, DC
  30. Chipman HA, George EI, McCulloch RE. 2010. BART: Bayesian additive regression trees. Ann. Appl. Stat. 4:1266–98 [Google Scholar]
  31. Clingingsmith D, Khwaja AI, Kremer M. 2009. Estimating the impact of the Hajj: religion and tolerance in Islam's global gathering. Q. J. Econ. 124:31133–70 [Google Scholar]
  32. Cooper H, Hedges LV, Valentine JC. 2009. The Handbook of Research Synthesis and Meta-Analysis New York: Russell Sage Found.
  33. Cooper L. 1980. Goldberg's forgotten footnote: Is there a due process right to a hearing prior to the termination of welfare benefits when the only issue raised is a question of law?. Minn. Law Rev. 64:61107–79 [Google Scholar]
  34. Dafoe A. 2014. Science deserves better: the imperative to share complete replication files. PS: Polit. Sci. Polit. 47:160–66 [Google Scholar]
  35. Dal Bó E, Rossi MA. 2011. Term length and the effort of politicians. Rev. Econ. Stud. 78:41237–63 [Google Scholar]
  36. de Andrade GH, Bruhn M, McKenzie D. 2013. A helping hand or the long arm of the law? Experimental evidence on what governments can do to formalize firms Work. Pap. No. 6435, Finance Private Sect. Dev. Team, Dev. Res. Group, World Bank, Washington, DC
  37. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J. et al. 2005. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. Ann. Int. Med. 143:2146–48 [Google Scholar]
  38. de Rooij EA, Green DP, Gerber AS. 2009. Field experiments on political behavior and collective action. Annu. Rev. Polit. Sci. 12:389–95 [Google Scholar]
  39. Deschenes EP, Turner S, Greenwood PW. 1995. Drug court or probation? An experimental evaluation of Maricopa County's drug court. Justice Syst. J. 18:155–73 [Google Scholar]
  40. Dewald WG, Thursby JG, Anderson RG. 1986. Replication in empirical economics: the Journal of Money, Credit and Banking Project. Am. Econ. Rev. 76:4587–603 [Google Scholar]
  41. Diamond SS, Mueller P. 2010. Empirical legal scholarship in law reviews. Annu. Rev. Law Soc. Sci. 6:581–99 [Google Scholar]
  42. Druckman JN, Green DP, Kuklinski JH, Lupia A. 2010. Experimentation in political science. Cambridge Handbook of Experimental Political Science JN Druckman, DP Green, JH Kuklinski, A Lupia 3–14 Cambridge, UK: Cambridge Univ. Press [Google Scholar]
  43. Eigen ZJ, Listokin Y. 2012. Do lawyers really believe their own hype, and should they? A natural experiment. J. Leg. Stud. 41:2239–67 [Google Scholar]
  44. Eisenberg T, Fisher T, Rosen-Zvi I. 2012. Does the judge matter? Exploiting random assignment on a court of last resort to assess judge and case selection effects. J. Empir. Leg. Stud. 9:2246–90 [Google Scholar]
  45. Elbers NA, Akkermans AJ, Cuijpers P, Bruinvels DJ. 2013. Effectiveness of a web-based intervention for injured claimants: a randomized controlled trial. Trials 14:227 [Google Scholar]
  46. Ellis JW. 1990. Plans, protections, and professional intervention: innovations in divorce custody reform and the role of legal professionals. Univ. Mich. J. Law Reform 24:65–188 [Google Scholar]
  47. Evangelou E, Trikalinos TA, Ioannidis JPA. 2005. Unavailability of online supplementary scientific information from articles published in major journals. FASEB J. 19:1943–44 [Google Scholar]
  48. Farrington DP. 2013. Longitudinal and experimental research in criminology. Crime Justice 42:1453–527 [Google Scholar]
  49. Farrington DP, Welsh BC. 2006. A half century of randomized experiments on crime and justice. Crime Justice 34:155–120 [Google Scholar]
  50. Fellner G, Sausgruber R, Traxler C. 2013. Testing enforcement strategies in the field: threat, moral appeal and social information. J. Eur. Econ. Assoc. 11:2634–60 [Google Scholar]
  51. Gazal-Ayal O, Sulitzeanu-Kenan R. 2010. Let my people go: ethnic in-group bias in judicial decisions—evidence from a randomized natural experiment. J. Empir. Leg. Stud. 7:3403–28 [Google Scholar]
  52. Gerber AS, Green DP. 2012. Field Experiments: Design Analysis and Interpretation New York: Norton
  53. Gerber AS, Malhotra N. 2008. Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Q. J. Polit. Sci. 3:3313–26 [Google Scholar]
  54. Gertler P. 2004. Do conditional cash transfers improve child health? Evidence from PROGRESA's control randomized experiment. Am. Econ. Rev. 94:2336–41 [Google Scholar]
  55. Gill CJ. 2012. How often do US-based human subjects research studies register on time, and how often do they post their results? A statistical analysis of the ClinicalTrials.gov database. BMJ Open 2:e001186 [Google Scholar]
  56. Goldkamp JS, Gottfredson MR. 1985. Policy Guidelines for Bail: An Experiment in Court Reform Philadelphia: Temple Univ. Press
  57. Gottfredson DC, Najaka SS, Kearley BW, Rocha CM. 2006. Long-term effects of participation in the Baltimore City Drug Treatment Court: results from an experimental study. J. Exp. Criminol. 2:167–98 [Google Scholar]
  58. Green DP, Winik D. 2010. Using random judge assignments to estimate the effects of incarceration and probation on recidivism among drug offenders. Criminology 48:2357–87 [Google Scholar]
  59. Greenberg D, Shroder M. 2004. The Digest of Social Experiments. Washington, DC: Urban Inst, 3rd ed..
  60. Greiner DJ, Pattanayak CW. 2012. Randomized evaluation in legal assistance: What difference does representation (offer and actual use) make?. Yale Law J. 121:82118–214 [Google Scholar]
  61. Greiner DJ, Pattanayak CW, Hennessy J. 2013. The limits of unbundled legal assistance: a randomized study in a Massachusetts district court and prospects for the future. Harvard Law Rev. 126:901–89 [Google Scholar]
  62. Hall M. 2009. Experimental justice: random judicial assignment and the partisan process of Supreme Court review. Am. Polit. Res. 37:2195–226 [Google Scholar]
  63. Hall M. 2010. Randomness reconsidered: modeling random judicial assignment in the U.S. courts of appeals. J. Empir. Leg. Stud. 7:3574–89 [Google Scholar]
  64. Harcourt BE, Ludwig J. 2006. Broken windows: new evidence from New York City and a five-city social experiment. Univ. Chic. Law Rev. 73:1271–320 [Google Scholar]
  65. Harrell A, Cavanagh S, Roman J. 2000. Evaluation of the D.C. Superior Court drug intervention programs. Res. Brief, Nat. Inst. Justice, Washington, DC
  66. Hastings JS, Weinstein JM. 2008. Information, school choice, and academic achievement: evidence from two experiments. Q. J. Econ. 123:41373–414 [Google Scholar]
  67. Haynes LC, Green DP, Gallagher R, John P, Torgerson DJ. 2013. Collection of delinquent fines: an adaptive randomized trial to assess the effectiveness of alternative text messages. J. Polit. Anal. Manag. 32:4718–30 [Google Scholar]
  68. Heuer L, Penrod SD. 1989. Instructing jurors: a field experiment with written and preliminary instructions. Law Hum. Behav. 13:4409–30 [Google Scholar]
  69. Hudgens MG, Halloran ME. 2008. Toward causal inference with interference. J. Am. Stat. Assoc. 103:482832–43 [Google Scholar]
  70. Humphreys M, Sanchez de la Sierra R, van der Windt P. 2013. Fishing, commitment, and communication: a proposal for comprehensive nonbinding research registration. Polit. Anal. 21:11–20 [Google Scholar]
  71. Humphreys M, Weinstein J. 2012. Policing politicians: citizen empowerment and political accountability in Uganda: preliminary analysis. Work. Pap., Dep. Polit. Sci., Columbia Univ./Dep. Polit. Sci., Stanford Univ., Stanford, CA
  72. Imai K, Ratkovic M. 2013. Estimating treatment effect heterogeneity in randomized program evaluation. Ann. Appl. Stat. 7:1443–70 [Google Scholar]
  73. Johnson SL. 1985. Black innocence and the white jury. Mich. Law Rev. 83:7611–708 [Google Scholar]
  74. Kellerman M, Shepsle KA. 2009. Congressional careers, committee assignments, and seniority randomization in the U.S. House of Representatives. Q. J. Polit. Sci. 4:287–101 [Google Scholar]
  75. Killias M, Aebi M, Ribeaud D. 2000. Does community service rehabilitate better than short-term imprisonment? Results of a controlled experiment. Howard J. Crim. Justice 39:140–57 [Google Scholar]
  76. Kleven HJ, Knudsen MB, Kreiner CT, Pedersen S, Saez E. 2011. Unwilling or unable to cheat? Evidence from a randomized tax audit experiment in Denmark. Econometrica 79:3651–92 [Google Scholar]
  77. Leiter B. 2009. The “top 20” legal journals. Brian Leiter's Law School Reports Blog Mar:19 [Google Scholar]
  78. Levitt SD, List JA. 2009. Field experiments in economics: the past, present, and future. Eur. Econ. Rev. 53:11–18 [Google Scholar]
  79. Lin W. 2013. Agnostic notes on regression adjustments to experimental data: reexamining Freedman's critique. Ann. Appl. Stat. 7:1295–318 [Google Scholar]
  80. Listokin Y. 2010. The meaning of contractual silence: a field experiment. J. Leg. Anal. 3:2397–416 [Google Scholar]
  81. Loewen PJ, Koop R, Settle JE, Fowler JH. 2013. A natural experiment in proposal power and electoral success. Am. J. Polit. Sci. 58:1189–96 [Google Scholar]
  82. Ludwig J, Kling JR. 2007. Is crime contagious?. J. Law Econ. 50:3491–518 [Google Scholar]
  83. Ludwig J, Kling JR, Mullainathan S. 2011. Mechanism experiments and policy evaluations. J. Econ. Perspect. 25:317–38 [Google Scholar]
  84. Ludwig J, Liebman JB, Kling JR, Duncan GJ, Katz LF. et al. 2008. What can we learn about neighborhood effects from the Moving to Opportunity experiment?. Am. J. Sociol. 114:1144–88 [Google Scholar]
  85. Manski CF. 1989. Anatomy of the selection problem. J. Hum. Resour. 24:3343–60 [Google Scholar]
  86. Manski CF. 1995. Identification Problems in the Social Sciences Cambridge, MA: Harvard Univ. Press
  87. Marceau JF. 2012. Embracing a new era of ineffective assistance of counsel. Univ. Pa. J. Const. Law 14:51161–218 [Google Scholar]
  88. Martin SE, Annan S, Forst B. 1993. The special deterrent effects of a jail sanction on first-time drunk drivers: a quasi-experimental study. Accid. Anal. Prev. 25:5561–68 [Google Scholar]
  89. McGinnis JO. 2011. Laws for learning in an age of acceleration. William Mary Law Rev. 53:2305–50 [Google Scholar]
  90. Miguel E, Camerer C, Casey K, Cohen J, Esterling KM. et al. 2014. Promoting transparency in social science research. Science 343:616630–31 [Google Scholar]
  91. Miratrix L, Sekhon J, Yu B. 2013. Adjusting treatment effect estimates by post-stratification in randomized experiments. J. R. Stat. Soc. Ser. B 75:2369–96 [Google Scholar]
  92. Moore U, Callahan CC. 1943. Law and learning theory: a study in legal control. Yale Law J. 53:11–136 [Google Scholar]
  93. Nagel SS. 1972. Effects of alternative types of counsel on criminal procedure treatment. Indiana Law J. 48:3404–26 [Google Scholar]
  94. Newhouse JP, Insur. Exp. Group 1993. Free for All? Lessons from the RAND Health Insurance Experiment. Cambridge, MA: Harvard Univ. Press
  95. Olken BA. 2010. Direct democracy and local public goods: evidence from a field experiment in Indonesia. Am. Polit. Sci. Rev. 104:2243–67 [Google Scholar]
  96. Orr LL. 1999. Social Experiments: Evaluating Public Programs with Experimental Methods Thousand Oaks, CA: Sage
  97. Ouellette LL. 2013. Patent experimentalism Work. Pap., Yale Law Sch., New Haven, CT
  98. Partridge A, Lind A. 1983. A reevaluation of the Civil Appeals Management Plan FJC-R-83-2, Fed. Judic. Cent., Washington, DC
  99. Peisakhin L. 2012. Transparency and corruption: evidence from India. J. Law Econ. 55:1129–49 [Google Scholar]
  100. Pleasence P, Balmer NJ. 2007. Changing fortunes: results from a randomized trial of the offer of debt advice in England and Wales. J. Empir. Leg. Stud. 4:3651–73 [Google Scholar]
  101. Rosenblum M, van der Laan MJ. 2011. Optimizing randomized trial designs to distinguish which subpopulations benefit from treatment. Biometrika 98:4845–60 [Google Scholar]
  102. Roth AE. 1993. On the early history of experimental economics. J. Hist. Econ. Thought 15:2184–209 [Google Scholar]
  103. Roth AE. 1995. Introduction to experimental economics. The Handbook of Experimental Economics JH Kagel, AE Roth 3–109 Princeton, NJ: Princeton Univ. Press [Google Scholar]
  104. Samaha AM. 2009. Randomization in adjudication. William Mary Law Rev. 51:11–86 [Google Scholar]
  105. Schulz KF, Altman DG, Moher D. CONSORT Group 2010. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Ann. Int. Med. 152:11726–32 [Google Scholar]
  106. SEC (US Secur. Exch. Comm.) 2007. Economic analysis of the short sale price restrictions under the regulation SHO pilot Rep., Off. Econ. Anal., SEC, Washington, DC
  107. Selbin J, Charn J, Alfieri A, Wizner S. 2012. Service delivery, resource allocation and access to justice: Greiner and Pattanayak and the research imperative. Yale Law J. Online 122:45–67 [Google Scholar]
  108. Seron C, Van Ryzin G, Frankel M, Kovath J. 2001. The impact of legal counsel on outcomes for poor tenants in New York City's housing court: results of a randomized experiment. Law Soc. Rev. 35:2419–32 [Google Scholar]
  109. Simonsohn U, Nelson LD, Simmons JP. 2014. P-curve: a key to the file drawer. J. Exp. Psychol. Gen. 143:2534–47 [Google Scholar]
  110. Slemrod J, Blumenthal M, Christian C. 2001. Taxpayer response to an increased probability of audit: evidence from a controlled experiment in Minnesota. J. Public Econ. 79:3455–83 [Google Scholar]
  111. Sunstein CR, Schkade D, Ellman LM, Sawicki A. 2006. Are Judges Political? An Empirical Analysis of the Federal Judiciary Washington, DC: Brookings Inst.
  112. Tingley D, Chilton A. 2013. Why the study of international law needs experiments. Columbia J. Transnatl. Law 34:167–148 [Google Scholar]
  113. Titiunik R. 2013. Drawing your senator from a jar: term length and legislative behavior Work. Pap., Dep. Polit. Sci., Univ. Mich., Ann Arbor
  114. Torgerson DJ, Torgerson CJ. 2008. Designing Randomised Trials in Health, Education and the Social Sciences: An Introduction Basingstoke, UK: Palgrave Macmillan
  115. van der Laan MJ, Rose S. 2011. Targeted Learning: Causal Inference for Observational and Experimental Data New York: Springer
  116. Waldfogel J. 1998. Reconciling asymmetric information and divergent expectations theories of litigation. J. Law Econ. 41:2451–76 [Google Scholar]
  117. Walker L. 1988. Perfecting federal civil rules: a proposal for restricted field experiments. Law Contemp. Probl. 51:367–85 [Google Scholar]
  118. Webb EJ, Campbell DT, Schwartz RD, Sechrest L. 1999. Unobtrusive Measures Thousand Oaks, CA: Sage
  119. Wood WJ. 1916. Necessity for public defender established by statistics. J. Am. Inst. Crim. Law Criminol. 7:2230–44 [Google Scholar]
  120. Zeiler K. 2010. Cautions on the use of economics experiments in law. J. Inst. Theor. Econ. 166:1178–93 [Google Scholar]
/content/journals/10.1146/annurev-lawsocsci-110413-030936
Loading
/content/journals/10.1146/annurev-lawsocsci-110413-030936
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error