1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Criminology
  4. Volume 2, 2019
  5. Article

Annual Review of Criminology

Volume 2, 2019

Methodological Challenges and Opportunities in Testing for Racial Discrimination in Policing

Abstract

A large body of empirical research exists that attempts to determine whether or not police discriminate on the basis of race. We investigate whether the methods used typically produce valid inferences. We find that they often most likely do not and that results may diverge from reality in either direction, indicating discrimination when it is not present or alternatively indicating a lack of discrimination when it is in fact present. The reason for this is that tests make assumptions about police behavior that are often implausible. Because of this, the simplest forms of benchmark and outcome tests should not be used, although the problem is more general. We discuss several possible ways to improve inferences about the absence or presence of discrimination, such as employing matching or weighting techniques and using novel, computationally intensive methods.

Keyword(s): discrimination, inference, methods, police, race, racial bias

[Erratum, Closure]

An erratum has been published for this article:
Erratum: Methodological Challenges and Opportunities in Testing for Racial Discrimination in Policing
Loading

Article metrics loading...

/content/journals/10.1146/annurev-criminol-011518-024731
2019-01-13
2024-04-24
Loading full text...

Full text loading...

/deliver/fulltext/criminol/2/1/annurev-criminol-011518-024731.html?itemId=/content/journals/10.1146/annurev-criminol-011518-024731&mimeType=html&fmt=ahah

Literature Cited

  1. Alpert GP, Dunham RG, Smith MR 2007. Investigating racial profiling by the Miami-Dade Police Department: a multimethod approach. Criminol. Public Policy 6:125–55
     [Google Scholar]
  2. Alpert GP, Smith MR, Dunham RG 2004. Toward a better benchmark: assessing the utility of not-at-fault traffic crash data in racial profiling research. Justice Res. Policy 6:143–69
     [Google Scholar]
  3. Antonovics K, Knight BG 2009. A new look at racial profiling: evidence from the Boston Police Department. Rev. Econ. Stat. 91:1163–77
     [Google Scholar]
  4. Anwar S, Fang H 2006. An alternative test of racial prejudice in motor vehicle searches: theory and evidence. Am. Econ. Rev. 96:1127–51
     [Google Scholar]
  5. Arrow KJ 1973. The theory of discrimination. Discrimination in Labor Markets O Ashenfelter, A Rees 3–33 Princeton: Princeton Univ. Press
     [Google Scholar]
  6. Ayres I 2002. Outcome tests of racial disparities in police practices. Justice Res. Policy 4:1–2131–42
     [Google Scholar]
  7. Batton C, Kadleck C 2004. Theoretical and methodological issues in racial profiling research. Police Q 7:130–64
     [Google Scholar]
  8. Becker GS 1957. The Economics of Discrimination Chicago: Univ. Chicago Press
  9. Becker GS 1993. Nobel lecture: the economic way of looking at behavior. J. Political Econ. 101:3385–409
     [Google Scholar]
  10. Beckett K, Nyrop K, Pfingst L 2006. Race, drugs, and policing: understanding disparities in drug delivery arrests. Criminology 44:1105–37
     [Google Scholar]
  11. Beckett K, Nyrop K, Pfingst L, Bowen M 2005. Drug use, drug possession arrests, and the question of race: lessons from Seattle. Soc. Probl. 52:3419–41
     [Google Scholar]
  12. Bittner E 1970. The Functions of the Police in Modern Society: A Review of Background Factors, Current Practices, and Possible Role Models Chevy Chase, MD: Natl. Inst. Ment. Health Cent. Stud. Crime Delinquency
     [Google Scholar]
  13. Chouldechova A 2017. Fair prediction with disparate impact: a study of bias in recidivism prediction instruments. Big Data 5:2153–63
     [Google Scholar]
  14. City of Cleburne Tex. v. Cleburne Living Ctr., 473 U.S. 432 1985.
  15. Close BR, Mason PL 2007. Searching for efficient enforcement: officer characteristics and racially biased policing. Rev. Law Econ. 3:2263–321
     [Google Scholar]
  16. Correll J, Hudson SM, Guillermo S, Ma DS 2014. The police officer's dilemma: a decade of research on racial bias in the decision to shoot. Soc. Personal. Psychol. Compass 8:5201–13
     [Google Scholar]
  17. Correll J, Park B, Judd CM, Wittenbrink B, Sadler MS, Keesee T 2007. Across the thin blue line: police officers and racial bias in the decision to shoot. J. Pers. Soc. Psychol. 92:61006–23
     [Google Scholar]
  18. Coviello D, Persico N 2015. An economic analysis of black-white disparities in the New York Police Department's stop-and-frisk program. J. Leg. Stud. 44:2315–60
     [Google Scholar]
  19. Crutchfield RD, Skinner ML, Haggerty KP, McGlynn A, Catalano RF 2012. Racial disparity in police contacts. Race Justice 2:3179–202
     [Google Scholar]
  20. Dharmapala D, Ross SL 2004. Racial bias in motor vehicle searches: additional theory and evidence. Contrib. Econ. Anal. Policy 3:11310
     [Google Scholar]
  21. DiPrete TA, Eirich GM 2006. Cumulative advantage as a mechanism for inequality: a review of theoretical and empirical developments. Annu. Rev. Sociol. 32:1271–97
     [Google Scholar]
  22. Eberhardt JL, Goff PA, Purdie VJ, Davies PG 2004. Seeing black: race, crime, and visual processing. J. Pers. Soc. Psychol. 87:6876–93
     [Google Scholar]
  23. Elliott DS 1995. Lies, damn lies and arrest statistics Paper presented at the 47th Annual Meeting of the American Society of Criminology Boston: Nov. 15–18
  24. Elliott DS, Ageton SS 1980. Reconciling race and class differences in self-reported and official estimates of delinquency. Am. Sociol. Rev. 45:195–110
     [Google Scholar]
  25. Engel RS 2008. A critique of the “outcome test” in racial profiling research. Justice Q 25:11–36
     [Google Scholar]
  26. Engel RS, Calnon JM 2004. Comparing benchmark methodologies for police-citizen contacts: traffic stop data collection for the Pennsylvania State Police. Police Q 7:197–125
     [Google Scholar]
  27. Engel RS, Smith MR, Cullen FT 2012. Race, place, and drug enforcement. Criminol. Public Policy 11:4603–35
     [Google Scholar]
  28. Engel RS, Tillyer R 2008. Searching for equilibrium: the tenuous nature of the outcome test. Justice Q 25:154–71
     [Google Scholar]
  29. Fagan J, Braga AA, Brunson RK, Pattavina A 2016. Stops and stares: street stops, surveillance, and race in the new policing mental health, the law, and the urban environment. Fordham Urban Law J 43:539–614
     [Google Scholar]
  30. Fagan J, Davies G 2000. Street stops and broken windows: terry, race, and disorder in New York City. Fordham Urban Law J 28:457–504
     [Google Scholar]
  31. Feller A, Pierson E, Corbett-Davies S, Goel S 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It's actually not that clear. Washington Post Oct 17. https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/17/can-an-algorithm-be-racist-our-analysis-is-more-cautious-than-propublicas/?utm_term=.31dcded0ef69
  32. Fryer RG 2016. An empirical analysis of racial differences in police use of force NBER Work Pap. 22399
  33. Geller A, Fagan J 2010. Pot as pretext: marijuana, race, and the new disorder in New York City street policing. J. Empir. Leg. Stud. 7:4591–633
     [Google Scholar]
  34. Gelman A, Fagan J, Kiss A 2007. An analysis of the New York City Police Department's “stop-and-frisk” policy in the context of claims of racial bias. J. Am. Stat. Assoc. 102:479813–23
     [Google Scholar]
  35. Goel S, Rao JM, Shroff R 2016. Precinct or prejudice? Understanding racial disparities in New York City's stop-and-frisk policy. Ann. Appl. Stat. 10:1365–94
     [Google Scholar]
  36. Goff PA, Kahn KB 2012. Racial bias in policing: why we know less than we should. Soc. Issues Policy Rev. 6:1177–210
     [Google Scholar]
  37. Greiner DJ, Rubin DB 2011. Causal effects of perceived immutable characteristics. Rev. Econ. Stat. 93:3775–85
     [Google Scholar]
  38. Grogger J, Ridgeway G 2006. Testing for racial profiling in traffic stops from behind a veil of darkness. J. Am. Stat. Assoc. 101:475878–87
     [Google Scholar]
  39. Hindelang MJ 1978. Race and involvement in common law personal crimes. Am. Sociol. Rev. 43:193–109
     [Google Scholar]
  40. Hindelang MJ, Hirschi T, Weis JG 1979. Correlates of delinquency: the illusion of discrepancy between self-report and official measures. Am. Sociol. Rev. 44:6995–1014
     [Google Scholar]
  41. Hindelang MJ, Hirschi T, Weis JG 1981. Measuring Delinquency Beverly Hills, CA: SAGE
  42. Ho DE, Imai K, King G, Stuart EA 2007. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Anal 15:3199–236
     [Google Scholar]
  43. Huizinga D, Thornberry T, Knight K, Lovegrove P 2007. Disproportionate minority contact in the juvenile justice system: a study of differential minority arrest/referral to court in three cities Natl. Crim. Justice Ref. Serv. Rep. 219743, Off. Juv. Justice Delinquency Prev. Washington, DC:
  44. Imbens GW 2004. Nonparametric estimation of average treatment effects under exogeneity: a review. Rev. Econ. Stat. 86:14–29
     [Google Scholar]
  45. James L, James SM, Vila BJ 2016. The reverse racism effect. Criminol. Public Policy 15:2457–79
     [Google Scholar]
  46. King G 1997. A Solution to the Ecological Inference Problem: Reconstructing Individual Behavior from Aggregate Data Princeton: Princeton Univ. Press
  47. King G, Nielsen R 2016. Why propensity scores should not be used for matching Work. Pap., Inst. Quant. Soc. Sci., Harvard Univ. Cambridge, MA:
  48. King G, Nielsen R, Coberley C, Pope JE, Wells A 2011. Comparative effectiveness of matching methods for causal inference Work. Pap., Inst. Quant. Soc. Sci., Harvard Univ. Cambridge, MA:
  49. Kirk DS 2008. The neighborhood context of racial and ethnic disparities in arrest. Demography 45:155–77
     [Google Scholar]
  50. Kleinberg J, Mullainathan S, Raghavan M 2016. Inherent trade-offs in the fair determination of risk scores. ArXiv:1609.05807[cs.LG]
  51. Klinger DA 1997. Negotiating order in patrol work: an ecological theory of police response to deviance. Criminology 35:2277–306
     [Google Scholar]
  52. Knowles J, Persico N, Todd P 2001. Racial bias in motor vehicle searches: theory and evidence. J. Political Econ. 109:1203–29
     [Google Scholar]
  53. Kohler-Hausmann I 2018. Eddie Murphy and the dangers of counterfactual causal thinking about detecting racial discrimination Work Pap., Yale Univ. New Haven, CT:
  54. Lamberth JC 1994. Revised Statistical Analysis of the Incidence of Police Stops and Arrests of Black Drivers/travelers on the New Jersey Turnpike Dep. Psychol. Rep., Temple Univ. Philadelphia, PA:
  55. Lamberth JC 2003. Racial Profiling Data Analysis Study: Final Report for the San Antonio Police Department West Chester, PA: Lamberth Consult.
  56. Legewie J 2016. Racial profiling and use of force in police stops: how local events trigger periods of increased discrimination. Am. J. Sociol. 122:2379–424
     [Google Scholar]
  57. Levchak PJ 2017. Do precinct characteristics influence stop-and-frisk in New York City? A multi-level analysis of post-stop outcomes. Justice Q 34:3377–406
     [Google Scholar]
  58. Liberman AM, Kirk DS, Kim K 2014. Labeling effects of first juvenile arrests: secondary deviance and secondary sanctioning. Criminology 52:3345–70
     [Google Scholar]
  59. Lin DY, Psaty BM, Kronmal RA 1998. Assessing the sensitivity of regression results to unmeasured confounders in observational studies. Biometrics 54:3948–63
     [Google Scholar]
  60. MacDonald J, Braga AA 2018. Did post-Floyd et al. reforms reduce racial disparities in NYPD stop, question, and frisk practices? An exploratory analysis using external and internal benchmarks. Justice Q In press. https://doi.org/10.1080/07418825.2018.1427278
    [Crossref]
  61. McConnell, Scheidegger 2001. Race and speeding citations: comparing speeding citations issued by air traffic officers with those issued by ground traffic officers Paper presented at the 38th Annual Meeting of the Academy of Criminial Justice Sciences Washington, DC:
  62. Moore WF 2017. Street checks and crime in a statistical perspective: tracking criminal charges against persons checked MA Thesis, Univ. Cambridge Cambridge, UK:
  63. Morgan SL, Winship C 2015. Counterfactuals and Causal Inference Cambridge, UK: Cambridge Univ. Press. , 2nd ed..
  64. Natl. Acad. Sci. 2018. Proactive Policing: Effects on Crime and Communities Washington, DC: Natl. Acad. Press
  65. Pager D, Shepherd H 2008. The sociology of discrimination: racial discrimination in employment, housing, credit, and consumer markets. Annu. Rev. Sociol. 34:181–209
     [Google Scholar]
  66. Persico N, Todd P 2006. Generalising the hit rates test for racial bias in law enforcement, with an application to vehicle searches in Wichita. Econ. J. 116:515F351–67
     [Google Scholar]
  67. Phelps ES 1972. The statistical theory of racism and sexism. Am. Econ. Rev. 62:4659–61
     [Google Scholar]
  68. Pierson E, Corbett-Davies S, Goel S 2017.a Fast threshold tests for detecting discrimination. AISTATS. In press. ArXiv:1702.08536[stat.ML]
  69. Pierson E, Simoiu C, Overgoor J, Corbett-Davies S, Ramachandran V et al. 2017.b A large-scale analysis of racial disparities in police stops across the United States. ArXiv:1706.05678[stat.AP]
  70. Piquero AR 2009. Finding the right balance between data, research, findings, and policy in racial profiling. Criminol. Public Policy 8:2371–79
     [Google Scholar]
  71. Plant EA, Peruche BM 2005. The consequences of race for police officers’ responses to criminal suspects. Psychol. Sci. 16:3180–83
     [Google Scholar]
  72. Ridgeway G 2006. Assessing the effect of race bias in post-traffic stop outcomes using propensity scores. J. Quant. Criminol. 22:11–29
     [Google Scholar]
  73. Ridgeway G 2007. Analysis of Racial Disparities in the New York Police Department's Stop, Question, and Frisk Practices Santa Monica, CA: RAND
  74. Ridgeway G 2009. Cincinnati Police Department Traffic Stops: Applying RAND's Framework to Analyze Racial Disparities Santa Monica, CA: RAND
  75. Ridgeway G 2017. Stop-and-frisk is essential…and requires restraint. J. Policy Anal. Manag. 36:3683–89
     [Google Scholar]
  76. Ridgeway G, MacDonald JM 2009. Doubly robust internal benchmarking and false discovery rates for detecting racial bias in police stops. J. Am. Stat. Assoc. 104:486661–68
     [Google Scholar]
  77. Ridgeway G, MacDonald JM 2010. Methods for assessing racially biased policing. Race, Ethnicity, and Policing SK Rice MD White:180–204 New York: NYU Press
     [Google Scholar]
  78. Robinson WS 1950. Ecological correlations and the behavior of individuals. Am. Sociol. Rev. 15:351–57
     [Google Scholar]
  79. Rojek J, Rosenfeld R, Decker S 2012. Policing race: the racial stratification of searches in police traffic stops. Criminology 50:4993–1024
     [Google Scholar]
  80. Rosenbaum PR 2002. Observational Studies New York: Springer. , 2nd ed..
  81. Sampson RJ 2012. Great American City: Chicago and the Enduring Neighborhood Effect Chicago: Univ. Chicago Press
  82. Sanga S 2009. Reconsidering racial bias in motor vehicle searches: theory and evidence. J. Political Econ. 117:61155–59
     [Google Scholar]
  83. Sen M, Wasow O 2016. Race as a bundle of sticks: designs that estimate effects of seemingly immutable characteristics. Annu. Rev. Political Sci. 19:1499–522
     [Google Scholar]
  84. Simoiu C, Corbett-Davies S, Goel S 2017. The problem of infra-marginality in outcome tests for discrimination. Ann. Appl. Stat. 11:31193–216
     [Google Scholar]
  85. Smith DA 1986. The neighborhood context of police behavior. Crime Justice 8:313–41
     [Google Scholar]
  86. Spence M 1974. Market Signaling: Informational Transfer in Hiring and Related Screening Processes Cambridge, MA: Harvard Univ. Press
  87. Starr SB 2015. Explaining race gaps in policing: normative and empirical challenges Univ. Mich. Law Econ. Res. Pap. 15-003 Ann Arbor, MI:
  88. Starr SB 2016. Testing racial profiling: empirical assessment of disparate treatment by police. Univ. Chic. Leg. Forum. 2016:485–531
     [Google Scholar]
  89. Tapia M 2011. Gang membership and race as risk factors for juvenile arrest. J. Res. Crime Delinquency 48:3364–95
     [Google Scholar]
  90. Tillyer R 2014. Opening the black box of officer decision-making: an examination of race, criminal history, and discretionary searches. Justice Q 31:6961–85
     [Google Scholar]
  91. Tillyer R, Engel RS, Cherkauskas JC 2010. Best practices in vehicle stop data collection and analysis. Polic. Int. J. 33:169–92
     [Google Scholar]
  92. Tillyer R, Klahm CF, Engel RS 2012. The discretion to search: a multilevel examination of driver demographics and officer characteristics. J. Contemp. Crim. Justice 28:2184–205
     [Google Scholar]
  93. VanderWeele TJ 2008. Sensitivity analysis: distributional assumptions and confounding assumptions. Biometrics 64:2645–49
     [Google Scholar]
  94. VanderWeele TJ 2015. Explanation in Causal Inference: Methods for Mediation and Interaction Oxford, UK: Oxford Univ. Press
  95. Voigt R, Camp NP, Prabhakaran V, Hamilton WL, Hetey RC et al. 2017. Language from police body camera footage shows racial disparities in officer respect. PNAS 114:256521–26
     [Google Scholar]
  96. Walker S 2001. Searching for the denominator: problems with police traffic stop data and an early warning system solution. Justice Res. Policy 3:163–95
     [Google Scholar]
  97. Weisburd D, Telep CW, Lawton BA 2014. Could innovations in policing have contributed to the New York City crime drop even in a period of declining police strength?: the case of stop, question and frisk as a hot spots policing strategy. Justice Q 31:1129–53
     [Google Scholar]
  98. Werthman C, Piliavin I 1967. Gang members and the police. The Police: Six Sociological Essays DJ Bordua New York: John Wiley and Sons
     [Google Scholar]
  99. Wolfgang ME, Figlio RM, Sellin T 1972. Delinquency in a Birth Cohort Chicago: Univ. Chicago Press
  100. Worden RE, McLean SJ, Wheeler AP 2012. Testing for racial profiling with the veil-of-darkness method. Police Q 15:192–111
     [Google Scholar]
  101. Zimring FE 2011. The City That Became Safe: New York's Lessons for Urban Crime and Its Control Oxford, UK: Oxford Univ. Press
/content/journals/10.1146/annurev-criminol-011518-024731
Loading
Methodological Challenges and Opportunities in Testing for Racial Discrimination in Policing
Annual Review of Criminology 2, 73 (2019); https://doi.org/10.1146/annurev-criminol-011518-024731
/content/journals/10.1146/annurev-criminol-011518-024731
/content/journals/10.1146/annurev-criminol-011518-024731
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error