1932

Abstract

What role can statistics play in assessing the patterns of lethal violence in conflict? This article highlights the evolution of statistical applications in assessing lethal violence, from the presentation of data in the Nuremberg trials to current questions around machine learning and training data. We present examples from work conducted by our organization, the Human Rights Data Analysis Group, and others, primarily researching killings in the context of civil wars and international conflict. The primary challenge we encounter in this work is the question of whether observed patterns of violence represent the true underlying pattern or are a reflection of reports of violence, which are subject to many sources of bias. This is where we find the foundations of twentieth-century statistics to be most important: Is this sample representative? What methods are best suited to reduce the bias in nonprobability samples? These questions lead us to the approaches presented here: multiple systems estimation, surveys, complete data, and the question of bias within training data for machine learning models. We close with memories of Steve Fienberg's influence on these questions and on us personally. “It's all inference,” he told us, and that insight informs our concerns about bias in data used to create historical memory and advance justice in the wake of mass violence.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-statistics-030718-105222
2019-03-07
2024-10-11
Loading full text...

Full text loading...

/deliver/fulltext/statistics/6/1/annurev-statistics-030718-105222.html?itemId=/content/journals/10.1146/annurev-statistics-030718-105222&mimeType=html&fmt=ahah

Literature Cited

  1. Asher J, Banks DL, Scheuren F 2008. Statistical Methods for Human Rights New York: Springer
    [Google Scholar]
  2. Bales K, Hesketh O, Silverman B. 2015. Modern slavery in the UK: How many victims?. Significance 12:16–21
    [Google Scholar]
  3. Ball PD. 1996. Who did what to whom? Planning and implementing a large scale human rights data project Rep., AAAS, Washington, DC
    [Google Scholar]
  4. Ball P. 2000. The Guatemalan Commission for Historical Clarification: generating analytical reports, inter-sample analysis. See Ball et al. 2000 25976
    [Google Scholar]
  5. Ball P, Asher J, Sulmont D, Manrique D. 2003. How many Peruvians have died Rep., AAAS, Washington, DC
    [Google Scholar]
  6. Ball P, Betts W, Scheuren F, Dudukovich J, Asher J. 2002. Killings and refugee flow in Kosovo March-June 1999: a report to the International Criminal Tribunal for the former Yugoslavia Rep., AAAS, Washington, DC
    [Google Scholar]
  7. Ball P, Cifuentes R, Dueck J, Gregory R, Salcedo D, Saldarriaga C. 1994. A definition of database design standards for human rights agenciesRep., AAAS, Washington, DC
    [Google Scholar]
  8. Ball P, Kobrak P, Spirer HF 1999. State Violence in Guatemala, 1960–1996: A Quantitative Reflection Washington, DC: AAAS
    [Google Scholar]
  9. Ball P, Price M. 2018. The statistics of genocide. CHANCE 31:38–45
    [Google Scholar]
  10. Ball P, Spirer HF. 2000. The Haitian National Commission for Truth and Justice: collecting information, data processing, database representation, and generating analytical reports. See Ball et al. 2000 2740
    [Google Scholar]
  11. Ball P, Spirer HF, Spirer L 2000. Making the Case: Investigating Large Scale Human Rights Violations Using Information Systems and Data Analysis Washington, DC: AAAS
    [Google Scholar]
  12. Ball P, Tabeau E, Verwimp P. 2007. The Bosnian book of the dead: assessment of the database (full report) Rep., Households in Conflict Network, Inst. Dev. Stud., Univ. Sussex, Brighton, UK
    [Google Scholar]
  13. Banks D, Couzens L, Blanton C, Cribb D. 2015. Arrest-Related Deaths program assessment Tech. Rep. NCJ 248543, Bur. Justice Stat., US Dep. Justice, Washington, DC
    [Google Scholar]
  14. Betts WS. 2016. Evidence by the numbers: using statistical analyses as evidence of international atrocity crimes. USFL Rev 50:357
    [Google Scholar]
  15. Bilenko M, Mooney RJ. 2003. Adaptive duplicate detection using learnable string similarity measures. Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining39–48 New York: ACM
    [Google Scholar]
  16. Bird SM, King R. 2017. Multiple systems estimation (or capture-recapture estimation) to inform public policy. Annu. Rev. Stat. Appl 5:95–118
    [Google Scholar]
  17. Bishop YM, Fienberg SE, Holland PW. 1974. Discrete Multivariate Analysis: Theory and Practice. Thousand Oaks, CA: SAGE
    [Google Scholar]
  18. Breiman L. 2001. Random forests. Mach. Learn 45:5–32
    [Google Scholar]
  19. Brunborg H. 2001. Contribution of statistical analysis to the investigations of the international criminal tribunals. Stat. J. UN Econ. Comm. Europe 18:227–38
    [Google Scholar]
  20. Brunborg H, Lyngstad TH, Urdal H. 2003. Accounting for genocide: how many were killed in Srebrenica?. Eur. J. Popul. Rev. Eur. Démogr 19:229–48
    [Google Scholar]
  21. Chao A. 1998. Estimating animal abundance with capture frequency data. J. Wildl. Manag 52:295–300
    [Google Scholar]
  22. Checchi F, Roberts L. 2005. Interpreting and using mortality data in humanitarian emergencies Netw. Pap. 52, Humanit. Pract. Netw., Overseas Dev. Inst., London, UK
    [Google Scholar]
  23. Chen B, Shrivastava A, Steorts RC. 2018. Unique entity estimation with application to the Syrian conflict. Ann. Appl. Stat 12:1039–67
    [Google Scholar]
  24. Chen T, Guestrin C. 2016. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining785–94 New York: ACM
    [Google Scholar]
  25. Christen P 2012. Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection New York: Springer
    [Google Scholar]
  26. Cibelli K, Hoover A, Krüger J. 2009. Descriptive statistics from statements to the Liberian Truth and Reconciliation Commission: a report by the Human Rights Data Analysis Group at Benetech and annex to the Final Report of the Truth and Reconciliation Commission of Liberia Rep., Benetech, Palo Alto, CA
    [Google Scholar]
  27. Commission for Historical Clarification 1999. Guatemala: Memory of Silence Guatemala City, Guatemala: Commission for Historical Clarification
    [Google Scholar]
  28. Conibere R, Asher J, Cibelli K, Dudukovich J, Kaplan R, Ball P. 2004. Statistical appendix to the report of the Truth and Reconciliation Commission of Sierra Leone Rep., Benetech Hum. Rights Data Anal. Group, Palo Alto, CA
    [Google Scholar]
  29. Conti-Cook CH. 2016. Defending the public: police accountability in the courtroom. Seton Hall Law Rev 46:3
    [Google Scholar]
  30. Conti-Cook CH. 2017. Open data policing. Georgetown Law J 106:1–23
    [Google Scholar]
  31. Contreras JC, Ytajashi AP, Carrillo FR. 2014. Hatun Willakuy: Abbreviated Version of the Final Report of the Truth and Reconciliation Commission. Lima, Peru: Transfer Comm. Truth Reconcil. Comm. Peru
    [Google Scholar]
  32. Cormack R. 1992. Interval estimation for mark-recapture studies of closed populations. Biometrics 48:567–76
    [Google Scholar]
  33. Crawford K. 2017. The trouble with bias Presented at Thirty-First Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, Dec. 5
    [Google Scholar]
  34. Cruz M, Cibelli K, Dudukovic J. 2003. Preliminary statistical analysis of AVCRP and DDS documents—a report to Human Rights Watch about Chad under the government of Hissène Habré Rep., Benetech, Palo Alto, CA
    [Google Scholar]
  35. Dassin J, 1986. Torture in Brazil: A Shocking Report on the Pervasive Use of Torture by Brazilian Military Governments, 1964–1979, Secretly Prepared by the Archdiocese of São Paulo. Austin: Univ. Texas Press
    [Google Scholar]
  36. DeGroot MH, Fienberg SE, Kadane JB, eds. 1994. Statistics and the Law New York: Wiley
    [Google Scholar]
  37. Dunn HL. 1946. Record linkage. Am. J. Public Health Nations Health 36:1412–16
    [Google Scholar]
  38. Early JD. 1974. Revision of Ladino and Maya census populations of Guatemala, 1950 and 1964. Demography 11:105–17
    [Google Scholar]
  39. Evans S, Mejia R, Price M 2018. Special issue on human rights. CHANCE 31:1
    [Google Scholar]
  40. Fellegi IP, Sunter AB. 1969. A theory for record linkage. J. Am. Stat. Assoc 64:1183–210
    [Google Scholar]
  41. Fienberg SE, Manrique-Vallier D. 2009. Integrated methodology for multiple systems estimation and record linkage using a missing data formulation. AStA Adv. Stat. Anal 93:49–60
    [Google Scholar]
  42. Fienberg SE, Tanur JM. 1996. Reconsidering the fundamental contributions of Fisher and Neyman on experimentation and sampling. Int. Stat. Rev 15:237–53
    [Google Scholar]
  43. Fisher RA. 1922. On the mathematical foundations of theoretical statistics. Phil. Trans. R. Soc. Lond. A 222:309–68
    [Google Scholar]
  44. Freund Y, Mason L. 1999. The alternating decision tree learning algorithm. Proceedings of the Sixteenth International Conference on Machine Learning124–33 San Francisco: Morgan Kaufmann
    [Google Scholar]
  45. Gray MW Marek S. 2008. The statistics of genocide. Statistical Methods for Human Rights DL Banks, FJ Scheuren, J Asher37–50 New York: Springer
    [Google Scholar]
  46. Guberek T, Guzman D, Silva R, Cibelli K, Asher J, 2006. Truth and myth in Sierra Leone: an empirical analysis of the conflict, 1991–2000 Rep., Benetech, Palo Alto, CA
    [Google Scholar]
  47. Guzmán D, Guberek T, Price M. 2012. Unobserved union violence: statistical estimates of the total number of trade unionists killed in Colombia, 1999–2008 Rep., Benetech, Palo Alto, CA
    [Google Scholar]
  48. Guzmán D, Guberek T, Shapiro G, Zador P. 2009. Studying millions of rescued documents: sampling plan at the Guatemalan National Police Archive (GNPA). Proceedings of the Joint Statistical Meeting, Survey Research Methods Section—JSM 20093250–64 Alexandria, VA: ASA
    [Google Scholar]
  49. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH. 2009. The WEKA Data mining software: an update. SIGKDD Explor. Newsl 11:10–18
    [Google Scholar]
  50. Heckathorn DD. 1997. Respondent-driven sampling: a new approach to the study of hidden populations. Soc. Probl 44:174–99
    [Google Scholar]
  51. Herzog TH, Scheuren F, Winkler WE. 2010. Record linkage. Wiley Interdiscip. Rev. Comput. Stat 2:535–43
    [Google Scholar]
  52. Heuveline P. 1998. ‘Between one and three million’: towards the demographic reconstruction of a decade of Cambodian history (1970–79). Popul. Stud 52:49–65
    [Google Scholar]
  53. Hoover Green A. 2010. Learning the hard way at the ICTY: statistical evidence of human rights violations in an adversarial information environment. Collective Violence and International Criminal Justice: An Interdisciplinary Approach A Smeulers325–54 Antwerp, Belg.: Intersentia
    [Google Scholar]
  54. Howland T. 2008. How El Rescate, a small nongovernmental organization, contributed to the transformation of the human rights situation in El Salvador. Hum. Rights Q 30:703–57
    [Google Scholar]
  55. Huggins R. 2001. A note on the difficulties associated with the analysis of capture–recapture experiments with heterogeneous capture probabilities. Stat. Probab. Lett 54:147–52
    [Google Scholar]
  56. Human Rights Office, Archdiocese of Guatemala 1999. Guatemala: Never Again! Maryknoll, NY: Orbis
    [Google Scholar]
  57. Iacopino V. 1999. War crimes in Kosovo: a population-based assessment of human rights violations against Kosovar Albanians Rep., Physicians Hum. Rights, Boston and Washington, DC
    [Google Scholar]
  58. Isaac W, Lum K. 2018. Setting the record straight on predictive policing and race. The Appeal, Jan. 3. https://theappeal.org/setting-the-record-straight-on-predictive-policing-and-race-fe588b457ca2/
    [Google Scholar]
  59. Jabine TB, Claude RP 1992. Human Rights and Statistics: Getting the Record Straight Philadelphia: Univ. Pa. Press
    [Google Scholar]
  60. Johndrow JE, Lum K, Dunson DB. 2018a. Theoretical limits of microclustering for record linkage. Biometrika 105:431–46
    [Google Scholar]
  61. Johndrow JE, Lum K, Manrique-Vallier D. 2018b. Estimating the observable population size from biased samples: a new approach to population estimation with capture heterogeneity. Biometrika In press
    [Google Scholar]
  62. Krüger J, Ball P. 2014. Evaluation of the Database of the Kosovo Memory Book. San Francisco: Hum. Rights Data Anal. Group
    [Google Scholar]
  63. Kubheka T. 2000. The South African Truth and Reconciliation Commission: data processing. See Ball et al. 2000 4194
    [Google Scholar]
  64. Langan PA. 1995. The racial disparity in U.S. drug arrests Rep., Bur. Justice Stat., US Dep. Justice, Washington, DC
    [Google Scholar]
  65. Link WA. 2003. Nonidentifiability of population size from capture-recapture data with heterogeneous detection probabilities. Biometrics 59:1123–30
    [Google Scholar]
  66. Lum K. 2017. Limitations of mitigating judicial bias with machine learning. Nat. Hum. Behav 1:0141
    [Google Scholar]
  67. Lum K, Ball P. 2015. Estimating undocumented homicides with two lists and list dependence Rep., Hum. Rights Data Anal. Group, San Francisco, CA
    [Google Scholar]
  68. Lum K, Isaac W. 2016. To predict and serve?. Significance 13:14–19
    [Google Scholar]
  69. Madigan D, York JC. 1997. Bayesian methods for estimation of the size of a closed population. Biometrika 84:19–31
    [Google Scholar]
  70. Manning CD, Raghavan P, Schütze H 2008. Introduction to Information RetrievalVol 1 Cambridge, UK: Cambridge Univ. Press
    [Google Scholar]
  71. Manrique-Vallier D. 2016. Bayesian population size estimation using Dirichlet process mixtures. Biometrics 72:1246–54
    [Google Scholar]
  72. Marks ES, Seltzer W, Krotki KJ 1974. Population Growth Estimation: A Handbook of Vital Statistics Measurement New York: Popul. Counc.
    [Google Scholar]
  73. Mason TD, Hamner J, Phillips ME. 2012. Data annex to the United Nations Truth Commission on the civil war in El Salvador from 1979–1991 UNT Data Repository, updated Dec. 20, 2013
    [Google Scholar]
  74. Michelson M, Knoblock CA. 2006. Learning blocking schemes for record linkage. Proceedings of the Twenty-first National Conference on Artificial Intelligence440–45 Palo Alto, CA: AAAI
    [Google Scholar]
  75. Ministère Public c. Hissein Habré 2016. Jugement du 30 mai 2016. Chambre Africaine Extraordinaire d'Assises, Dakar, Senegal
  76. Mneimneh Z, Axinn W, Ghimire D, Cibelli K, Alkaisy M. 2014. Conducting surveys in areas of armed conflict. Hard-to-Survey Populations R Tourangeau, B Edwards, TP Johnson, KM Wolter, N Bates134–56 Cambridge, UK: Cambridge Univ. Press
    [Google Scholar]
  77. Newcombe HB, Kennedy JM, Axford S, James AP. 1959. Automatic linkage of vital records. Science 130:954–59
    [Google Scholar]
  78. Neyman J. 1934. On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. J. R. Stat. Soc 97:558–625
    [Google Scholar]
  79. Noval AM. 1993. HURIDOCS Standard Formats for the Recording and Exchange of Bibliographic Information Concerning Human Rights. Geneva: HURIDOCS
    [Google Scholar]
  80. O'Sullivan G. 2000. The South African Truth and Reconciliation Commission: database representation. See Ball et al. 2000 95136
    [Google Scholar]
  81. Otis DL, Burnham KP, White GC, Anderson DR. 1978. Statistical inference from capture data on closed animal populations. Wildlife Monogr 62:3–135
    [Google Scholar]
  82. Perry WL. 2013. Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations. Santa Monica, CA: RAND Corp.
    [Google Scholar]
  83. Pfahringer B, Holmes G, Kirkby R. 2001. Optimizing the induction of alternating decision trees. Pacific-Asia Conference on Knowledge Discovery and Data Mining D Cheung, GJ Williams, Q Li477–87 New York: Springer
    [Google Scholar]
  84. Pham P, Vinck P. 2018. Human rights and mixed methods. CHANCE 31:29–37
    [Google Scholar]
  85. Price M, Gohdes A, Ball P. 2015. Documents of war: understanding the Syrian conflict. Significance 12:14–19
    [Google Scholar]
  86. Price M, Guberek T, Guzmán D, Zador P, Shapiro G. 2009. A statistical analysis of the Guatemalan National Police Archive: Searching for documentation of human rights abuses. Proceedings of the Joint Statistical Meeting, Survey Research Methods Section—JSM 20092441–55 Alexandria, VA: ASA
    [Google Scholar]
  87. Rivest LP. 2011. A lower bound model for multiple record systems estimation with heterogeneous catchability. Int. J. Biostat 7:1–21
    [Google Scholar]
  88. Roth F, Guberek T, Green AH. 2011. Using quantitative data to assess conflict-related sexual violence in Colombia: challenges and opportunities Rep., Benetech, Palo Alto, CA
    [Google Scholar]
  89. Sadinle M. 2017. Bayesian estimation of bipartite matchings for record linkage. J. Am. Stat. Assoc 112:600–12
    [Google Scholar]
  90. Sadinle M, Fienberg SE. 2013. A generalized Fellegi–Sunter framework for multiple record linkage with application to homicide record systems. J. Am. Stat. Assoc 108:385–97
    [Google Scholar]
  91. Salganik MJ, Heckathorn DD. 2004. Sampling and estimation in hidden populations using respondent driven sampling. Sociol. Methodol 34:193–240
    [Google Scholar]
  92. Seber GA. 1965. A note on the multiple-recapture census. Biometrika 52:249–59
    [Google Scholar]
  93. Seltzer W. 1998. Population statistics, the Holocaust, and the Nuremberg trials. Popul. Dev. Rev 24:511–52
    [Google Scholar]
  94. Shapiro G, Guzmán D, Zador P, Guberek T, Price M, Lum K. 2009. Weighting for the Guatemalan National Police archive sample: unusual challenges and problems. Proceedings of the Joint Statistical Meeting, Survey Research Methods Section—JSM 20094656–70 Alexandria, VA: ASA
    [Google Scholar]
  95. Silva R. 2002. On ensuring a higher level of data quality when documenting human rights violations to support research into the origin and cause of human rights violations. Proceedings of the Joint Statistical Meeting, Social Statistics Section3242–51 Alexandria, VA: ASA
    [Google Scholar]
  96. Silva R, Ball PD. 2006. The profile of human rights violations in Timor-Leste. 1974–1999 Rep., Benetech, Palo Alto, CA
    [Google Scholar]
  97. Silva R, Ball P. 2008. The demography of conflict-related mortality in Timor-Leste (1974–1999): reflections on empirical quantitative measurement of civilian killings, disappearances, and famine-related deaths. Statistical Methods for Human Rights J Asher, D Banks, FJ Scheuren117–39 New York: Springer
    [Google Scholar]
  98. Silva R, Klingner J, Weikart S. 2010. State coordinated violence in Chad under Hissène Habré: a statistical analysis of reported prison mortality in Chad's DDS prisons and command responsibility of Hissène Habré, 1982–1990 Rep., Hum. Rights Data Anal. Group, Benetech, Palo Alto, CA
    [Google Scholar]
  99. Spiegel PB, Salama P. 2000. War and mortality in Kosovo, 1998–99: an epidemiological testimony. Lancet 355:2204–9
    [Google Scholar]
  100. Spirer HF, Spirer L. 1993. Data Analysis for Monitoring Human Rights. Washington, DC: AAAS
    [Google Scholar]
  101. Steorts RC, Ventura SL, Sadinle M, Fienberg SE. 2014. A comparison of blocking methods for record linkage. International Conference on Privacy in Statistical Databases J Domingo-Ferrer253–68 New York: Springer
    [Google Scholar]
  102. Stormorken B, MacMorris A. 1985. HURIDOCS Standard Formats for the Recording and Exchange of Information on Human Rights. New York: Springer
    [Google Scholar]
  103. Student. 1908. The probable error of a mean. Biometrika 6:1–25
    [Google Scholar]
  104. UN Gen. Assem. Resolut. 217 A (III). 1948. Universal Declaration of Human Rights Dec. 10. UN Doc. A/810
    [Google Scholar]
  105. Wang J, Kraska T, Franklin MJ, Feng J. 2012. CrowdER: crowdsourcing entity resolution. Proc. VLDB Endow 5:1483–94
    [Google Scholar]
  106. Weschler L 1998. A Miracle, a Universe: Settling Accounts with Torturers Chicago: Univ. Chicago Press
    [Google Scholar]
  107. Wittes J, Sidel VW. 1968. A generalization of the simple capture-recapture model with applications to epidemiological research. J. Chronic Dis 21:287–301
    [Google Scholar]
  108. Yip P, Bruno G, Tajima N, Seber G, Buckland S et al. 1995a. Capture-recapture and multiple-record systems estimation I: history and theoretical development. Am. J. Epidemiol 142:1047–58
    [Google Scholar]
  109. Yip P, Bruno G, Tajima N, Seber G, Buckland S et al. 1995b. Capture-recapture and multiple-record systems estimation II: applications in human diseases. Am. J. Epidemiol. 142:105968
    [Google Scholar]
  110. Zwierzchowski J, Tabeau E. 2010. The 1992–95 war in Bosnia and Herzegovina: census-based multiple system estimation of casualties’ undercount Paper presented at European Population Conference, September 1–4, Vienna, Austria
    [Google Scholar]
/content/journals/10.1146/annurev-statistics-030718-105222
Loading
  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error