1932

Abstract

The use of models to try to better understand reality is ubiquitous. Models have proven useful in testing our current understanding of reality; for instance, climate models of the 1980s were built for science discovery, to achieve a better understanding of the general dynamics of climate systems. Scientific insights often take the form of general qualitative predictions (i.e., “under these conditions, the Earth's poles will warm more than the rest of the planet”); such use of models differs from making quantitative forecasts of specific events (i.e. “high winds at noon tomorrow at London's Heathrow Airport”). It is sometimes hoped that, after sufficient model development, any model can be used to make quantitative forecasts for any target system. Even if that were the case, there would always be some uncertainty in the prediction. Uncertainty quantification aims to provide a framework within which that uncertainty can be discussed and, ideally, quantified, in a manner relevant to practitioners using the forecast system. A statistical formalism has developed that claims to be able to accurately assess the uncertainty in prediction. This article is a discussion of if and when this formalism can do so. The article arose from an ongoing discussion between the authors concerning this issue, the second author generally being considerably more skeptical concerning the utility of the formalism in providing quantitative decision-relevant information.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-statistics-030718-105232
2019-03-07
2024-03-28
Loading full text...

Full text loading...

/deliver/fulltext/statistics/6/1/annurev-statistics-030718-105232.html?itemId=/content/journals/10.1146/annurev-statistics-030718-105232&mimeType=html&fmt=ahah

Literature Cited

  1. Ba S, Joseph VR. 2012. Composite Gaussian process models for emulating expensive functions. Ann. Appl. Stat. 6:1838–60
    [Google Scholar]
  2. Bastos LS, O'Hagan A. 2009. Diagnostics for Gaussian process emulators. Technometrics 51:425–38
    [Google Scholar]
  3. Bayarri MJ, Berger JO, Calder ES, Dalbey K, Lunagomez S, et al. 2009a. Using statistical and computer models to quantify volcanic hazards. Technometrics 51:402–13
    [Google Scholar]
  4. Bayarri MJ, Berger JO, Calder ES, Patra AK, Pitman EB, et al. 2015. Probabilistic quantification of hazards: a methodology using small ensembles of physics-based simulations and statistical surrogates. Int. J. Uncertain. Quantif. 5:297–325
    [Google Scholar]
  5. Bayarri MJ, Berger JO, García-Donato G, Liu F, Palomo J, et al. 2007a. Computer model validation with functional output. Ann. Stat. 35:1874–906
    [Google Scholar]
  6. Bayarri MJ, Berger JO, Kennedy MC, Kottas A, Paulo R, et al. 2009b. Predicting vehicle crashworthiness: validation of computer models for functional and hierarchical data. J. Am. Stat. Assoc. 104:929–42
    [Google Scholar]
  7. Bayarri MJ, Berger JO, Paulo R, Sacks J, Cafeo JA, et al. 2007b. A framework for validation of computer models. Technometrics 49:138–54
    [Google Scholar]
  8. Berger JO 2013. Statistical Decision Theory and Bayesian Analysis New York: Springer
  9. Berliner LM. 2003. Physical-statistical modeling in geophysics. J. Geophys. Res. Atmos. 108:8776
    [Google Scholar]
  10. Bröcker J, Smith LA. 2008. From ensemble forecasts to predictive distribution functions. Tellus A 60:663–78
    [Google Scholar]
  11. Conti S, O'Hagan A. 2010. Bayesian emulation of complex multi-output and dynamic computer models. J. Stat. Plann. Inference 140:640–51
    [Google Scholar]
  12. Craig PS, Goldstein M, Rougier JC, Seheult AH. 2001. Bayesian forecasting for complex systems using computer simulators. J. Am. Stat. Assoc. 96:717–29
    [Google Scholar]
  13. Craig PS, Goldstein M, Seheult AH, Smith JA. 1997. Pressure matching for hydrocarbon reservoirs: a case study in the use of Bayes linear strategies for large computer experiments. Case Studies in Bayesian Statistics: Volume III C Gatsonis, JS Hodges, RE Kass, R McCulloch, P Rossi, ND Singpurwalla3794.
    [Google Scholar]
  14. Currin C, Mitchell T, Morris M, Ylvisaker D. 1991. Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J. Am. Stat. Assoc. 86:953–63
    [Google Scholar]
  15. Du H, Smith L. 2012. Parameter estimation through ignorance. Phys. Rev. E 86:016213
    [Google Scholar]
  16. Easterling RG. 2001. Measuring the predictive capability of computational models: principles and methods, issues and illustrations. Tech. Rep. SAND2001-0243, Sandia Natl. Lab., Albuquerque, NM
  17. Fricker TE, Oakley JE, Urban NM. 2013. Multivariate Gaussian process emulators with nonseparable covariance structures. Technometrics 55:47–56
    [Google Scholar]
  18. Goldstein M, Rougier JC. 2003. Calibrated Bayesian forecasting using large computer simulators. Tech. Rep., Stat. Probab. Group, Univ. Durham, UK http://www.maths.dur.ac.uk/stats/physpred/papers/CalibratedBayesian.ps
  19. Goldstein M, Rougier JC. 2004. Probabilistic formulations for transferring inferences from mathematical models to physical systems Tech. Rep., Stat. Probab. Group, Univ. Durham, UK http://www.maths.dur.ac.uk/stats/physpred/papers/directSim.pdf
  20. Gramacy RB, Lee HK. 2009. Adaptive design and analysis of supercomputer experiments. Technometrics 51:130–45
    [Google Scholar]
  21. Gu M, Berger JO, et al. 2016a. Parallel partial Gaussian process emulation for computer models with massive output. Ann. Appl. Stat. 10:1317–47
    [Google Scholar]
  22. Gu M, Palomo J, Berger J. 2016b. : robust Gaussian stochastic process emulation. R package version 0.5 https://cran.r-project.org/web/packages/RobustGaSP/index.html
    [Google Scholar]
  23. Gu M, Wang X, Berger JO. 2018. Robust Gaussian stochastic process emulation. Ann. Stat. 46:3038–66
    [Google Scholar]
  24. Hagedorn R, Smith L. 2009. Communicating the value of probabilistic forecasts with weather roulette. Meteorol. Appl. 16:1749–72
    [Google Scholar]
  25. Higdon D, Gattiker J, Williams B, Rightley M. 2008. Computer model calibration using high-dimensional output. J. Am. Stat. Assoc. 103:570–83
    [Google Scholar]
  26. Hirsch M, Smale S 1974. Differential Equations, Dynamical Systems, and Linear Algebra New York: Academic
  27. Howson C 2000. Hume's Problem: Induction and the Justification of Belief Oxford, UK: Oxford Univ. Press
  28. Insua DR, Ruggeri F 2012. Robust Bayesian Analysis New York: Springer
  29. Judd K, Smith L. 2004. Indistinguishable states II: imperfect model scenario. Physica D 196:224–42
    [Google Scholar]
  30. Kennedy MC, O'Hagan A. 2000. Predicting the output from a complex computer code when fast approximations are available. Biometrika 87:1–13
    [Google Scholar]
  31. Kennedy MC, O'Hagan A. 2001. Bayesian calibration of computer models (with discussion). J. R. Stat. Soc. B 63:425–64
    [Google Scholar]
  32. Le Gratiet L, Cannamela C, Iooss B. 2014. A Bayesian approach for global sensitivity analysis of (multifidelity) computer codes. J. Uncertain. Quantif. 2:336–63
    [Google Scholar]
  33. Levi I 1983. The Enterprise of Knowledge: An Essay on Knowledge, Credal Probability, and Chance Cambridge, MA: MIT press
  34. Linkletter C, Bingham D, Hengartner N, Higdon D, Kenny QY. 2006. Variable selection for Gaussian process models in computer experiments. Technometrics 48:478–90
    [Google Scholar]
  35. Liu F, Bayarri MJ, Berger J. 2009. Modularization in Bayesian analysis, with emphasis on analysis of computer models. Bayesian Anal. 4:119–50
    [Google Scholar]
  36. Lorenz E. 1985. The growth of errors in prediction. Turbulence and Predictability in Geophysical Fluid Dynamics and Climate Dynamics M Ghil, R Benzi, G Parisi243–65 Amsterdam: North-Holland
    [Google Scholar]
  37. Molteni F, Buizza R, Palmer TN, Petroliagis T. 1996. The ECMWF Ensemble Prediction System: methodology and validation. Q. J. R. Meteorol. Soc. 122:73–119
    [Google Scholar]
  38. Morris MD, Mitchell TJ, Ylvisaker D. 1993. Bayesian design and analysis of computer experiments: use of derivatives in surface prediction. Technometrics 35:243–55
    [Google Scholar]
  39. NRC (Natl. Res. Counc.) 1979. Carbon Dioxide and Climate: A Scientific Assessment Tech. Rep., Natl. Acad. Sci., Washington, DC
  40. Oakley J, O'Hagan A. 2004. Probabilistic sensitivity analysis of complex models: a Bayesian approach. J. R. Stat. Soc. B 66:751–69
    [Google Scholar]
  41. Oberkampf WL, Trucano T. 2000. Validation methodology in computational fluid dynamics Paper presented at Fluids 2000, June 19–22, Denver, CO
  42. Ogburn SE, Berger J, Calder ES, Lopes D, Patra A, et al. 2016. Pooling strength amongst limited datasets using hierarchical Bayesian analysis, with application to pyroclastic density current mobility metrics. Stat. Volcanol. 2:1–26
    [Google Scholar]
  43. Parker WS 2011. When climate models agree. Philos. Sci. 78:4579–600
    [Google Scholar]
  44. Parker WS. 2014. Simulation and understanding in the study of weather and climate. Perspect. Sci. 22:336–56
    [Google Scholar]
  45. Parker WS, Risbey JS. 2015. False precision, surprise and improved uncertainty assessment. Phil. Trans. R. Soc. A 373:20140453
    [Google Scholar]
  46. Paulo R, García-Donato G, Palomo J. 2012. Calibration of computer models with multivariate output. Comput. Stat. Data Anal. 56:3959–74
    [Google Scholar]
  47. Pilch M, Trucano T, Moya JL, Froehlich G, Hodges A, Peercy D. 2001. Guidelines for Sandia ASCI verification and validation plans—content and format: Version 2.0. Tech. Rep. SAND 2001-3101, Sandia Natl. Lab., Albuquerque, NM
  48. Plumlee M. 2017. Bayesian calibration of inexact computer models. J. Am. Stat. Assoc. 112:127485
    [Google Scholar]
  49. Qian PZG, Wu H, Wu CFJ. 2008. Gaussian process models for computer experiments with qualitative and quantitative factors. Technometrics 50:383–96
    [Google Scholar]
  50. Roache PJ 1998. Verification and Validation in Computational Science and Engineering Albuquerque, NM: Hermosa
  51. Rougier J. 2008. Efficient emulators for multivariate deterministic functions. J. Comput. Gr. Stat. 17:827–43
    [Google Scholar]
  52. Sacks J, Welch WJ, Mitchell TJ, Wynn HP. 1989. Design and analysis of computer experiments. Stat. Sci. 4:409–23
    [Google Scholar]
  53. Santner TJ, Williams BJ, Notz WI 2003. The Design and Analysis of Computer Experiments New York: Springer
  54. Savitsky T, Vannucci M, Sha N. 2011. Variable selection for nonparametric Gaussian process priors: models and computational strategies. Stat. Sci. 26:130–49
    [Google Scholar]
  55. Schonlau M, Welch WJ. 2006. Screening the input variables to a computer model via analysis of variance and visualization. Screening A Dean, S Lewis308–27 New York: Springer
    [Google Scholar]
  56. Smith LA 1995. Accountability and error in ensemble forecasting. 1995 ECMWF Seminar on Predictability 135168 Reading, UK: ECMWF
    [Google Scholar]
  57. Smith LA. 1997. The maintenance of uncertainty. Past and Present Variability of the Solar-Terrestrial System: Measurement, Data Analysis and Theoretical Models GC Castagnoli, A Provenzale177–246 Bologna, Italy: Societá Italiana di Fisica
    [Google Scholar]
  58. Smith LA. 2002. What might we learn from climate forecasts. PNAS 99:2487–92
    [Google Scholar]
  59. Smith LA. 2016. Integrating information, misinformation and desire: improved weather-risk management for the energy sector. UK Success Stories in Industrial Mathematics PJ Aston, AJ Mulholland, KMM Tant289–96 New York: Springer
    [Google Scholar]
  60. Smith LA, Petersen AC. 2015. Variations on reliability: connecting climate predictions to climate policy. Error and Uncertainty in Scientific Practice M Boumans, G Hon, AC Petersen151–70 London: Routledge
    [Google Scholar]
  61. Smith LA, Stern N. 2011. Uncertainty in science and its role in climate policy. Phil. Trans. R. Soc. A 369:4818–41
    [Google Scholar]
  62. Smith RC 2014. Uncertainty Quantification: Theory, Implementation, and Applications Philadelphia: SIAM
  63. Smith RL, Tebaldi C, Nychka D, Mearns LO. 2009. Bayesian modeling of uncertainty in ensembles of climate models. J. Am. Stat. Assoc. 104:97–116
    [Google Scholar]
  64. Sobol IM. 2001. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math. Comput. Simul. 55:271–80
    [Google Scholar]
  65. Spiller ET, Bayarri M, Berger JO, Calder ES, Patra AK, et al. 2014. Automating emulator construction for geophysical hazard maps. J. Uncertain. Quantif. 2:126–52
    [Google Scholar]
  66. Stainforth DA, Aina T, Christensen C, Collins M, Faull N, et al. 2005. Uncertainty in predictions of the climate response to rising levels of greenhouse gases. Nature 433:403–6
    [Google Scholar]
  67. Stocker T, Qin D, Plattner G, Tignor M, Allen S 2013. Climate Change 2013: The Physical Science Basis. Cambridge, UK: Cambridge Univ. Press
  68. Storlie CB, Lane WA, Ryan EM, Gattiker JR, Higdon DM. 2015. Calibration of computational models with categorical parameters and correlated outputs via Bayesian smoothing spline ANOVA. J. Am. Stat. Assoc. 110:68–82
    [Google Scholar]
  69. Tarantola S, Gatelli D, Kucherenko S, Mauntz W, et al. 2007. Estimating the approximation error when fixing unessential factors in global sensitivity analysis. Reliability Eng. Syst. Saf. 92:957–60
    [Google Scholar]
  70. Tennekes H, Baede APM, Opsteegh JD. 1986. Forecasting forecast skill Presented at Workshop on Predictability in the Medium and Extended Range, March 17–19, Shinfield Park, Reading, UK
  71. Trucano T, Pilch M, Oberkampf WO. 2002. General concepts for experimental validation of ASCI code applications Tech. Rep. SAND 2002-0341, Sandia Natl. Lab., Albuquerque, NM
  72. van den Dool H 2007. Empirical Methods in Short-Term Climate Prediction Oxford, UK: Oxford Univ. Press
  73. Welch WJ, Buck RJ, Sacks J, Wynn HP, Mitchell TJ, Morris MD. 1992. Screening, predicting, and computer experiments. Technometrics 34:15–25
    [Google Scholar]
/content/journals/10.1146/annurev-statistics-030718-105232
Loading
/content/journals/10.1146/annurev-statistics-030718-105232
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error