1932

Abstract

The first mobile camera phone was sold only 20 years ago, when taking pictures with one's phone was an oddity, and sharing pictures online was unheard of. Today, the smartphone is more camera than phone. How did this happen? This transformation was enabled by advances in computational photography—the science and engineering of making great images from small-form-factor, mobile cameras. Modern algorithmic and computing advances, including machine learning, have changed the rules of photography, bringing to it new modes of capture, postprocessing, storage, and sharing. In this review, we give a brief history of mobile computational photography and describe some of the key technological components, including burst photography, noise reduction, and super-resolution. At each step, we can draw naive parallels to the human visual system.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-vision-093019-115521
2021-09-15
2024-03-29
Loading full text...

Full text loading...

/deliver/fulltext/vision/7/1/annurev-vision-093019-115521.html?itemId=/content/journals/10.1146/annurev-vision-093019-115521&mimeType=html&fmt=ahah

Literature Cited

  1. Abdelhamed A, Lin S, Brown MS. 2018. A high-quality denoising dataset for smartphone cameras. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT1692–700 Piscataway, NJ: IEEE
    [Google Scholar]
  2. Adams A. 1935. Making a Photograph: An Introduction to Photography London: Studio Ed. Ltd.
  3. Afifi M, Price B, Cohen S, Brown MS. 2019. Image recoloring based on object color distributions Work. Pap. York Univ. Toronto, ON, Can.:
  4. Ahn H, Keum B, Kim D, Lee HS 2013. Adaptive local tone mapping based on Retinex for high dynamic range images. Proceedings of the 2013 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV153–56 Piscataway, NJ: IEEE
    [Google Scholar]
  5. Banterle F, Artusi A, Debattista K, Chalmers A 2017. Advanced High Dynamic Range Imaging Boca Raton, FL: CRC Press
  6. Barron JT, Tsai YT. 2017. Fast Fourier color constancy. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI886–94 Piscataway, NJ: IEEE
    [Google Scholar]
  7. Bayer BE. 1975. Color imaging array US Patent 3,971,065
  8. Beil K. 2020. Photography has gotten climate change wrong from the start. The Atlantic Nov. 27. https://www.theatlantic.com/ideas/archive/2020/11/photography-has-never-known-how-handle-climate-change/617224
    [Google Scholar]
  9. Bruhn A, Weickert J, Schnörr C. 2005. Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. Int. J. Comput. Vis. 61:211–31
    [Google Scholar]
  10. Buades A, Coll B, Morel JM. 2005. A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4:490–530
    [Google Scholar]
  11. Buades A, Coll B, Morel JM. 2006. Image enhancement by non-local reverse heat equation CMLA Preprint 22 Cent. Math. Leurs Appl. Cachan, France:
  12. Buchsbaum G. 1980. A spatial processor model for object colour perception. J. Franklin Inst. 310:1–26
    [Google Scholar]
  13. Burger HC, Schuler CJ, Harmeling S. 2012. Image denoising: Can plain neural networks compete with BM3D?. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition2392–99 Piscataway, NJ: IEEE
    [Google Scholar]
  14. Burr D, Ross J, Morrone MC. 1986. Seeing objects in motion. Proc. R. Soc. Lond. B 227:249–65
    [Google Scholar]
  15. Cerda X, Parraga CA, Otazu X. 2018. Which tone-mapping operator is the best? A comparative study of perceptual quality. J. Opt. Soc. Am. A 35:626–38
    [Google Scholar]
  16. Chatterjee P, Milanfar P. 2010. Is denoising dead?. IEEE Trans. Image Proc. 19:895–911
    [Google Scholar]
  17. Chen Y, Yu W, Pock T 2015. On learning optimized reaction diffusion processes for effective image restoration. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA5261–69 Piscataway, NJ: IEEE
    [Google Scholar]
  18. Cheng D, Prasad DK, Brown MS. 2014. Illuminant estimation for color constancy: why spatial-domain methods work and the role of the color distribution. J. Opt. Soc. Am. A 31:1049–58
    [Google Scholar]
  19. Cheng D, Price B, Cohen S, Brown MS. 2015. Effective learning-based illuminant estimation using simple features. Proceedings of the 2015 IEEE Conference of Computer Vision and Pattern Recognition (CVPR), Boston, MA1000–8 Piscataway, NJ: IEEE
    [Google Scholar]
  20. Dabov K, Foi A, Katkovnik V, Egiazarian K. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Proc. 16:2080–95
    [Google Scholar]
  21. Debevec PE, Malik J. 1997. Recovering high dynamic range radiance maps from photographs. SIGGRAPH '97: Proceedings of the 24th Annual Conference on Computer Graphics and Iterative Techniques369–78 New York: ACM
    [Google Scholar]
  22. Delbracio M, Garcia-Dorado I, Choi S, Kelly D, Milanfar P. 2020. Polyblur: removing mild blur by polynomial reblurring. arXiv:2012.09322 [cs.CV]
  23. Efros AA, Leung TK. 1999. Texture synthesis by non-parametric sampling. Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, Vol. 21033–38 Piscataway, NJ: IEEE
    [Google Scholar]
  24. El-Henawy I, Amin A, Kareem Ahmed HA 2014. A comparative study on image deblurring techniques. Int. J. Adv. Comput. Sci. Technol. 3:1–8
    [Google Scholar]
  25. Elad M. 2002. On the origin of the bilateral filter and ways to improve it. IEEE Trans. Image Proc. 11:1141–50
    [Google Scholar]
  26. Farsiu S, Elad M, Milanfar P. 2006. Multiframe demosaicing and super-resolution of color images. IEEE Trans. Image Proc. 15:141–59
    [Google Scholar]
  27. Fergus R, Singh B, Hertzmann A, Roweis ST, Freeman WT. 2006. Removing camera shake from a single photograph. SIGGRAPH '06: ACM SIGGRAPH 2006 Papers787–94 New York: ACM
    [Google Scholar]
  28. Gehler PV, Rother C, Blake A, Minka T, Sharp T. 2008. Bayesian color constancy revisited. Proceedings of the 2008 IEEE Conference of Computer Vision and Pattern Recognition, Anchorage, AK art. 4587765 Piscataway, NJ: IEEE
    [Google Scholar]
  29. Gharbi M, Chaurasia G, Paris S, Durand F 2016. Deep joint demosaicking and denoising. ACM Trans. Graph. 35:191
    [Google Scholar]
  30. Godard C, Matzen K, Uyttendaele M. 2018. Deep burst denoising. Proceedings of the 15th European Conference on Computer Vision, Munich, Germany560–77 Berlin: Springer
    [Google Scholar]
  31. Hasinoff SW, Durand F, Freeman WT 2010. Noise-optimal capture for high dynamic range photography. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA553–60 Piscataway, NJ: IEEE
    [Google Scholar]
  32. Hasinoff SW, Sharlet D, Geiss R, Adams A, Barron JT et al. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. 35:192
    [Google Scholar]
  33. Horn BK, Schunck BG. 1981. Determining optical flow. Artif. Intel. 17:185–203
    [Google Scholar]
  34. Hosseini MS, Plataniotis KN. 2019. Convolutional deblurring for natural imaging. IEEE Trans. Image Proc. 29:250–64
    [Google Scholar]
  35. Hu Y, Wang B, Lin S. 2017. FC4: fully convolutional color constancy with confidence-weighted pooling. Proceedings of the 2017 IEEE Conference of Computer Vision and Pattern Recognition (CVPR), Honolulu, HI330–39 Piscataway, NJ: IEEE
    [Google Scholar]
  36. Ignatov A, Van Gool L, Timofte R. 2020. Replacing mobile camera ISP with a single deep learning model. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA2275–85 Piscataway, NJ: IEEE
    [Google Scholar]
  37. Intoy J, Rucci M. 2020. Finely tuned eye movements enhance visual acuity. Nat. Commun. 11:795
    [Google Scholar]
  38. Jiang J, Liu D, Gu J, Süsstrunk S. 2013. What is the space of spectral sensitivity functions for digital color cameras?. Proceedings of the 2013 IEEE Workshop on Applications of Computer Vision (WACV), Clearwater Beach, FL168–79 Piscataway, NJ: IEEE
    [Google Scholar]
  39. Kainz F, Murthy K. 2019. Astrophotography with night sight on Pixel phones. Google AI Blog Nov. 26. https://ai.googleblog.com/2019/11/astrophotography-with-night-sight-on.html
    [Google Scholar]
  40. Karaimer HC, Brown MS. 2016. A software platform for manipulating the camera imaging pipeline Paper presented at the European Conference on Computer Vision (ECCV '16), Oct. 8–16, Amsterdam
  41. Kelber A, Yovanovich C, Olsson P. 2017. Thresholds and noise limitations of colour vision in dim light. Philos. Trans. R. Soc. Lond. B 372:20160065
    [Google Scholar]
  42. Kovásznay LS, Joseph HM. 1955. Image processing. Proc. Inst. Radio Eng 43:560–70
    [Google Scholar]
  43. Kramer RH, Davenport CM. 2015. Lateral inhibition in the vertebrate retina: the case of the missing neurotransmitter. PLOS Biol 13:e1002322
    [Google Scholar]
  44. Land EH. 1974. The retinex theory of colour vision. Proc. Roy. Inst. Gr. Br. 47:23–58
    [Google Scholar]
  45. Lebrun M, Colom M, Buades A, Morel JM. 2012. Secrets of image denoising cuisine. Acta Numer 21:475–576
    [Google Scholar]
  46. Levin A, Weiss Y, Durand F, Freeman WT 2009. Understanding and evaluating blind deconvolution algorithms. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL1964–71 Piscataway, NJ: IEEE
    [Google Scholar]
  47. Levoy M, Pritch Y. 2018. Night sight: seeing in the dark on Pixel phones. Google AI Blog Nov. 14. https://ai.googleblog.com/2018/11/night-sight-seeing-in-dark-on-pixel.html
    [Google Scholar]
  48. Liba O, Murthy K, Tsai YT, Brooks T, Xue T et al. 2019. Handheld mobile photography in very low light. ACM Trans. Graph. 38:164
    [Google Scholar]
  49. Lindenbaum M, Fischer M, Bruckstein A. 1994. On Gabor's contribution to image enhancement. Pattern Recognit 27:1–8
    [Google Scholar]
  50. Liu D, Wen B, Fan Y, Loy CC, Huang TS. 2018. Non-local recurrent network for image restoration. Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS 2018), Montreal, Canada1680–89 New York: ACM
    [Google Scholar]
  51. Liu P, Zhang H, Zhang K, Lin L, Zuo W. 2018. Multi-level wavelet-CNN for image restoration. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT886–95 Piscataway, NJ: IEEE
    [Google Scholar]
  52. Longere P, Zhang X, Delahunt PB, Brainard DH. 2002. Perceptual assessment of demosaicing algorithm performance. Proc. IEEE 90:123–32
    [Google Scholar]
  53. Lovejoy B. 2020. What is a periscope lens, and what would it mean for future iPhones?. 9to5Mac July 22. https://9to5mac.com/2020/07/22/periscope-lens/
    [Google Scholar]
  54. Ma K, Yeganeh H, Zeng K, Wang Z. 2015. High dynamic range image compression by optimizing tone mapped image quality index. IEEE Trans. Image Proc. 24:3086–97
    [Google Scholar]
  55. Mao X, Shen C, Yang YB. 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain2810–18 New York: ACM
    [Google Scholar]
  56. Meinhardt T, Moeller M, Hazirbas C, Cremers D. 2017. Learning proximal operators: using denoising networks for regularizing inverse imaging problems. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy1799–808 Piscataway, NJ: IEEE
    [Google Scholar]
  57. Mertens T, Kautz J, Reeth FV. 2007. Exposure fusion. Proceedings of the 15th Pacific Conference on Computer Graphics and Applications (PG'07), Maui, HI382–90 Piscataway, NJ: IEEE
    [Google Scholar]
  58. Milanfar P. 2013. A tour of modern image filtering: new insights and methods, both practical and theoretical. IEEE Signal Proc. Mag. 30:106–28
    [Google Scholar]
  59. Mildenhall B, Barron JT, Chen J, Sharlet D, Ng R, Carroll R 2018. Burst denoising with kernel prediction networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT2502–10 Piscataway, NJ: IEEE
    [Google Scholar]
  60. Osher S, Rudin LI. 1990. Feature-oriented image enhancement using shock filters. SIAM J. Numer. Anal. 27:919–40
    [Google Scholar]
  61. Paris S, Hasinoff SW, Kautz J. 2011. Local Laplacian filters: edge-aware image processing with a Laplacian pyramid. ACM Trans. Graph. 30:68
    [Google Scholar]
  62. Plötz T, Roth S. 2017. Benchmarking denoising algorithms with real photographs. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI2750–59 Piscataway, NJ: IEEE
    [Google Scholar]
  63. Poynton C. 2012. Digital Video and HD: Algorithms and Interfaces Amsterdam: Morgan Kaufmann, 2nd ed..
  64. Ramanath R, Drew MS. 2014. Von Kries Hypothesis Berlin: Springer
  65. Ratliff F. 1965. Mach Bands: Quantitative Studies on Neural Networks, Vol. 2 San Francisco: Holden-Day
  66. Reinhard E, Stark M, Shirley P, Ferwerda J 2002. Photographic tone reproduction for digital images. ACM Trans. Graph. 21:267–76
    [Google Scholar]
  67. Remez T, Litany O, Giryes R, Bronstein AM. 2018. Class-aware fully convolutional Gaussian and Poisson denoising. IEEE Trans. Image Proc. 27:5707–22
    [Google Scholar]
  68. Riviere C, Rader RS, Thakor NV. 1998. Adaptive cancelling of physiological tremor for improved precision in microsurgery. IEEE Trans. Biomed. Eng. 45:839–46
    [Google Scholar]
  69. Romano Y, Isidoro J, Milanfar P. 2017. RAISR: rapid and accurate image super resolution. IEEE Trans. Comput. Imaging 3:110–25
    [Google Scholar]
  70. Rucci M, Iovin R, Poletti M, Santini F. 2007. Miniature eye movements enhance fine spatial detail. Nature 447:852–55
    [Google Scholar]
  71. Sampat N, Venkataraman S, Yeh T, Kremens RL 1999. System implications of implementing auto-exposure on consumer digital cameras. Proceedings of the SPIE 3650: Sensors, Cameras, and Applications for Digital Photography N Sampat, T Yeh, art. 342854 Bellingham, WA: SPIE
    [Google Scholar]
  72. Schwartz E, Giryes R, Bronstein AM. 2018. DeepISP: toward learning an end-to-end image processing pipeline. IEEE Trans. Image Proc. 28:912–23
    [Google Scholar]
  73. Smith SM, Brady JM. 1997. SUSAN: a new approach to low level image processing. Int. J. Comput. Vis. 23:45–78
    [Google Scholar]
  74. Spira A, Kimmel R, Sochen N. 2007. A short time Beltrami kernel for smoothing images and manifolds. IEEE Trans. Image Proc. 16:1628–36
    [Google Scholar]
  75. Stevens SS. 1961. To honor Fechner and repeal his law: A power function, not a log function, describes the operating characteristic of a sensory system. Science 133:80–86
    [Google Scholar]
  76. Sun D, Yang X, Liu M, Kautz J. 2018. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT8934–43
    [Google Scholar]
  77. Süsstrunk S, Buckley R, Swen S 1999. Standard RGB color spaces. Proceedings of the Seventh Color Imaging Conference: Color Science, Systems, and Applications127–34 Springfield, VA: Soc. Imaging Sci. Technol.
    [Google Scholar]
  78. Tai Y, Yang J, Liu X, Xu C. 2017. Memnet: a persistent memory network for image restoration. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy4549–57 Piscataway, NJ: IEEE
    [Google Scholar]
  79. Takeda H, Farsiu S, Milanfar P. 2006. Robust kernel regression for restoration and reconstruction of images from sparse, noisy data. Proceedings of the 2006 IEEE International Conference on Image Processing, Atlanta, GA1257–60 Piscataway, NJ: IEEE
    [Google Scholar]
  80. Takeda H, Farsiu S, Milanfar P. 2007. Kernel regression for image processing and reconstruction. IEEE Trans. Image Proc. 16:349–66
    [Google Scholar]
  81. Talebi H, Milanfar P. 2018. Nima: neural image assessment. IEEE Trans. Image Proc. 27:3998–4011
    [Google Scholar]
  82. Tan H, Zeng X, Lai S, Liu Y, Zhang M. 2017. Joint demosaicing and denoising of noisy Bayer images with ADMM. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China2951–55 Piscataway, NJ: IEEE
    [Google Scholar]
  83. Telleen J, Sullivan A, Yee J, Wang O, Gunawardane P et al. 2007. Synthetic shutter speed imaging. Comput. Graph. Forum 26:591–98
    [Google Scholar]
  84. Tomasi C, Manduchi R. 1998. Bilateral filtering for gray and color images. Proceedings of the Sixth IEEE International Conference on Computer Vision, Bombay, India836–46 Piscataway, NJ: IEEE
    [Google Scholar]
  85. Van De Weijer J, Gevers T, Gijsenij A. 2007. Edge-based color constancy. IEEE Trans. Image Proc. 16:2207–14
    [Google Scholar]
  86. Wadhwa N, Garg R, Jacobs DE, Feldman BE, Kanazawa N et al. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM Trans. Graph. 37:64
    [Google Scholar]
  87. Wang Z, Chen J, Hoi SC 2021. Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. In press
    [Google Scholar]
  88. Wang Z, Liu D, Yang J, Han W, Huang T. 2015. Deep networks for image super-resolution with sparse prior. Proceedings of the 2015 IEEE Conference on Computer Vision (ICCV), Santiago, Chile370–78
    [Google Scholar]
  89. Weickert J. 1999. Coherence-enhancing diffusion. Int. J. Comput. Vis. 31:111–27
    [Google Scholar]
  90. Westheimer G. 1975. Visual acuity and hyperacuity. Investig. Ophthalmol. Vis. Sci. 14:570–72
    [Google Scholar]
  91. Wronski B, Garcia-Dorado I, Ernst M, Kelly D, Krainin M et al. 2019. Handheld multi-frame super-resolution. ACM Trans. Graph. 38:28
    [Google Scholar]
  92. Wronski B, Milanfar P. 2018. See better and further with super res zoom on the Pixel 3. Google AI Blog Oct. 15. https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html
    [Google Scholar]
  93. You YL, Xu W, Tannenbaum A, Kaveh M. 1996. Behavioral analysis of anisotropic diffusion in image processing. IEEE Trans. Image Proc. 5:1539–53
    [Google Scholar]
  94. Zhang K, Zuo W, Chen Y, Meng D, Zhang L. 2017. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Proc. 26:3142–55
    [Google Scholar]
  95. Zhang K, Zuo W, Zhang L. 2018. FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Proc. 27:4608–22
    [Google Scholar]
  96. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y. 2018. Residual dense network for image restoration. arXiv:1812.10477 [cs.CV]
  97. Zoran D, Weiss Y. 2011. From learning models of natural image patches to whole image restoration. Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain479–86 Piscataway, NJ: IEEE
    [Google Scholar]
/content/journals/10.1146/annurev-vision-093019-115521
Loading
/content/journals/10.1146/annurev-vision-093019-115521
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error