1932
0

Annual Review of Vision Science

Volume 10, 2024

Interactions Between 3D Surface Shape and Material Perception

Abstract

Our visual systems are remarkably adept at deriving the shape and material properties of surfaces even when only one image of a surface is available. This ability implies that a single image of a surface contains potent information about both surface shape and material. However, from a computational perspective, the problem of deriving surface shape and material is formally ill posed. Any given image could be due to many combinations of shape, material, and illumination. Early computational models required prior knowledge about two of the three scene variables to derive the third. However, such models are biologically implausible because our visual systems are tasked with extracting all relevant scene variables from images simultaneously. This review describes recent progress in understanding how the visual system solves this problem by identifying complex forms of image structure that support its ability to simultaneously derive the shape and material properties of surfaces from images.

Keyword(s): material perception, shape from shading, shape perception, specular reflections, translucency
Loading

Article metrics loading...

/content/journals/10.1146/annurev-vision-102122-094213
2024-09-18
2025-06-15
Loading full text...

Full text loading...

/deliver/fulltext/vision/10/1/annurev-vision-102122-094213.html?itemId=/content/journals/10.1146/annurev-vision-102122-094213&mimeType=html&fmt=ahah

Literature Cited

  1. Anderson BL, Kim J. 2009.. Image statistics do not explain the perception of gloss and lightness. . J. Vis. 9:(11):10
     [Crossref] [Google Scholar]
  2. Barron JT, Malik J. 2011.. High-frequency shape and albedo from shading using natural image statistics. . In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2521–28. Piscataway, NJ:: IEEE
     [Google Scholar]
  3. Barron JT, Malik J. 2012.. Color constancy, intrinsic images, and shape estimation. . In Proceedings of the 12th European Conference on Computer Vision, pp. 57–70. Berlin:: Springer
     [Google Scholar]
  4. Barron JT, Malik J. 2015.. Shape, illumination, and reflectance from shading. . IEEE Trans. Pattern Anal. Mach. Intel. 37::1670–87
     [Crossref] [Google Scholar]
  5. Barrow HG, Tenenbaum JM. 1978.. Recovering intrinsic scene characteristics from images. . In Computer Vision Systems, ed. A Hanson, E Riseman , pp. 3–26. New York:: Academic
     [Google Scholar]
  6. Beck J, Prazdny S. 1981.. Highlights and the perception of glossiness. . Percept. Psychophys. 30:(4):407–10
     [Crossref] [Google Scholar]
  7. Belheumer PN, Kriegman DL, Yuille AL. 1999.. The bas-relief ambiguity. . Int. J. Comput. Vis. 35:(1):33–44
     [Crossref] [Google Scholar]
  8. Cook RL, Torrance KE. 1982.. A reflectance model for computer graphics. . ACM Trans. Graph. 1:(1):7–24
     [Crossref] [Google Scholar]
  9. Egan EJ, Todd JT. 2015.. The effects of smooth occlusions and directions of illumination on the visual perception of 3-D shape from shading. . J. Vis. 15:(2):24
     [Crossref] [Google Scholar]
  10. Erens RG, Kapper AM, Koenderink JJ. 1993.. Estimating local shape from shading in the presence of global shading. . Percept. Psychophys. 54:(3):334–42
     [Crossref] [Google Scholar]
  11. Ferwerda JA, Fabio P, Greenberg DP. 2001.. Psychophysically based model of surface gloss perception. . Proc. SPIE. 4299::291–301
     [Crossref] [Google Scholar]
  12. Fleming RW, Bülthoff HH. 2005.. Low-level image cues in the perception of translucent materials. . ACM Trans. Appl. Percept. 2:(3):346–82
     [Crossref] [Google Scholar]
  13. Fleming RW, Dror RO, Adelson EH. 2003.. Real-world illumination and the perception of surface reflectance properties. . J. Vis. 3:(5):347–68
     [Crossref] [Google Scholar]
  14. Fleming RW, Torralba A, Adelson EH. 2004.. Specular reflections and the perception of shape. . J. Vis. 4:(9):10
     [Crossref] [Google Scholar]
  15. Fleming RW, Torralba A, Adelson EH. 2009.. Shape from sheen. Tech. Rep. MIT-CSAIL-TR-2009-051 , Mass. Inst. Technol., Cambridge, MA:
     [Google Scholar]
  16. Gibson JJ. 1979.. The Ecological Approach to Visual Perception. Boston:: Houghton Mifflin
     [Google Scholar]
  17. Gkioulekas I, Xiao B, Zhao S, Adelson EH, Zickler T, Bala K. 2013.. Understanding the role of phase function in translucent appearance. . ACM Trans. Graph. 32:(5):147
     [Crossref] [Google Scholar]
  18. Han X, Zickler T. 2023.. Curvature fields from shading fields. . In Proceedings of the NeurIPS 2023 Workshop on Symmetry and Geometry in Neural Representations, pp. 1–23. San Diego:: NeurIPS
     [Google Scholar]
  19. Ho YX, Landy MS, Maloney LT. 2008.. Conjoint measurement of gloss and surface texture. . Psychol. Sci. 19::196–204
     [Crossref] [Google Scholar]
  20. Horn BKP. 1975.. Obtaining shape from shading information. . In The Psychology of Computer Vision, ed. PH Winston , pp. 115–55. New York:: McGraw Hill
     [Google Scholar]
  21. Huggins PS, Chen HF, Belhumeur PN, Zucker SW. 2001.. Finding folds: on the appearance and identification of occlusion. . In Proceedings of the 2001 IEEE Conference on Computer Vision and Pattern Recognition, pp. 718–25. Piscataway, NJ:: IEEE
     [Google Scholar]
  22. Huggins PS, Zucker SW. 2001a.. Folds and cuts: how shading flows into edges. . In Proceedings of the 2001 IEEE Conference on Computer Vision and Pattern Recognition, pp. 153–58. Piscataway, NJ:: IEEE
     [Google Scholar]
  23. Huggins PS, Zucker SW. 2001b.. How folds cut a scene. . In Proceedings of the Fourth International Workshop on Visual Form, pp. 323–32. Berlin:: Springer
     [Google Scholar]
  24. Hunter RS, Harold RW. 1987.. The Measurement of Appearance. Hoboken, NJ:: Wiley
     [Google Scholar]
  25. Ikeuchi K, Horn BK. 1981.. Numerical shape from shading and occluding boundaries. . Artif. Intell. 17:(1–3):141–84
     [Crossref] [Google Scholar]
  26. Jensen HW. 2001.. A practical model of sub-surface transport. . In SIGGRAPH ‘01: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 511–18. New York:: ACM
     [Google Scholar]
  27. Kim J, Marlow P, Anderson BL. 2011.. The perception of gloss depends on highlight congruence with surface shading. . J. Vis. 11:(9):4
     [Crossref] [Google Scholar]
  28. Kim M, Wilcox LM, Murray RF. 2006.. Perceived three-dimensional shape toggles perceived glow. . Curr. Biol. 26:(9):R350–51
     [Crossref] [Google Scholar]
  29. Klinker GJ, Shafer SA, Kanade T. 1988.. The measurement of highlights in color images. . Int. J. Comput. Vis. 2::7–32
     [Crossref] [Google Scholar]
  30. Knill DC, Kersten D. 1991.. Apparent surface curvature affects lightness perception. . Nature 351::228–30
     [Crossref] [Google Scholar]
  31. Koenderink JJ. 1984.. What does the occluding contour tell us about solid shape?. Perception 13:(3):321–30
     [Crossref] [Google Scholar]
  32. Koenderink JJ, van Doorn AJ. 1980.. Photometric invariants related to solid shape. . J. Mod. Opt. 27:(7):981–96
     [Google Scholar]
  33. Koenderink JJ, van Doorn AJ. 2001.. Shading in the case of translucent objects. . In Proceedings of the 2001 SPIE Conference on Human Vision and Electronic Imaging, Vol. 6, pp. 312–20. Bellingham, WA:: SPIE
     [Google Scholar]
  34. Koenderink JJ, van Doorn AJ, Christou C, Lappin JS. 1996a.. Perturbation study of shading in pictures. . Perception 25::1009–26
     [Crossref] [Google Scholar]
  35. Koenderink JJ, van Doorn AJ, Christou C, Lappin JS. 1996b.. Shape constancy in pictorial relief. . Perception 25::155–64
     [Crossref] [Google Scholar]
  36. Koenderink JJ, van Doorn AJ, Kappers AM. 1992.. Surface perception in pictures. . Percept. Psychophys. 52::487–96
     [Crossref] [Google Scholar]
  37. Koenderink JJ, van Doorn AJ, Kappers AM, Todd JT. 2001.. Ambiguity and the “mental eye” in pictorial relief. . Perception 30:(4):431–48
     [Crossref] [Google Scholar]
  38. Kunsberg B, Holtmann-Rice D, Alexander E, Cholewiak S, Fleming RW, Zucker SW. 2018.. Colour, contours, shading, and shape: Flow interactions reveal anchor neighbourhoods. . Interface Focus 8::20180019
     [Crossref] [Google Scholar]
  39. Kunsberg B, Zucker SW. 2013.. Characterizing ambiguity in light source invariant shape from shading. . arXiv:1306.5480 [cs.CV]
  40. Kunsberg B, Zucker SW. 2018.. Critical contours: an invariant linking image flow salient surface organization. . SIAM J. Imaging Sci. 11::1849–77
     [Crossref] [Google Scholar]
  41. Kunsberg B, Zucker SW. 2021.. From boundaries to bumps: when closed (extremal) contours are critical. . J. Vis. 21:(13):7
     [Crossref] [Google Scholar]
  42. Landy MS, Maloney LT, Johnston EB, Young M. 1995.. Measurement and modelling of depth cue combination: in defense of weak fusion. . Vis. Res. 35:(3):389–412
     [Crossref] [Google Scholar]
  43. Larson GW, Shakespeare R. 1998.. Rendering with Radiance. Burlington, MA:: Morgan Kaufmann
     [Google Scholar]
  44. Mamassian P, Kersten D. 1996.. Illumination, shading, and the perception of local orientation. . Vis. Res. 36:(15):2351–67
     [Crossref] [Google Scholar]
  45. Marlow PJ, Anderson BL. 2013.. Generative constraints on image cues for perceived gloss. . J. Vis. 13:(14):2
     [Crossref] [Google Scholar]
  46. Marlow PJ, Anderson BL. 2015.. Material properties derived from three-dimensional shape representations. . Vis. Res. 115::199–208
     [Crossref] [Google Scholar]
  47. Marlow PJ, Anderson BL. 2016.. Motion and texture shape cues modulate perceived material properties. . J. Vis. 16:(1):5
     [Crossref] [Google Scholar]
  48. Marlow PJ, Anderson BL. 2021.. The co-specification of the shape and material properties of light-permeable materials. . PNAS 118::e2024798118
     [Crossref] [Google Scholar]
  49. Marlow PJ, Kim J, Anderson BL. 2011.. The role of brightness and orientation congruence in the perception of surface gloss. . J. Vis. 11:(9):16
     [Crossref] [Google Scholar]
  50. Marlow PJ, Kim J, Anderson BL. 2012.. The perception and misperception of specular surface reflectance. . Curr. Biol. 22::1909–13
     [Crossref] [Google Scholar]
  51. Marlow PJ, Kim J, Anderson BL. 2017.. Perception and misperception of surface opacity. . PNAS 114::13840–45
     [Crossref] [Google Scholar]
  52. Marlow PJ, Mooney SWJ, Anderson BL. 2019.. Photogeometric cues to perceived surface shading. . Curr. Biol. 29::306–11
     [Crossref] [Google Scholar]
  53. Marlow PJ, Prior de Heer B, Anderson BL. 2023.. The role of self-occluding contours in material perception. . Curr. Biol. 33:(12):2528–34
     [Crossref] [Google Scholar]
  54. Marlow PJ, Todorovïc D, Anderson BL. 2015.. Coupled computations of 3D shape and material. . Curr. Biol. 25::R221–22
     [Crossref] [Google Scholar]
  55. Marr D. 1977.. Analysis of occluding contour. . Proc. R. Soc. B 197:(1129):441–75
     [Google Scholar]
  56. Mooney SWJ, Anderson BL. 2014.. Specular image structure modulates the perception of three-dimensional shape. . Curr. Biol. 24::2737–42
     [Crossref] [Google Scholar]
  57. Mooney SWJ, Marlow PJ, Anderson BL. 2019.. The perception and misperception of optical defocus, shading, and shape. . eLife 8::e48214
     [Crossref] [Google Scholar]
  58. Motoyoshi I. 2010.. Highlight-shading relationship as a cue for the perception of translucent and transparent materials. . J. Vis. 10:(9):6
     [Crossref] [Google Scholar]
  59. Murray RF. 2021.. Lightness perception in complex scenes. . Annu. Rev. Vis. Sci. 7::417–36
     [Crossref] [Google Scholar]
  60. Nagai T, Ono Y, Tani Y, Koida K, Kitazaki M, Nakauchi S. 2013.. Image regions contributing to perceptual translucency: a psychophysical reverse-correlation study. . i-Perception 4::407–28
     [Crossref] [Google Scholar]
  61. Nefs HT, Koenderink JJ, Kappers AML. 2005.. The influence of illumination direction on the pictorial reliefs of Lambertian surfaces. . Perception 34::275–87
     [Crossref] [Google Scholar]
  62. Nefs HT, Koenderink JJ, Kappers AML. 2006.. Shape-from-shading for matte and glossy objects. . Acta Psychol. 121:(3):297–316
     [Crossref] [Google Scholar]
  63. Nicodemus F. 1965.. Directional reflectance and emissivity of an opaque surface. . Appl. Opt. 4:(7):767–75
     [Crossref] [Google Scholar]
  64. Nishida S, Shinya M. 1998.. Use of image-based information in judgments of surface-reflectance properties. . J. Opt. Soc. Am. A 15::2951–65
     [Crossref] [Google Scholar]
  65. Olkkonen M, Brainard DH. 2010.. Perceived glossiness and lightness under real-world illumination. . J. Vis. 10:(9):5
     [Crossref] [Google Scholar]
  66. Olkkonen M, Brainard DH. 2011.. Joint effects of illumination geometry and object shape in the perception of surface reflectance. . i-Perception 2:(9):1014–34
     [Crossref] [Google Scholar]
  67. Oxholm G, Nishino K. 2015.. Shape and reflectance estimation in the wild. . IEEE Trans. Pattern Anal. Mach. Intel. 38:(2):376–89
     [Crossref] [Google Scholar]
  68. Pont SC, te Pas SF. 2006.. Material-illumination ambiguities and the perception of solid objects. . Perception 35:(10):1331–50
     [Crossref] [Google Scholar]
  69. Ramachandran V. 1988.. Perception of shape from shading. . Nature 331::163–66
     [Crossref] [Google Scholar]
  70. Shimokawa T, Nishio A, Sato M, Kawato M, Komatsu H. 2019.. Computational model for human 3D shape perception from a single specular image. . Front. Comp. Neurol. 13::10
     [Crossref] [Google Scholar]
  71. Storrs KS, Anderson BL, Fleming RW. 2021.. Unsupervised learning predicts human perception and misperception of gloss. . Nat. Hum. Behav. 5::1402–17
     [Crossref] [Google Scholar]
  72. Todd JT. 2004.. The visual perception of 3D shape. . Trends Cogn. Sci. 8:(3):115–21
     [Crossref] [Google Scholar]
  73. Todd JT, Egan EGL, Phillips F. 2014.. Is the perception of 3D shape from shading based on assumed reflectance and illumination?. i-Perception 5:(6):497–514
     [Crossref] [Google Scholar]
  74. Todd JT, Koenderink JJ, van Doorn AJ, Kappers AML. 1996.. Effect of changing viewing conditions on the perceived structure of smoothly curved surfaces. . J. Exp. Psychol. Hum. Percept. Perform. 22::695–706
     [Crossref] [Google Scholar]
  75. Todd JT, Norman JF. 1995.. The visual discrimination of relative surface orientation. . Perception 24:(8):855–66
     [Crossref] [Google Scholar]
  76. Todd JT, Norman JF. 2003.. The visual perception of 3-D shape from multiple cues: Are observers capable of perceiving metric structure?. Percept. Psychophys. 61::1269–86
     [Google Scholar]
  77. Todd JT, Norman JF, Mingolla E. 2004.. Lightness constancy in the presence of specular highlights. . Psychol. Sci. 15:(1):33–39
     [Crossref] [Google Scholar]
  78. Todd JT, Reichel FD. 1989.. Ordinal structure in the visual perception and cognition of smoothly curved surfaces. . Psychol. Rev. 96::643–57
     [Crossref] [Google Scholar]
  79. Todorović D. 2014.. How shape from contours affects shape from shading. . Vis. Res. 103::1–10
     [Crossref] [Google Scholar]
  80. Vangorp P, Laurijssen J, Dutré P. 2007.. The influence of shape on the perception of material reflectance. . ACM Trans. Graph. 26::77
     [Crossref] [Google Scholar]
  81. Wilder JD, Adams WJ, Murray RF. 2019.. Shape from shading under inconsistent illumination. . J. Vis. 19:(6):2
     [Crossref] [Google Scholar]
  82. Xiao B, Gkioulekas I, Adelson EH, Zickler T, Bala K. 2014.. Looking against the light: how perception of translucency depends on lighting direction. . J. Vis. 14:(3):17
     [Crossref] [Google Scholar]
  83. Xiong Y, Chakrabarti A, Basri R, Gortler SJ, Jacobs DW, Zickler T. 2013.. From shading to local shape. . arXiv:1310.2916 [cs.CV]
/content/journals/10.1146/annurev-vision-102122-094213
Loading
Interactions Between 3D Surface Shape and Material Perception
Annual Review of Vision Science 10, 69 (2024); https://doi.org/10.1146/annurev-vision-102122-094213
/content/journals/10.1146/annurev-vision-102122-094213
/content/journals/10.1146/annurev-vision-102122-094213
Loading

Data & Media loading...

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error