1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Control, Robotics, and Autonomous Systems
  4. Volume 6, 2023
  5. Article

Annual Review of Control, Robotics, and Autonomous Systems

Volume 6, 2023

Grasp Learning: Models, Methods, and Performance

Abstract

Grasp learning has become an exciting and important topic in robotics. Just a few years ago, the problem of grasping novel objects from unstructured piles of clutter was considered a serious research challenge. Now, it is a capability that is quickly being incorporated into industrial supply chain automation. How did that happen? What is the current state of the art in robotic grasp learning, what are the different methodological approaches, and what machine learning models are used? This review attempts to give an overview of the current state of the art of grasp learning research.

Keyword(s): grasp learning, grasping, machine learning, manipulation
Loading

Article metrics loading...

/content/journals/10.1146/annurev-control-062122-025215
2023-05-03
2024-04-16
Loading full text...

Full text loading...

/deliver/fulltext/control/6/1/annurev-control-062122-025215.html?itemId=/content/journals/10.1146/annurev-control-062122-025215&mimeType=html&fmt=ahah

Literature Cited

  1. 1.
    Murray RM, Li Z, Sastry SS. 1994. A Mathematical Introduction to Robotic Manipulation Boca Raton, FL: CRC
  2. 2.
    Mason MT. 2001. Mechanics of Robotic Manipulation Cambridge, MA: MIT Press
  3. 3.
    Lenz I, Lee H, Saxena A 2013. Deep learning for detecting robotic grasps. Robotics: Science and Systems IX P Newman, D Fox, D Hsu, pap. 12. N.p.: Robot. Sci. Syst. Found .
     [Google Scholar]
  4. 4.
    Lenz I, Lee H, Saxena A. 2015. Deep learning for detecting robotic grasps. Int. J. Robot. Res. 34:705–24
     [Google Scholar]
  5. 5.
    Wyrobek K. 2017. The origin story of ROS, the Linux of robotics. IEEE Spectrum Oct. 31. https://spectrum.ieee.org/the-origin-story-of-ros-the-linux-of-robotics
    [Google Scholar]
  6. 6.
    Pinto L, Gupta A. 2015. Supersizing self-supervision: learning to grasp from 50K tries and 700 robot hours. arXiv:1509.06825 [cs.LG]
  7. 7.
    Krizhevsky A, Sutskever I, Hinton GE 2012. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25 F Pereira, CJ Burges, L Bottou, KQ Weinberger 1097–105 Red Hook, NY: Curran
     [Google Scholar]
  8. 8.
    Mahler J, Liang J, Niyaz S, Laskey M, Doan R et al. 2017. Dex-Net 2.0: deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv:1703.09312 [cs.RO]
  9. 9.
    Rubinstein RY. 1997. Optimization of computer simulation models with rare events. Eur. J. Oper. Res. 99:89–112
     [Google Scholar]
  10. 10.
    Redmon J, Angelova A. 2015. Real-time grasp detection using convolutional neural networks. 2015 IEEE International Conference on Robotics and Automation (ICRA)1316–22 Piscataway, NJ: IEEE
     [Google Scholar]
  11. 11.
    Kumra S, Kanan C. 2017. Robotic grasp detection using deep convolutional neural networks. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)769–76 Piscataway, NJ: IEEE
     [Google Scholar]
  12. 12.
    Ren S, He K, Girshick R, Sun J 2015. Faster R-CNN: towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems 28 C Cortes, N Lawrence, D Lee, M Sugiyama, R Garnett 91–99 Red Hook, NY: Curran
     [Google Scholar]
  13. 13.
    Johns E, Leutenegger S, Davison AJ. 2016. Deep learning a grasp function for grasping under gripper pose uncertainty. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)4461–68 Piscataway, NJ: IEEE
     [Google Scholar]
  14. 14.
    Zhou X, Lan X, Zhang H, Tian Z, Zhang Y, Zheng N. 2018. Fully convolutional grasp detection network with oriented anchor box. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)7223–30 Piscataway, NJ: IEEE
     [Google Scholar]
  15. 15.
    He K, Zhang X, Ren S, Sun J. 2016. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)770–78 Piscataway, NJ: IEEE
     [Google Scholar]
  16. 16.
    Chu FJ, Xu R, Vela PA. 2018. Real-world multiobject, multigrasp detection. IEEE Robot. Autom. Lett. 3:3355–62
     [Google Scholar]
  17. 17.
    Morrison D, Corke P, Leitner J. 2018. Closing the loop for robotic grasping: a real-time, generative grasp synthesis approach. arXiv:1804.05172 [cs.RO]
  18. 18.
    Zeng A, Song S, Yu KT, Donlon E, Hogan FR et al. 2018. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. 2018 IEEE International Conference on Robotics and Automation (ICRA)3750–57 Piscataway, NJ: IEEE
     [Google Scholar]
  19. 19.
    Asif U, Tang J, Harrer S 2018. GraspNet: an efficient convolutional neural network for real-time grasp detection for low-powered devices. IJCAI'18: Proceedings of the 27th International Joint Conference on Artificial Intelligence J Lang 4875–82 Palo Alto, CA: AAAI Press
     [Google Scholar]
  20. 20.
    Satish V, Mahler J, Goldberg K. 2019. On-policy dataset synthesis for learning robot grasping policies using fully convolutional deep networks. IEEE Robot. Autom. Lett. 4:1357–64
     [Google Scholar]
  21. 21.
    Mahler J, Pokorny FT, Hou B, Roderick M, Laskey M et al. 2016. Dex-Net 1.0: a cloud-based network of 3D objects for robust grasp planning using a multi-armed bandit model with correlated rewards. 2016 IEEE international conference on robotics and automation (ICRA)1957–64 Piscataway, NJ: IEEE
     [Google Scholar]
  22. 22.
    Kumra S, Joshi S, Sahin F. 2020. Antipodal robotic grasping using generative residual convolutional neural network. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)9626–33 Piscataway, NJ: IEEE
     [Google Scholar]
  23. 23.
    Zhu X, Wang D, Biza O, Su G, Walters R, Platt R 2022. Sample efficient grasp learning using equivariant models. Robotics: Science and Systems XVIII K Hauser, D Shell, S Huang, pap. 71. N.p.: Robot. Sci. Syst. Found .
     [Google Scholar]
  24. 24.
    Zeng A, Song S, Welker S, Lee J, Rodriguez A, Funkhouser T. 2018. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)4238–45 Piscataway, NJ: IEEE
     [Google Scholar]
  25. 25.
    Levine S, Pastor P, Krizhevsky A, Quillen D. 2016. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. arXiv:1603.02199 [cs.LG]
  26. 26.
    Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D. 2018. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int. J. Robot. Res. 37:421–36
     [Google Scholar]
  27. 27.
    Sutton R, Barto A. 1998. Reinforcement Learning: An Introduction Cambridge, MA: MIT Press
  28. 28.
    Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I et al. 2013. Playing Atari with deep reinforcement learning. arXiv:1312.5602 [cs.LG]
  29. 29.
    Kalashnikov D, Irpan A, Pastor P, Ibarz J, Herzog A et al. 2018. QT-Opt: scalable deep reinforcement learning for vision-based robotic manipulation. arXiv:1806.10293 [cs.LG]
  30. 30.
    Viereck U, ten Pas A, Saenko K, Platt R 2017. Learning a visuomotor controller for real world robotic grasping using simulated depth images. Proceedings of the 1st Annual Conference on Robot Learning S Levine, V Vanhoucke, K Goldberg 291–300 Proc. Mach. Learn. Res. 78. N.p.: PMLR
    [Google Scholar]
  31. 31.
    Gualtieri M, ten Pas A, Saenko K, Platt R. 2016. High precision grasp pose detection in dense clutter. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)598–605 Piscataway, NJ: IEEE
     [Google Scholar]
  32. 32.
    ten Pas A, Gualtieri M, Saenko K, Platt R. 2017. Grasp pose detection in point clouds. Int. J. Robot. Res. 36:1455–73
     [Google Scholar]
  33. 33.
    Liang H, Ma X, Li S, Görner M, Tang S et al. 2019. PointNetGPD: detecting grasp configurations from point sets. 2019 International Conference on Robotics and Automation (ICRA)3629–35 Piscataway, NJ: IEEE
     [Google Scholar]
  34. 34.
    Mousavian A, Eppner C, Fox D. 2019. 6-DOF GraspNet: variational grasp generation for object manipulation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)2901–10 Piscataway, NJ: IEEE
     [Google Scholar]
  35. 35.
    Kingma DP, Welling M. 2013. Auto-encoding variational bayes. arXiv:1312.6114 [stat.ML]
  36. 36.
    Qi CR, Yi L, Su H, Guibas LJ 2017. PointNet++: deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems 30 I Guyon, U Von Luxburg, S Bengio, H Wallach, R Fergus et al.5100–9 Red Hook, NY: Curran
     [Google Scholar]
  37. 37.
    Murali A, Mousavian A, Eppner C, Paxton C, Fox D. 2020. 6-DOF grasping for target-driven object manipulation in clutter. 2020 IEEE International Conference on Robotics and Automation (ICRA)6232–38 Piscataway, NJ: IEEE
     [Google Scholar]
  38. 38.
    Charles RQ, Su H, Kaichun M, Guibas LJ. 2017. PointNet: deep learning on point sets for 3D classification and segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)77–85 Piscataway, NJ: IEEE
     [Google Scholar]
  39. 39.
    Qin Y, Chen R, Zhu H, Song M, Xu J, Su H 2020. S4G: amodal single-view single-shot SE(3) grasp detection in cluttered scenes. Proceedings of the Conference on Robot Learning LP Kaelbling, D Kragic, K Sugiura 53–65 Proc. Mach. Learn. Res. 100. N.p.: PMLR
    [Google Scholar]
  40. 40.
    Wu C, Chen J, Cao Q, Zhang J, Tai Y et al. 2020. Grasp proposal networks: an end-to-end solution for visual learning of robotic grasps. Advances in Neural Information Processing Systems 33 H Larochelle, M Ranzato, R Hadsell, MF Balcan, H Lin 13174–84 Red Hook, NY: Curran
     [Google Scholar]
  41. 41.
    Sundermeyer M, Mousavian A, Triebel R, Fox D. 2021. Contact-GraspNet: efficient 6-DOF grasp generation in cluttered scenes. 2021 IEEE International Conference on Robotics and Automation (ICRA)13438–44 Piscataway, NJ: IEEE
     [Google Scholar]
  42. 42.
    Zhao B, Zhang H, Lan X, Wang H, Tian Z, Zheng N. 2021. REGNet: region-based grasp network for end-to-end grasp detection in point clouds. 2021 IEEE International Conference on Robotics and Automation (ICRA)13474–80 Piscataway, NJ: IEEE
     [Google Scholar]
  43. 43.
    Wei W, Luo Y, Li F, Xu G, Zhong J et al. 2021. GPR: grasp pose refinement network for cluttered scenes. 2021 IEEE International Conference on Robotics and Automation (ICRA)4295–302 Piscataway, NJ: IEEE
     [Google Scholar]
  44. 44.
    Chen H, Liu S, Chen W, Li H, Hill R. 2021. Equivariant point network for 3D point cloud analysis. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)14509–18 Piscataway, NJ: IEEE
     [Google Scholar]
  45. 45.
    Breyer M, Chung JJ, Ott L, Siegwart R, Nieto J 2021. Volumetric Grasping Network: real-time 6 DOF grasp detection in clutter. Proceedings of the 2020 Conference on Robot Learning J Kober, F Ramos, C Tomlin 1602–11 Proc. Mach. Learn. Res. 155. N.p.: PMLR
    [Google Scholar]
  46. 46.
    Cai J, Cen J, Wang H, Wang MY. 2022. Real-time collision-free grasp pose detection with geometry-aware refinement using high-resolution volume. IEEE Robot. Autom. Lett. 7:1888–95
     [Google Scholar]
  47. 47.
    James S, Wada K, Laidlow T, Davison AJ. 2022. Coarse-to-fine Q-attention: efficient learning for visual robotic manipulation via discretisation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)pp. 13729–38 Piscataway, NJ: IEEE
     [Google Scholar]
  48. 48.
    Varley J, DeChant C, Richardson A, Ruales J, Allen P. 2017. Shape completion enabled robotic grasping. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)2442–47 Piscataway, NJ: IEEE
     [Google Scholar]
  49. 49.
    Lorensen WE, Cline HE 1987. Marching cubes: a high resolution 3D surface construction algorithm. SIGGRAPH '87: Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques MC Stone 163–69 New York: ACM
     [Google Scholar]
  50. 50.
    Miller AT, Allen PK. 2004. GraspIt! A versatile simulator for robotic grasping. IEEE Robot. Autom. Mag. 11:4110–22
     [Google Scholar]
  51. 51.
    Lundell J, Verdoja F, Kyrki V. 2019. Robust grasp planning over uncertain shape completions. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)1526–32 Piscataway, NJ: IEEE
     [Google Scholar]
  52. 52.
    Lundell J, Verdoja F, Kyrki V. 2020. Beyond top-grasps through scene completion. 2020 IEEE International Conference on Robotics and Automation (ICRA)545–51 Piscataway, NJ: IEEE
     [Google Scholar]
  53. 53.
    Yang D, Tosun T, Eisner B, Isler V, Lee D. 2021. Robotic grasping through combined image-based grasp proposal and 3D reconstruction. 2021 IEEE International Conference on Robotics and Automation (ICRA)6350–56 Piscataway, NJ: IEEE
     [Google Scholar]
  54. 54.
    Van der Merwe M, Lu Q, Sundaralingam B, Matak M, Hermans T. 2020. Learning continuous 3D reconstructions for geometrically aware grasping. 2020 IEEE International Conference on Robotics and Automation (ICRA)11516–22 Piscataway, NJ: IEEE
     [Google Scholar]
  55. 55.
    Jiang Z, Zhu Y, Svetlik M, Fang K, Zhu Y 2021. Synergies between affordance and geometry: 6-DoF grasp detection via implicit representations. Robotics: Science and Systems XVII D Shell, M Toussaint, MA Hsieh, pap. 24. N.p.: Robot. Sci. Syst. Found .
     [Google Scholar]
  56. 56.
    Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R 2020. NeRF: representing scenes as neural radiance fields for view synthesis. Computer Vision – ECCV 2020 A Vedaldi, H Bischof, T Brox, JM Frahm 405–21 Cham, Switz: Springer
     [Google Scholar]
  57. 57.
    Wu W, Qi Z, Fuxin L. 2019. PointConv: deep convolutional networks on 3D point clouds. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)9613–22 Piscataway, NJ: IEEE
     [Google Scholar]
  58. 58.
    Berscheid L, Friedrich C, Kröger T. 2021. Robot learning of 6 DoF grasping using model-based adaptive primitives. 2021 IEEE International Conference on Robotics and Automation (ICRA)4474–80 Piscataway, NJ: IEEE
     [Google Scholar]
  59. 59.
    Kasaei H, Kasaei M. 2021. MVGrasp: real-time multi-view 3D object grasping in highly cluttered environments. arXiv:2103.10997 [cs.RO]
  60. 60.
    Mahler J, Platt R, Rodriguez A, Ciocarlie M, Dollar A et al. 2018. Guest editorial open discussion of robot grasping benchmarks, protocols, and metrics. IEEE Trans. Autom. Sci. Eng. 15:1440–42
     [Google Scholar]
  61. 61.
    Morrison D, Corke P, Leitner J. 2020. EGAD! An Evolved Grasping Analysis Dataset for diversity and reproducibility in robotic manipulation. IEEE Robot. Autom. Lett. 5:4368–75
     [Google Scholar]
  62. 62.
    Jiang Y, Moseson S, Saxena A. 2011. Efficient grasping from RGBD images: learning using a new rectangle representation. 2011 IEEE International Conference on Robotics and Automation3304–11 Piscataway, NJ: IEEE
     [Google Scholar]
  63. 63.
    Depierre A, Dellandréa E, Chen L 2018. Jacquard: a large scale dataset for robotic grasp detection. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)3511–16 Piscataway, NJ: IEEE
     [Google Scholar]
  64. 64.
    Goldfeder C, Ciocarlie M, Dang H, Allen PK. 2009. The Columbia Grasp Database. 2009 IEEE International Conference on Robotics and Automation (ICRA)1710–16 Piscataway, NJ: IEEE
     [Google Scholar]
  65. 65.
    Shilane P, Min P, Kazhdan M, Funkhouser T. 2004. The Princeton Shape Benchmark. Proceedings of the International Conference on Shape Modeling Applications167–78 Piscataway, NJ: IEEE
     [Google Scholar]
  66. 66.
    Miller A, Allen P, Santos V, Valero-Cuevas F. 2005. From robotic hands to human hands: a visualization and simulation engine for grasping research. Ind. Robot 32:55–63
     [Google Scholar]
  67. 67.
    Calli B, Walsman A, Singh A, Srinivasa S, Abbeel P, Dollar AM. 2015. Benchmarking in manipulation research: the YCB object and model set and benchmarking protocols. arXiv:1502.03143 [cs.RO]
  68. 68.
    Berkeley AUTOLAB 2022. Dex-Net. https://berkeleyautomation.github.io/dex-net
  69. 69.
    Fang HS, Wang C, Gou M, Lu C. 2020. GraspNet-1Billion: a large-scale benchmark for general object grasping. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition11441–50 Piscataway, NJ: IEEE
     [Google Scholar]
  70. 70.
    Eppner C, Mousavian A, Fox D. 2021. ACRONYM: a large-scale grasp dataset based on simulation. 2021 IEEE International Conference on Robotics and Automation (ICRA)6222–27 Piscataway, NJ: IEEE
     [Google Scholar]
  71. 71.
    Savva M, Chang AX, Hanrahan P. 2015. Semantically-enriched 3D models for common-sense knowledge. 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops24–31 Piscataway, NJ: IEEE
     [Google Scholar]
  72. 72.
    Macklin M, Müller M, Chentanez N, Kim TY. 2014. Unified particle physics for real-time applications. ACM Trans. Graph. 33:153
     [Google Scholar]
/content/journals/10.1146/annurev-control-062122-025215
Loading
Grasp Learning: Models, Methods, and Performance
Annual Review of Control, Robotics, and Autonomous Systems 6, 363 (2023); https://doi.org/10.1146/annurev-control-062122-025215
/content/journals/10.1146/annurev-control-062122-025215
/content/journals/10.1146/annurev-control-062122-025215
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error