1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Control, Robotics, and Autonomous Systems
  4. Volume 2, 2019
  5. Article

Annual Review of Control, Robotics, and Autonomous Systems

Volume 2, 2019

From Visual Understanding to Complex Object Manipulation

Abstract

Planning and executing object manipulation requires integrating multiple sensory and motor channels while acting under uncertainty and complying with task constraints. As the modern environment is tuned for human hands, designing robotic systems with similar manipulative capabilities is crucial. Research on robotic object manipulation is divided into smaller communities interested in, e.g., motion planning, grasp planning, sensorimotor learning, and tool use. However, few attempts have been made to combine these areas into holistic systems. In this review, we aim to unify the underlying mechanics of grasping and in-hand manipulation by focusing on the temporal aspects of manipulation, including visual perception, grasp planning and execution, and goal-directed manipulation. Inspired by human manipulation, we envision that an emphasis on the temporal integration of these processes opens the way for human-like object use by robots.

Keyword(s): grasping, in-hand manipulation, task planning
Loading

Article metrics loading...

/content/journals/10.1146/annurev-control-053018-023735
2019-05-03
2024-05-12
Loading full text...

Full text loading...

/deliver/fulltext/control/2/1/annurev-control-053018-023735.html?itemId=/content/journals/10.1146/annurev-control-053018-023735&mimeType=html&fmt=ahah

Literature Cited

  1. 1.  Johansson RS, Westling G, Bäckström A, Flanagan JR 2001. Eye–hand coordination in object manipulation. J. Neurosci. 21:6917–32
     [Google Scholar]
  2. 2.  Terao Y, Andersson NM, Flanagan JR, Johansson RS 2002. Engagement of gaze in capturing targets for future sequential manual actions. J. Neurophysiol. 88:1716–25
     [Google Scholar]
  3. 3.  van Zoest W, Donk M, Theeuwes J 2004. The role of stimulus-driven and goal-driven control in saccadic visual selection. J. Exp. Psychol. Hum. Percept. Perform. 30:746–59
     [Google Scholar]
  4. 4.  Schütz-Bosbach S, Prinz W 2007. Perceptual resonance: action-induced modulation of perception. Trends Cogn. Sci. 11:349–55
     [Google Scholar]
  5. 5.  von Hofsten C, Lindhagen K 1979. Observations on the development of reaching for moving objects. J. Exp. Child Psychol. 28:158–73
     [Google Scholar]
  6. 6.  Gilster R, Hesse C, Deubel H 2012. Contact points during multidigit grasping of geometric objects. Exp. Brain Res. 217:137–51
     [Google Scholar]
  7. 7.  Feix T, Romero J, Schmiedmayer HB, Dollar AM, Kragic D 2016. The grasp taxonomy of human grasp types. IEEE Trans. Hum.-Mach. Syst. 46:66–77
     [Google Scholar]
  8. 8.  Feix T, Bullock IM, Dollar AM 2014. Analysis of human grasping behavior: object characteristics and grasp type. IEEE Trans. Haptics 7:311–23
     [Google Scholar]
  9. 9.  Feix T, Bullock IM, Dollar AM 2014. Analysis of human grasping behavior: correlating tasks, objects and grasps. IEEE Trans. Haptics 7:430–41
     [Google Scholar]
  10. 10.  Ghez C, Krakauer J 2000. The organization of movement. Principles of Neural Science ER Kandel, JH Schwartz, TM Jessell653–73 New York: McGraw-Hill. 4th ed.
     [Google Scholar]
  11. 11.  Morasso P 1981. Spatial control of arm movements. Exp. Brain Res. 42:223–27
     [Google Scholar]
  12. 12.  Flash T, Hogan N 1985. The coordination of arm movements: an experimentally confirmed mathematical model. J. Neurosci. 5:1688–703
     [Google Scholar]
  13. 13.  Nelson WL 1983. Physical principles for economies of skilled movements. Biol. Cybern. 46:135–47
     [Google Scholar]
  14. 14.  Uno Y, Kawato M, Suzuki R 1989. Formation and control of optimal trajectory in human multijoint arm movement. Biol. Cybern. 61:89–101
     [Google Scholar]
  15. 15.  Rosenbaum DA, Meulenbroek RJ, Vaughan J, Jansen C 2001. Posture-based motion planning: applications to grasping. Psychol. Rev. 108:709
     [Google Scholar]
  16. 16.  Herbort O, Butz MV 2010. Planning and control of hand orientation in grasping movements. Exp. Brain Res. 202:867–78
     [Google Scholar]
  17. 17.  ElKoura G, Singh K 2003. Handrix: animating the human hand. Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation110–19 Aire-la-Ville, Switz.: Eurograph. Assoc.
     [Google Scholar]
  18. 18.  Santello M, Baud-Bovy G, Jörntell H 2013. Neural bases of hand synergies. Front. Comput. Neurosci. 7:23
     [Google Scholar]
  19. 19.  Todorov E, Ghahramani Z 2004. Analysis of the synergies underlying complex hand manipulation. The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society 24637–40 New York: IEEE
     [Google Scholar]
  20. 20.  Penfield W, Boldrey E 1937. Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain 60:389–443
     [Google Scholar]
  21. 21.  Johansson RS, Flanagan JR 2009. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nat. Rev. Neurosci. 10:345–59
     [Google Scholar]
  22. 22.  Orban GA, Caruana F 2014. The neural basis of human tool use. Front. Psychol. 5:310
     [Google Scholar]
  23. 23.  Gibson JJ 2014. The Ecological Approach to Visual Perception New York: Psychol. Press
  24. 24.  van Leeuwen L, Smitsman A, van Leeuwen C 1994. Affordances, perceptual complexity, and the development of tool use. J. Exp. Psychol. Hum. Percept. Perform. 20:174–91
     [Google Scholar]
  25. 25.  Osiurak F, Jarry C, Le Gall D 2010. Grasping the affordances, understanding the reasoning: toward a dialectical theory of human tool use. Psychol. Rev. 117:517–40
     [Google Scholar]
  26. 26.  Sahbani A, El-Khoury S, Bidaud P 2012. An overview of 3D object grasp synthesis algorithms. Robot. Auton. Syst. 60:326–36
     [Google Scholar]
  27. 27.  Bohg J, Morales A, Asfour T, Kragic D 2014. Data-driven grasp synthesis—a survey. IEEE Trans. Robot. 30:289–309
     [Google Scholar]
  28. 28.  Miller AT, Knoop S, Christensen HI, Allen PK 2003. Automatic grasp planning using shape primitives. 2003 IEEE International Conference on Robotics and Automation 21824–29 New York: IEEE
     [Google Scholar]
  29. 29.  Huebner K, Kragic D 2008. Selection of robot pre-grasps using box-based shape approximation. 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems1765–70 New York: IEEE
     [Google Scholar]
  30. 30.  Przybylski M, Asfour T, Dillmann R 2011. Planning grasps for robotic hands using a novel object representation based on the medial axis transform. 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems1781–88 New York: IEEE
     [Google Scholar]
  31. 31.  Kamon I, Flash T, Edelman S 1996. Learning to grasp using visual information. Proceedings of IEEE International Conference on Robotics and Automation 32470–76 New York: IEEE
     [Google Scholar]
  32. 32.  Maitin-Shepard J, Cusumano-Towner M, Lei J, Abbeel P 2010. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. 2010 IEEE International Conference on Robotics and Automation2308–15 New York: IEEE
     [Google Scholar]
  33. 33.  Lenz I, Lee H, Saxena A 2015. Deep learning for detecting robotic grasps. Int. J. Robot. Res. 34:705–24
     [Google Scholar]
  34. 34.  Jiang Y, Moseson S, Saxena A 2011. Efficient grasping from RGBD images: learning using a new rectangle representation. 2011 IEEE International Conference on Robotics and Automation3304–11 New York: IEEE
     [Google Scholar]
  35. 35.  Pinto L, Gupta A 2016. Supersizing self-supervision: learning to grasp from 50K tries and 700 robot hours. 2016 IEEE International Conference on Robotics and Automation (ICRA)3406–13 New York: IEEE
     [Google Scholar]
  36. 36.  Mahler J, Liang J, Niyaz S, Laskey M, Doan R et al. 2017. Dex-Net 2.0: deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. Robotics: Science and Systems XIII N Amato, S Srinivasa, N Ayanian, S Kuindersma chap. 58 N.p.: Robot. Sci. Syst. Found.
     [Google Scholar]
  37. 37.  Elbanhawi M, Simic M 2014. Sampling-based robot motion planning: a review. IEEE Access 2:56–77
     [Google Scholar]
  38. 38.  Fontanals J, Dang-Vu BA, Porges O, Rosell J, Roa MA 2014. Integrated grasp and motion planning using independent contact regions. 2014 IEEE-RAS International Conference on Humanoid Robots887–93 New York: IEEE
     [Google Scholar]
  39. 39.  Roa MA, Suarez R 2009. Computation of independent contact regions for grasping 3-D objects. IEEE Trans. Robot. 25:839–50
     [Google Scholar]
  40. 40.  Vahrenkamp N, Asfour T, Dillmann R 2012. Simultaneous grasp and motion planning: humanoid robot ARMAR-III. IEEE Robot. Autom. Mag. 19:43–57
     [Google Scholar]
  41. 41.  Haustein JA, Hang K, Kragic D 2017. Integrating motion and hierarchical fingertip grasp planning. 2017 IEEE International Conference on Robotics and Automation (ICRA)3439–46 New York: IEEE
     [Google Scholar]
  42. 42.  Hang K, Stork JA, Kragic D 2014. Hierarchical Fingertip Space for multi-fingered precision grasping. 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems1641–48 New York: IEEE
     [Google Scholar]
  43. 43.  Roa MA, Suárez R 2014. Grasp quality measures: review and performance. Auton. Robots 38:65–88
     [Google Scholar]
  44. 44.  Rubert C, León B, Morales A, Sancho-Bru J 2018. Characterisation of grasp quality metrics. J. Intell. Robot. Syst. 89:319–42
     [Google Scholar]
  45. 45.  Rodriguez A, Mason MT, Ferry S 2012. From caging to grasping. Int. J. Robot. Res. 31:886–900
     [Google Scholar]
  46. 46.  Varava A, Kragic D, Pokorny FT 2016. Caging grasps of rigid and partially deformable 3-D objects with double fork and neck features. IEEE Trans. Robot. 32:1479–97
     [Google Scholar]
  47. 47.  Hsiao K, Kaelbling LP, Lozano-Pérez T 2011. Robust grasping under object pose uncertainty. Auton. Robots 31:253–68
     [Google Scholar]
  48. 48.  Yousef H, Boukallel M, Althoefer K 2011. Tactile sensing for dexterous in-hand manipulation in robotics—a review. Sens. Actuators A 167:171–87
     [Google Scholar]
  49. 49.  Kappassov Z, Corrales JA, Perdereau V 2015. Tactile sensing in dexterous robot hands—review. Robot. Auton. Syst. 74:195–220
     [Google Scholar]
  50. 50.  Dang H, Allen PK 2013. Grasp adjustment on novel objects using tactile experience from similar local geometry. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems4007–12 New York: IEEE
     [Google Scholar]
  51. 51.  Romano JM, Hsiao K, Niemeyer G, Chitta S, Kuchenbecker KJ 2011. Human-inspired robotic grasp control with tactile sensing. IEEE Trans. Robot. 27:1067–79
     [Google Scholar]
  52. 52.  Sommer N, Billard A 2016. Multi-contact haptic exploration and grasping with tactile sensors. Robot. Auton. Syst. 85:48–61
     [Google Scholar]
  53. 53.  Maria GD, Natale C, Pirozzi S 2013. Slipping control through tactile sensing feedback. 2013 IEEE International Conference on Robotics and Automation3523–28 New York: IEEE
     [Google Scholar]
  54. 54.  Hang K, Li M, Stork JA, Bekiroglu Y, Pokorny FT et al. 2016. Hierarchical fingertip space: a unified framework for grasp planning and in-hand grasp adaptation. IEEE Trans. Robot. 32:960–72
     [Google Scholar]
  55. 55.  Bicchi A 2000. Hands for dexterous manipulation and robust grasping: a difficult road toward simplicity. IEEE Trans. Robot. Autom. 16:652–62
     [Google Scholar]
  56. 56.  Ozawa R, Tahara K 2017. Grasp and dexterous manipulation of multi-fingered robotic hands: a review from a control view point. Adv. Robot. 31:1030–50
     [Google Scholar]
  57. 57.  Tournassoud P, Lozano-Perez T, Mazer E 1987. Regrasping. Proceedings of the 1987 IEEE International Conference on Robotics and Automation 41924–28 New York: IEEE
     [Google Scholar]
  58. 58.  Furukawa N, Namiki A, Taku S, Ishikawa M 2006. Dynamic regrasping using a high-speed multifingered hand and a high-speed vision system. Proceedings of the 2006 IEEE International Conference on Robotics and Automation181–87 New York: IEEE
     [Google Scholar]
  59. 59.  Shi J, Woodruff JZ, Umbanhowar PB, Lynch KM 2017. Dynamic in-hand sliding manipulation. IEEE Trans. Robot. 33:778–95
     [Google Scholar]
  60. 60.  Sundaralingam B, Hermans T 2017. Relaxed-rigidity constraints: in-grasp manipulation using purely kinematic trajectory optimization. Robotics: Science and Systems XIII N Amato, S Srinivasa, N Ayanian, S Kuindersma chap. 15 N.p.: Robot. Sci. Syst. Found.
     [Google Scholar]
  61. 61.  Antonova R, Cruciani S, Smith C, Kragic D 2017. Reinforcement learning for pivoting task. arXiv:1703.00472 [cs.RO]
  62. 62.  Sintov A, Tslil O, Shapiro A 2016. Robotic swing-up regrasping manipulation based on the impulse–momentum approach and CLQR control. IEEE Trans. Robot. 32:1079–90
     [Google Scholar]
  63. 63.  Viña B FE, Karayiannidis Y, Smith C, Kragic D 2016. Adaptive control for pivoting with visual and tactile feedback. 2016 IEEE International Conference on Robotics and Automation (ICRA)399–406 New York: IEEE
     [Google Scholar]
  64. 64.  Dafle NC, Rodriguez A, Paolini R, Tang B, Srinivasa SS et al. 2014. Extrinsic dexterity: in-hand manipulation with external forces. 2014 IEEE International Conference on Robotics and Automation (ICRA)1578–85 New York: IEEE
     [Google Scholar]
  65. 65.  Chavan-Dafle N, Rodriguez A 2015. Prehensile pushing: in-hand manipulation with push-primitives. 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)6215–22 New York: IEEE
     [Google Scholar]
  66. 66.  Holladay A, Paolini R, Mason MT 2015. A general framework for open-loop pivoting. 2015 IEEE International Conference on Robotics and Automation (ICRA)3675–81 New York: IEEE
     [Google Scholar]
  67. 67.  Cruciani S, Smith C, Kragic D, Hang K 2018. Dexterous manipulation graphs. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) New York: IEEE
     [Google Scholar]
  68. 68.  Chavan-Dafle N, Mason MT, Staab H, Rossano G, Rodriguez A 2015. A two-phase gripper to reorient and grasp. 2015 IEEE International Conference on Automation Science and Engineering (CASE)1249–55 New York: IEEE
     [Google Scholar]
  69. 69.  Bircher WG, Dollar AM, Rojas N 2017. A two-fingered robot gripper with large object reorientation range. 2017 IEEE International Conference on Robotics and Automation (ICRA)3453–60 New York: IEEE
     [Google Scholar]
  70. 70.  Kemp CC, Edsinger A 2006. Robot manipulation of human tools: autonomous detection and control of task relevant features Paper presented at the 5th International Conference on Development and Learning, Bloomington, IN, May 31–June 3
  71. 71.  Hoffmann H, Chen Z, Earl D, Mitchell D, Salemi B, Sinapov J 2014. Adaptive robotic tool use under variable grasps. Robot. Auton. Syst. 62:833–46
     [Google Scholar]
  72. 72.  Yamazaki K, Watanabe Y, Nagahama K, Okada K, Inaba M 2010. Recognition and manipulation integration for a daily assistive robot working on kitchen environments. 2010 IEEE International Conference on Robotics and Biomimetics196–201 New York: IEEE
     [Google Scholar]
  73. 73.  Seib V, Manthe S, Holzmann J, Memmesheimer R, Peters A et al. 2015. RoboCup 2015 - homer@UniKoblenz (Germany) Tech. Rep., Univ. Koblenz-Landau, Koblenz, Ger.
/content/journals/10.1146/annurev-control-053018-023735
Loading
From Visual Understanding to Complex Object Manipulation
Annual Review of Control, Robotics, and Autonomous Systems 2, 161 (2019); https://doi.org/10.1146/annurev-control-053018-023735
/content/journals/10.1146/annurev-control-053018-023735
/content/journals/10.1146/annurev-control-053018-023735
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error