1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Control, Robotics, and Autonomous Systems
  4. Volume 7, 2024
  5. Article

Annual Review of Control, Robotics, and Autonomous Systems

Volume 7, 2024

Machine Learning in Robotic Ultrasound Imaging: Challenges and Perspectives

Abstract

This article reviews recent advances in intelligent robotic ultrasound imaging systems. We begin by presenting the commonly employed robotic mechanisms and control techniques in robotic ultrasound imaging, along with their clinical applications. Subsequently, we focus on the deployment of machine learning techniques in the development of robotic sonographers, emphasizing crucial developments aimed at enhancing the intelligence of these systems. The methods for achieving autonomous action reasoning are categorized into two sets of approaches: those relying on implicit environmental data interpretation and those using explicit interpretation. Throughout this exploration, we also discuss practical challenges, including those related to the scarcity of medical data, the need for a deeper understanding of the physical aspects involved, and effective data representation approaches. We conclude by highlighting the open problems in the field and analyzing different possible perspectives on how the community could move forward in this research area.

Keyword(s): deep learning, ethics and regulations, learning from demonstration, medical robotics, registration, reinforcement learning, segmentation, ultrasound image analysis, ultrasound physics, ultrasound simulation
Loading

Article metrics loading...

/content/journals/10.1146/annurev-control-091523-100042
2024-07-10
2024-12-13
Loading full text...

Full text loading...

/deliver/fulltext/control/7/1/annurev-control-091523-100042.html?itemId=/content/journals/10.1146/annurev-control-091523-100042&mimeType=html&fmt=ahah

Literature Cited

  1. 1.
    Dupont PE, Nelson BJ, Goldfarb M, Hannaford B, Menciassi A, et al. 2021.. A decade retrospective of medical robotics research from 2010 to 2020. . Sci. Robot. 6:(60):eabi8017
     [Crossref] [Google Scholar]
  2. 2.
    Yang GZ, Cambias J, Cleary K, Daimler E, Drake J, et al. 2017.. Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy. . Sci. Robot. 2:(4):eaam8638
     [Crossref] [Google Scholar]
  3. 3.
    Yang GZ, Nelson BJ, Murphy RR, Choset H, Christensen H, et al. 2020.. Combating COVID-19—the role of robotics in managing public health and infectious diseases. . Sci. Robot. 5:(40):eabb5589
     [Crossref] [Google Scholar]
  4. 4.
    Yip M, Salcudean S, Goldberg K, Althoefer K, Menciassi A, et al. 2023.. Artificial intelligence meets medical robotics. . Science 381:(6654):141–46
     [Crossref] [Google Scholar]
  5. 5.
    Zemmar A, Lozano AM, Nelson BJ. 2020.. The rise of robots in surgical environments during COVID-19. . Nat. Mach. Intell. 2:(10):566–72
     [Crossref] [Google Scholar]
  6. 6.
    Jiang Z, Salcudean SE, Navab N. 2023.. Robotic ultrasound imaging: state-of-the-art and future perspectives. . Med. Image Anal. 89::102878
     [Crossref] [Google Scholar]
  7. 7.
    von Haxthausen F, Böttger S, Wulff D, Hagenah J, Garca-Vázquez V, Ipsen S. 2021.. Medical robotics for ultrasound imaging: current systems and future trends. . Curr. Robot. Rep. 2::55–71
     [Crossref] [Google Scholar]
  8. 8.
    Li K, Xu Y, Meng MQH. 2021.. An overview of systems and techniques for autonomous robotic ultrasound acquisitions. . IEEE Trans. Med. Robot. Bionics 3:(2):510–24
     [Crossref] [Google Scholar]
  9. 9.
    Navab N, Hennersperger C, Frisch B, Fürst B. 2016.. Personalized, relevance-based multimodal robotic imaging and augmented reality for computer assisted interventions. . Med. Image Anal. 33::64–71
     [Crossref] [Google Scholar]
  10. 10.
    Salcudean SE, Bell G, Bachmann S, Zhu WH, Abolmaesumi P, Lawrence PD. 1999.. Robot-assisted diagnostic ultrasound—design and feasibility experiments. . In Medical Image Computing and Computer-Assisted Intervention—MICCAI'99, ed. C Taylor, A Colchester , pp. 1062–71. Berlin:: Springer
     [Google Scholar]
  11. 11.
    Jiang Z, Grimm M, Zhou M, Esteban J, Simson W, et al. 2020.. Automatic normal positioning of robotic ultrasound probe based only on confidence map optimization and force measurement. . IEEE Robot. Autom. Lett. 5:(2):1342–49
     [Crossref] [Google Scholar]
  12. 12.
    Jiang Z, Grimm M, Zhou M, Hu Y, Esteban J, Navab N. 2020.. Automatic force-based probe positioning for precise robotic ultrasound acquisition. . IEEE Trans. Ind. Electron. 68:(11):11200–11
     [Crossref] [Google Scholar]
  13. 13.
    Mustafa ASB, Ishii T, Matsunaga Y, Nakadate R, Ishii H, et al. 2013.. Development of robotic system for autonomous liver screening using ultrasound scanning device. . In 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 804–9. Piscataway, NJ:: IEEE
     [Google Scholar]
  14. 14.
    Giuliani M, Szczęśniak-Stańczyk D, Mirnig N, Stollnberger G, Szyszko M, et al. 2020.. User-centred design and evaluation of a tele-operated echocardiography robot. . Health Technol. 10::649–65
     [Crossref] [Google Scholar]
  15. 15.
    Ma X, Zhang Z, Zhang HK. 2021.. Autonomous scanning target localization for robotic lung ultrasound imaging. . In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9467–74. Piscataway, NJ:: IEEE
     [Google Scholar]
  16. 16.
    Tan J, Li B, Li Y, Li B, Chen X, et al. 2022.. A flexible and fully autonomous breast ultrasound scanning system. . IEEE Trans. Autom. Sci. Eng. 20:(3):1920–33
     [Crossref] [Google Scholar]
  17. 17.
    Huang D, Bi Y, Navab N, Jiang Z. 2023.. Motion magnification in robotic sonography: enabling pulsation-aware artery segmentation. . In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway, NJ:: IEEE. Forthcoming
    [Google Scholar]
  18. 18.
    Zielke J, Eilers C, Busam B, Weber W, Navab N, Wendler T. 2022.. RSV: robotic sonography for thyroid volumetry. . IEEE Robot. Autom. Lett. 7:(2):3342–48
     [Crossref] [Google Scholar]
  19. 19.
    Esmaeeli S, Hrdlicka CM, Bastos AB, Wang J, Gomez-Paz S, et al. 2020.. Robotically assisted transcranial doppler with artificial intelligence for assessment of cerebral vasospasm after subarachnoid hemorrhage. . J. Neurocrit. Care 13:(1):32–40
     [Crossref] [Google Scholar]
  20. 20.
    Tirindelli M, Victorova M, Esteban J, Kim ST, Navarro-Alarcon D, et al. 2020.. Force-ultrasound fusion: bringing spine robotic-US to the next ``level. .'' IEEE Robot. Autom. Lett. 5:(4):5661–68
     [Crossref] [Google Scholar]
  21. 21.
    Virga S, Zettinig O, Esposito M, Pfister K, Frisch B, et al. 2016.. Automatic force-compliant robotic ultrasound screening of abdominal aortic aneurysms. . In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 508–13. Piscataway, NJ:: IEEE
     [Google Scholar]
  22. 22.
    Shida Y, Tsumura R, Watanabe T, Iwata H. 2021.. Heart position estimation based on bone distribution toward autonomous robotic fetal ultrasonography. . In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 11393–99. Piscataway, NJ:: IEEE
     [Google Scholar]
  23. 23.
    Stilli A, Dimitrakakis E, D'Ettorre C, Tran M, Stoyanov D. 2019.. Pneumatically attachable flexible rails for track-guided ultrasound scanning in robotic-assisted partial nephrectomy—a preliminary design study. . IEEE Robot. Autom. Lett. 4:(2):1208–15
     [Crossref] [Google Scholar]
  24. 24.
    Hungr N, Baumann M, Long JA, Troccaz J. 2012.. A 3-D ultrasound robotic prostate brachytherapy system with prostate motion tracking. . IEEE Trans. Robot. 28:(6):1382–97
     [Crossref] [Google Scholar]
  25. 25.
    Jiang Z, Gao Y, Xie L, Navab N. 2022.. Towards autonomous atlas-based ultrasound acquisitions in presence of articulated motion. . IEEE Robot. Autom. Lett. 7:(3):7423–30
     [Crossref] [Google Scholar]
  26. 26.
    Zhang T, Pang Y, Zeng T, Wang G, Yin S, et al. 2023.. Robotic drilling for the Chinese Chang'E 5 lunar sample-return mission. . Int. J. Robot. Res. 42:(8):586–613
     [Crossref] [Google Scholar]
  27. 27.
    Li G, Chen X, Zhou F, Liang Y, Xiao Y, et al. 2021.. Self-powered soft robot in the Mariana Trench. . Nature 591:(7848):66–71
     [Crossref] [Google Scholar]
  28. 28.
    Jiang Z, Bi Y, Zhou M, Hu Y, Burke M, Navab N. 2024.. Intelligent robotic sonographer: mutual information-based disentangled reward learning from few demonstrations. . Int. J. Robot. Res. https://doi.org/10.1177/02783649231223547
     [Google Scholar]
  29. 29.
    Baumgartner CF, Kamnitsas K, Matthew J, Fletcher TP, Smith S, et al. 2017.. SonoNet: real-time detection and localisation of fetal standard scan planes in freehand ultrasound. . IEEE Trans. Med. Imaging 36:(11):2204–15
     [Crossref] [Google Scholar]
  30. 30.
    Droste R, Drukker L, Papageorghiou AT, Noble JA. 2020.. Automatic probe movement guidance for freehand obstetric ultrasound. . In Medical Image Computing and Computer Assisted Intervention—MICCAI 2020, ed. AL Martel, P Abolmaesumi, D Stoyanov, D Mateus, MA Zuluaga , et al., pp. 583–92. Cham, Switz.:: Springer
     [Google Scholar]
  31. 31.
    Li T, Meng X, Tavakoli M. 2022.. Dual mode pHRI-telHRI control system with a hybrid admittance-force controller for ultrasound imaging. . Sensors 22:(11):4025
     [Crossref] [Google Scholar]
  32. 32.
    Chatelain P, Krupa A, Navab N. 2017.. Confidence-driven control of an ultrasound probe. . IEEE Trans. Robot. 33:(6):1410–24
     [Crossref] [Google Scholar]
  33. 33.
    Tsumura R, Iwata H. 2020.. Robotic fetal ultrasonography platform with a passive scan mechanism. . Int. J. Comput. Assist. Radiol. Surg. 15::1323–33
     [Crossref] [Google Scholar]
  34. 34.
    Wang S, Housden RJ, Noh Y, Singh A, Lindenroth L, et al. 2019.. Analysis of a customized clutch joint designed for the safety management of an ultrasound robot. . Appl. Sci. 9:(9):1900
     [Crossref] [Google Scholar]
  35. 35.
    Welleweerd MK, de Groot AG, de Looijer S, Siepel FJ, Stramigioli S. 2020.. Automated robotic breast ultrasound acquisition using ultrasound feedback. . In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9946–52. Piscataway, NJ:: IEEE
     [Google Scholar]
  36. 36.
    Facundo-Flores L, Treesatayapun C, Baltazar A. 2020.. Design of a pose and force controller for a robotized ultrasonic probe based on neural networks and stochastic gradient approximation. . IEEE Sens. J. 21:(5):6224–33
     [Crossref] [Google Scholar]
  37. 37.
    Wang Y, Liu T, Hu X, Yang K, Zhu Y, Jin H. 2023.. Compliant joint based robotic ultrasound scanning system for imaging human spine. . IEEE Robot. Autom. Lett. 8:(9):5966–73
     [Crossref] [Google Scholar]
  38. 38.
    Bao X, Wang S, Zheng L, Housden RJ, Hajnal JV, Rhode K. 2023.. A novel ultrasound robot with force/torque measurement and control for safe and efficient scanning. . IEEE Trans. Instrum. Meas. 72::4002012
     [Google Scholar]
  39. 39.
    Goel R, Abhimanyu F, Patel K, Galeotti J, Choset H. 2022.. Autonomous ultrasound scanning using Bayesian optimization and hybrid force control. . In 2022 International Conference on Robotics and Automation (ICRA), pp. 8396–402. Piscataway, NJ:: IEEE
     [Google Scholar]
  40. 40.
    Napoli ME, Freitas C, Goswami S, McAleavey S, Doyley M, Howard TM. 2018.. Hybrid force/velocity control with compliance estimation via strain elastography for robot assisted ultrasound screening. . In 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), pp. 1266–73. Piscataway, NJ:: IEEE
     [Google Scholar]
  41. 41.
    Dyck M, Sachtler A, Klodmann J, Albu-Schäffer A. 2022.. Impedance control on arbitrary surfaces for ultrasound scanning using discrete differential geometry. . IEEE Robot. Autom. Lett. 7:(3):7738–46
     [Crossref] [Google Scholar]
  42. 42.
    Fang TY, Zhang HK, Finocchi R, Taylor RH, Boctor EM. 2017.. Force-assisted ultrasound imaging system through dual force sensing and admittance robot control. . Int. J. Comput. Assist. Radiol. Surg. 12::983–91
     [Crossref] [Google Scholar]
  43. 43.
    Wang J, Lu C, Lv Y, Yang S, Zhang M, Shen Y. 2023.. Task space compliant control and six-dimensional force regulation toward automated robotic ultrasound imaging. . IEEE Trans. Autom. Sci. Eng. https://doi.org/10.1109/TASE.2023.3282974
    [Google Scholar]
  44. 44.
    Guerrero J, Salcudean S, McEwen JA, Masri BA, Nicolaou S. 2003.. Deep venous thrombosis screening system using numerical measures. . In Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vol. 1, pp. 894–97. Piscataway, NJ:: IEEE
     [Google Scholar]
  45. 45.
    Duan A, Victorova M, Zhao J, Sun Y, Zheng Y, Navarro-Alarcon D. 2022.. Ultrasound-guided assistive robots for scoliosis assessment with optimization-based control and variable impedance. . IEEE Robot. Autom. Lett. 7:(3):8106–13
     [Crossref] [Google Scholar]
  46. 46.
    Xiao S, Wang C, Shi Y, Yu J, Xiong L, et al. 2021.. Visual optimization of ultrasound-guided robot-assisted procedures using variable impedance control. . In 2021 WRC Symposium on Advanced Robotics and Automation (WRC SARA), pp. 128–33. Piscataway, NJ:: IEEE
     [Google Scholar]
  47. 47.
    Santos L, Cortesão R. 2018.. Computed-torque control for robotic-assisted tele-echography based on perceived stiffness estimation. . IEEE Trans. Autom. Sci. Eng. 15:(3):1337–54
     [Crossref] [Google Scholar]
  48. 48.
    Kim YJ, Park CK, Kim KG. 2019.. Gain determination of feedback force for an ultrasound scanning robot using genetic algorithm. . Int. J. Comput. Assisted Radiol. Surg. 14::797–807
     [Crossref] [Google Scholar]
  49. 49.
    Abbas M, Al Issa S, Dwivedy SK. 2021.. Event-triggered adaptive hybrid position-force control for robot-assisted ultrasonic examination system. . J. Intell. Robot. Syst. 102::84
     [Crossref] [Google Scholar]
  50. 50.
    Wang Z, Zhao B, Zhang P, Yao L, Wang Q, et al. 2022.. Full-coverage path planning and stable interaction control for automated robotic breast ultrasound scanning. . IEEE Trans. Ind. Electron. 70:(7):7051–61
     [Crossref] [Google Scholar]
  51. 51.
    Raina D, Chandrashekhara S, Voyles R, Wachs J, Saha SK. 2023.. Deep kernel and image quality estimators for optimizing robotic ultrasound controller using Bayesian optimization. . In 2023 International Symposium on Medical Robotics (ISMR). Piscataway, NJ:: IEEE. https://doi.org/10.1109/ISMR57123.2023.10130193
    [Google Scholar]
  52. 52.
    Akbari M, Carriere J, Meyer T, Sloboda R, Husain S, et al. 2021.. Robotic ultrasound scanning with real-time image-based force adjustment: quick response for enabling physical distancing during the COVID-19 pandemic. . Front. Robot. AI 8::645424
     [Crossref] [Google Scholar]
  53. 53.
    Sutedjo V, Tirindelli M, Eilers C, Simson W, Busam B, Navab N. 2022.. Acoustic shadowing aware robotic ultrasound: lighting up the dark. . IEEE Robot. Autom. Lett. 7:(2):1808–15
     [Crossref] [Google Scholar]
  54. 54.
    Merouche S, Allard L, Montagnon E, Soulez G, Bigras P, Cloutier G. 2015.. A robotic ultrasound scanner for automatic vessel tracking and three-dimensional reconstruction of B-mode images. . IEEE Trans. Ultrason. Ferroelectr. Freq. Control 63:(1):35–46
     [Crossref] [Google Scholar]
  55. 55.
    Akbari M, Carriere J, Sloboda R, Meyer T, Usmani N, et al. 2021.. Robot-assisted breast ultrasound scanning using geometrical analysis of the seroma and image segmentation. . In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3784–91. Piscataway, NJ:: IEEE
     [Google Scholar]
  56. 56.
    Liu S, Wang Y, Yang X, Lei B, Liu L, et al. 2019.. Deep learning in medical ultrasound analysis: a review. . Engineering 5:(2):261–75
     [Crossref] [Google Scholar]
  57. 57.
    Ronneberger O, Fischer P, Brox T. 2015.. U-Net: convolutional networks for biomedical image segmentation. . In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, ed. N Navab, J Hornegger, W Wells, A Frangi , pp. 234–41. Cham, Switz:.: Springer
     [Google Scholar]
  58. 58.
    Jiang Z, Li Z, Grimm M, Zhou M, Esposito M, et al. 2021.. Autonomous robotic screening of tubular structures based only on real-time ultrasound imaging feedback. . IEEE Trans. Ind. Electron. 69:(7):7064–75
     [Crossref] [Google Scholar]
  59. 59.
    Koskinopoulou M, Acemoglu A, Penza V, Mattos LS. 2023.. Dual robot collaborative system for autonomous venous access based on ultrasound and bioimpedance sensing technology. . In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 4648–53. Piscataway, NJ:: IEEE
     [Google Scholar]
  60. 60.
    Chen M, Huang Y, Chen J, Zhou T, Chen J, Liu H. 2023.. Fully robotized 3D ultrasound image acquisition for artery. . In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2690–96. Piscataway, NJ:: IEEE
     [Google Scholar]
  61. 61.
    Chen Y, Wang Y, Lai B, Chen Z, Cao X, et al. 2021.. Semi-supervised vein segmentation of ultrasound images for autonomous venipuncture. . In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9475–81. Piscataway, NJ:: IEEE
     [Google Scholar]
  62. 62.
    Jiang Z, Duelmer F, Navab N. 2023.. DopUS-Net: quality-aware robotic ultrasound imaging based on Doppler signal. . IEEE Trans. Autom. Sci. Eng. https://doi.org/10.1109/TASE.2023.3277331
    [Google Scholar]
  63. 63.
    Ning G, Liang H, Zhang X, Liao H. 2023.. Autonomous robotic ultrasound vascular imaging system with decoupled control strategy for external-vision-free environments. . IEEE Trans. Biomed. Eng. 70:(11):3166–77
     [Crossref] [Google Scholar]
  64. 64.
    Velikova Y, Simson W, Salehi M, Azampour MF, Paprottka P, Navab N. 2022.. CACTUSS: Common Anatomical CT-US Space for US examinations. . In Medical Image Computing and Computer Assisted Intervention—MICCAI 2022, ed. L Wang, Q Dou, PT Fletcher, S Speidel, S Li , pp. 492–501. Cham, Switz:.: Springer
     [Google Scholar]
  65. 65.
    Velikova Y, Azampour MF, Simson W, Duque VG, Navab N. 2023.. LOTUS: Learning to Optimize Task-based US representations. . In Medical Image Computing and Computer Assisted Intervention—MICCAI 2023, ed. H Greenspan, A Madabhushi, P Mousavi, S Salcudean, J Duncan, et al. , pp. 435–45. Cham, Switz:.: Springer
     [Google Scholar]
  66. 66.
    Che C, Mathai TS, Galeotti J. 2017.. Ultrasound registration: a review. . Methods 115::128–43
     [Crossref] [Google Scholar]
  67. 67.
    Hennersperger C, Fuerst B, Virga S, Zettinig O, Frisch B, et al. 2016.. Towards MRI-based autonomous robotic US acquisitions: a first feasibility study. . IEEE Trans. Med. Imaging 36:(2):538–48
     [Crossref] [Google Scholar]
  68. 68.
    Langsch F, Virga S, Esteban J, Göbl R, Navab N. 2019.. Robotic ultrasound for catheter navigation in endovascular procedures. . In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5404–10. Piscataway, NJ:: IEEE
     [Google Scholar]
  69. 69.
    Jiang Z, Li X, Zhang C, Bi Y, Stechele W, Navab N. 2023.. Skeleton graph-based ultrasound-CT non-rigid registration. . IEEE Robot. Autom. Lett. 8:(8):4394–401
     [Crossref] [Google Scholar]
  70. 70.
    Jiang Z, Li C, Li X, Navab N. 2023.. Thoracic cartilage ultrasound-CT registration using dense skeleton graph. . In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway, NJ:: IEEE. Forthcoming
    [Google Scholar]
  71. 71.
    Jiang Z, Wang H, Li Z, Grimm M, Zhou M, et al. 2021.. Motion-aware robotic 3D ultrasound. . In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 12494–500. Piscataway, NJ:: IEEE
     [Google Scholar]
  72. 72.
    Jiang Z, Danis N, Bi Y, Zhou M, Kroenke M, et al. 2022.. Precise repositioning of robotic ultrasound: improving registration-based motion compensation using ultrasound confidence optimization. . IEEE Trans. Instrum. Meas. 71::5020611
     [Google Scholar]
  73. 73.
    Wein W, Brunke S, Khamene A, Callstrom MR, Navab N. 2008.. Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention. . Med. Image Anal. 12:(5):577–85
     [Crossref] [Google Scholar]
  74. 74.
    Heinrich MP, Jenkinson M, Bhushan M, Matin T, Gleeson FV, et al. 2012.. MIND: modality independent neighbourhood descriptor for multi-modal deformable registration. . Med. Image Anal. 16:(7):1423–35
     [Crossref] [Google Scholar]
  75. 75.
    Ronchetti M, Wein W, Navab N, Zettinig O, Prevost R. 2023.. DISA: DIfferentiable Similarity Approximation for universal multimodal registration. . In Medical Image Computing and Computer Assisted Intervention—MICCAI 2023, ed. H Greenspan, A Madabhushi, P Mousavi, S Salcudean, J Duncan, et al. , pp. 761–70. Cham, Switz:.: Springer
     [Google Scholar]
  76. 76.
    Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, et al. 2015.. Human-level control through deep reinforcement learning. . Nature 518:(7540):529–33
     [Crossref] [Google Scholar]
  77. 77.
    Hase H, Azampour MF, Tirindelli M, Paschali M, Simson W, et al. 2020.. Ultrasound-guided robotic navigation with deep reinforcement learning. . In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5534–41. Piscataway, NJ:: IEEE
     [Google Scholar]
  78. 78.
    Li K, Wang J, Xu Y, Qin H, Liu D, et al. 2021.. Autonomous navigation of an ultrasound probe towards standard scan planes with deep reinforcement learning. . In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 8302–8. Piscataway, NJ:: IEEE
     [Google Scholar]
  79. 79.
    Karamalis A, Wein W, Klein T, Navab N. 2012.. Ultrasound confidence maps using random walks. . Med. Image Anal. 16:(6):1101–12
     [Crossref] [Google Scholar]
  80. 80.
    Ning G, Zhang X, Liao H. 2021.. Autonomic robotic ultrasound imaging system based on reinforcement learning. . IEEE Trans. Biomed. Eng. 68:(9):2787–97
     [Crossref] [Google Scholar]
  81. 81.
    Bi Y, Jiang Z, Gao Y, Wendler T, Karlas A, Navab N. 2022.. VesNet-RL: simulation-based reinforcement learning for real-world US probe navigation. . IEEE Robot. Autom. Lett. 7:(3):6638–45
     [Crossref] [Google Scholar]
  82. 82.
    Li K, Mao X, Ye C, Li A, Xu Y, Meng MQH. 2023.. Style transfer enabled Sim2Real framework for efficient learning of robotic ultrasound image analysis using simulated data. . arXiv:2305.09169 [cs.RO]
  83. 83.
    Men Q, Teng C, Drukker L, Papageorghiou AT, Noble JA. 2022.. Multimodal-GuideNet: gaze-probe bidirectional guidance in obstetric ultrasound scanning. . In Medical Image Computing and Computer Assisted Intervention—MICCAI 2022, ed. L Wang, Q Dou, PT Fletcher, S Speidel, S Li , pp. 94–103. Cham, Switz:.: Springer
     [Google Scholar]
  84. 84.
    Deng X, Chen Y, Chen F, Li M. 2021.. Learning robotic ultrasound scanning skills via human demonstrations and guided explorations. . In 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 372–78. Piscataway, NJ:: IEEE
     [Google Scholar]
  85. 85.
    Abbeel P, Ng AY. 2004.. Apprenticeship learning via inverse reinforcement learning. . In ICML'04: Proceedings of the Twenty-First International Conference on Machine Learning. New York:: ACM. https://doi.org/10.1145/1015330.1015430
    [Google Scholar]
  86. 86.
    Ziebart BD, Maas AL, Bagnell JA, Dey AK. 2008.. Maximum entropy inverse reinforcement learning. . Proc. AAAI Conf. Artif. Intell. 23:(1):1433–38
     [Google Scholar]
  87. 87.
    Burke M, Lu K, Angelov D, Straižys A, Innes C, et al. 2023.. Learning rewards from exploratory demonstrations using probabilistic temporal ranking. . Auton. Robots 47::733–51
     [Crossref] [Google Scholar]
  88. 88.
    d'Aulignac D, Laugier C, Troccaz J, Vieira S. 2006.. Towards a realistic echographic simulator. . Med. Image Anal. 10:(1):71–81
     [Crossref] [Google Scholar]
  89. 89.
    Goksel O, Salcudean SE. 2009.. B-mode ultrasound image simulation in deformable 3-D medium. . IEEE Trans. Med. Imaging 28:(11):1657–69
     [Crossref] [Google Scholar]
  90. 90.
    Treeby BE, Cox BT. 2010.. k-Wave: MATLAB toolbox for the simulation and reconstruction of photoacoustic wave fields. . J. Biomed. Opt. 15:(2):021314
     [Crossref] [Google Scholar]
  91. 91.
    Gao H, Choi HF, Claus P, Boonen S, Jaecques S, et al. 2009.. A fast convolution-based methodology to simulate 2-D/3-D cardiac ultrasound images. . IEEE Trans. Ultrason. Ferroelectr. Freq. Control 56:(2):404–9
     [Crossref] [Google Scholar]
  92. 92.
    Burger B, Bettinghausen S, Radle M, Hesser J. 2012.. Real-time GPU-based ultrasound simulation using deformable mesh models. . IEEE Trans. Med. Imaging 32:(3):609–18
     [Crossref] [Google Scholar]
  93. 93.
    Salehi M, Ahmadi SA, Prevost R, Navab N, Wein W. 2015.. Patient-specific 3D ultrasound simulation based on convolutional ray-tracing and appearance optimization. . In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, ed. N Navab, J Hornegger, W Wells, A Frangi , pp. 510–18. Cham, Switz:.: Springer
     [Google Scholar]
  94. 94.
    Mattausch O, Makhinya M, Goksel O. 2018.. Realistic ultrasound simulation of complex surface models using interactive Monte-Carlo path tracing. . Comput. Graph. Forum 37:(1):202–13
     [Crossref] [Google Scholar]
  95. 95.
    Tom F, Sheet D. 2018.. Simulating patho-realistic ultrasound images using deep generative networks with adversarial learning. . In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1174–77. Piscataway, NJ:: IEEE
     [Google Scholar]
  96. 96.
    Hu Y, Gibson E, Lee LL, Xie W, Barratt DC, et al. 2017.. Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks. . In Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment, ed. MJ Cardoso, T Arbel, F Gao, B Kainz, T van Walsum , pp. 105–15. Cham, Switz:.: Springer
     [Google Scholar]
  97. 97.
    Vitale S, Orlando JI, Iarussi E, Larrabide I. 2020.. Improving realism in patient-specific abdominal ultrasound simulation using CycleGANs. . Int. J. Comput. Assist. Radiol. Surg. 15:(2):183–92
     [Crossref] [Google Scholar]
  98. 98.
    Zhang L, Wang X, Yang D, Sanford T, Harmon S, et al. 2020.. Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. . IEEE Trans. Med. Imaging 39:(7):2531–40
     [Crossref] [Google Scholar]
  99. 99.
    Yin S, Peng Q, Li H, Zhang Z, You X, et al. 2020.. Automatic kidney segmentation in ultrasound images using subsequent boundary distance regression and pixelwise classification networks. . Med. Image Anal. 60::101602
     [Crossref] [Google Scholar]
  100. 100.
    Lee LH, Gao Y, Noble JA. 2021.. Principled ultrasound data augmentation for classification of standard planes. . In Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, ed. A Feragen, S Sommer, J Schnabel, M Nielsen , pp. 729–41. Cham, Switz.:: Springer
     [Google Scholar]
  101. 101.
    Pang T, Wong JHD, Ng WL, Chan CS. 2021.. Semi-supervised GAN-based radiomics model for data augmentation in breast ultrasound mass classification. . Comput. Methods Programs Biomed. 203::106018
     [Crossref] [Google Scholar]
  102. 102.
    Tiago C, Gilbert A, Beela AS, Aase SA, Snare SR, et al. 2022.. A data augmentation pipeline to generate synthetic labeled datasets of 3D echocardiography images using a GAN. . IEEE Access 10::98803–15
     [Crossref] [Google Scholar]
  103. 103.
    Shi G, Wang J, Qiang Y, Yang X, Zhao J, et al. 2020.. Knowledge-guided synthetic medical image adversarial augmentation for ultrasonography thyroid nodule classification. . Comput. Methods Programs Biomed. 196::105611
     [Crossref] [Google Scholar]
  104. 104.
    Zhang R, Lu W, Wei X, Zhu J, Jiang H, et al. 2021.. A progressive generative adversarial method for structurally inadequate medical image data augmentation. . IEEE J. Biomed. Health Inform. 26:(1):7–16
     [Crossref] [Google Scholar]
  105. 105.
    Pesteie M, Abolmaesumi P, Rohling RN. 2019.. Adaptive augmentation of medical data using independently conditional variational auto-encoders. . IEEE Trans. Med. Imaging 38:(12):2807–20
     [Crossref] [Google Scholar]
  106. 106.
    Wulff D, Dohnke T, Nguyen NT, Ernst F. 2023.. Towards realistic 3D ultrasound synthesis: deformable augmentation using conditional variational autoencoders. . In 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS), pp. 821–26. Piscataway, NJ:: IEEE
     [Google Scholar]
  107. 107.
    Tirindelli M, Eilers C, Simson W, Paschali M, Azampour MF, Navab N. 2021.. Rethinking ultrasound augmentation: a physics-inspired approach. . In Medical Image Computing and Computer Assisted Intervention—MICCAI 2021, ed. M de Bruijne, PC Cattin, S Cotin, N Padoy, S Speidel, et al. , pp. 690–700. Cham, Switz:.: Springer
     [Google Scholar]
  108. 108.
    Yu H, Li Y, Wu Q, Zhao Z, Chen D, et al. 2023.. Mining negative temporal contexts for false positive suppression in real-time ultrasound lesion detection. . arXiv:2305.18060 [cs.CV]
  109. 109.
    Painchaud N, Skandarani Y, Judge T, Bernard O, Lalande A, Jodoin PM. 2020.. Cardiac segmentation with strong anatomical guarantees. . IEEE Trans. Med. Imaging 39:(11):3703–13
     [Crossref] [Google Scholar]
  110. 110.
    Jiang Z, Zhou Y, Cao D, Navab N. 2023.. DefCor-Net: physics-aware ultrasound deformation correction. . Med. Image Anal. 90::102923
     [Crossref] [Google Scholar]
  111. 111.
    Jiang Z, Zhou Y, Bi Y, Zhou M, Wendler T, Navab N. 2021.. Deformation-aware robotic 3D ultrasound. . IEEE Robot. Autom. Lett. 6:(4):7675–82
     [Crossref] [Google Scholar]
  112. 112.
    Dou H, Han L, He Y, Xu J, Ravikumar N, et al. 2022.. Localizing the recurrent laryngeal nerve via ultrasound with a Bayesian shape framework. . In Medical Image Computing and Computer Assisted Intervention—MICCAI 2022, ed. L Wang, Q Dou, PT Fletcher, S Speidel, S Li , pp. 258–67. Cham, Switz:.: Springer
     [Google Scholar]
  113. 113.
    Painchaud N, Duchateau N, Bernard O, Jodoin PM. 2022.. Echocardiography segmentation with enforced temporal consistency. . IEEE Trans. Med. Imaging 41:(10):2867–78
     [Crossref] [Google Scholar]
  114. 114.
    Gare GR, Li J, Joshi R, Magar R, Vaze MP, et al. 2022.. W-Net: dense and diagnostic semantic segmentation of subcutaneous and breast tissue in ultrasound images by incorporating ultrasound RF waveform data. . Med. Image Anal. 76::102326
     [Crossref] [Google Scholar]
  115. 115.
    Ning G, Liang H, Chen F, Zhang X, Liao H. 2023.. Doppler image-based weakly-supervised vascular ultrasound segmentation with transformer. . In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI). Piscataway, NJ:: IEEE. https://doi.org/10.1109/ISBI53787.2023.10230548
    [Google Scholar]
  116. 116.
    Li K, Xu Y, Wang J, Ni D, Liu L, Meng MQH. 2021.. Image-guided navigation of a robotic ultrasound probe for autonomous spinal sonography using a shadow-aware dual-agent framework. . IEEE Trans. Med. Robot. Bionics 4:(1):130–44
     [Crossref] [Google Scholar]
  117. 117.
    Klein T, Wells WM. 2015.. RF ultrasound distribution-based confidence maps. . In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, ed. N Navab, J Hornegger, W Wells, A Frangi , pp. 595–602. Cham, Switz:.: Springer
     [Google Scholar]
  118. 118.
    Meng Q, Sinclair M, Zimmer V, Hou B, Rajchl M, et al. 2019.. Weakly supervised estimation of shadow confidence maps in fetal ultrasound imaging. . IEEE Trans. Med. Imaging 38:(12):2755–67
     [Crossref] [Google Scholar]
  119. 119.
    Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R. 2021.. NeRF: representing scenes as neural radiance fields for view synthesis. . Commun. ACM 65:(1):99–106
     [Crossref] [Google Scholar]
  120. 120.
    Wysocki M, Azampour MF, Eilers C, Busam B, Salehi M, Navab N. 2023.. Ultra-NeRF: neural radiance fields for ultrasound imaging. . In Proceedings of the Sixth Conference on Medical Imaging with Deep Learning. Forthcoming
     [Google Scholar]
  121. 121.
    Milletari F, Navab N, Ahmadi SA. 2016.. V-Net: fully convolutional neural networks for volumetric medical image segmentation. . In 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–71. Piscataway, NJ:: IEEE
     [Google Scholar]
  122. 122.
    Degel MA, Navab N, Albarqouni S. 2018.. Domain and geometry agnostic CNNs for left atrium segmentation in 3D ultrasound. . In Medical Image Computing and Computer Assisted Intervention—MICCAI 2018, ed. A Frangi, J Schnabel, C Davatzikos, C Alberola-López, G Fichtinger , pp. 630–37. Cham, Switz:.: Springer
     [Google Scholar]
  123. 123.
    Meng Q, Pawlowski N, Rueckert D, Kainz B. 2019.. Representation disentanglement for multi-task learning with application to fetal ultrasound. . In Smart Ultrasound Imaging and Perinatal, Preterm and Paediatric Image Analysis: First International Workshop, SUSI 2019, and 4th International Workshop, PIPPI 2019, ed. Q Wang, A Gomez, J Hutter, K McLeod, V Zimmer, et al. , pp. 47–55. Cham, Switz:.: Springer
     [Google Scholar]
  124. 124.
    Meng Q, Rueckert D, Kainz B. 2020.. Unsupervised cross-domain image classification by distance metric guided feature alignment. . In Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis: First International Workshop, ASMUS 2020, and 5th International Workshop, PIPPI 2020, ed. Y Hu, R Licandro, JA Noble, J Hutter, S Aylward , pp. 146–57. Cham, Switz:.: Springer
     [Google Scholar]
  125. 125.
    Ying X, Liu Z, Gao J, Zhang R, Jiang H, Wei X. 2022.. Multi-task class feature space fusion domain adaptation network for thyroid ultrasound images: research on generalization of smart healthcare systems. . In Wireless Algorithms, Systems, and Applications: 17th International Conference, WASA 2022, ed. L Wang, M Segal, J Chen, T Qiu , pp. 139–52. Cham, Switz.:: Springer
     [Google Scholar]
  126. 126.
    Zhang K, Li Z, Cai C, Liu J, Xu D, et al. 2023.. Semi-supervised graph convolutional networks for the domain adaptive recognition of thyroid nodules in cross-device ultrasound images. . Med. Phys. 50:(12):7806–21
     [Crossref] [Google Scholar]
  127. 127.
    Morris C, Ritzert M, Fey M, Hamilton WL, Lenssen JE, et al. 2019.. Weisfeiler and Leman go neural: higher-order graph neural networks. . Proc. AAAI Conf. Artif. Intell. 33:(1):4602–9
     [Google Scholar]
  128. 128.
    Belghazi MI, Baratin A, Rajeshwar S, Ozair S, Bengio Y, et al. 2018.. Mutual information neural estimation. . In Proceedings of the 35th International Conference on Machine Learning, ed. J Dy, A Krause , pp. 531–40. Proc. Mach. Learn. Res. 80. N.p.:: PMLR
     [Google Scholar]
  129. 129.
    Cha J, Lee K, Park S, Chun S. 2022.. Domain generalization by mutual-information regularization with pre-trained models. . In Computer Vision—ECCV 2022, ed. S Avidan, G Brostow, M Cissé, GM Farinella, T Hassner , pp. 440–57. Cham, Switz.:: Springer
     [Google Scholar]
  130. 130.
    Liu X, Yang C, You J, Kuo CCJ, Kumar BV. 2021.. Mutual information regularized feature-level Frankenstein for discriminative recognition. . IEEE Trans. Pattern Anal. Mach. Intell. 44:(9):5243–60
     [Google Scholar]
  131. 131.
    Peng X, Huang Z, Sun X, Saenko K. 2019.. Domain agnostic learning with disentangled representations. . In Proceedings of the 36th International Conference on Machine Learning, ed. K Chaudhuri, R Salakhutdinov , pp. 5102–12. Proc. Mach. Learn. Res. 97 . N.p.:: PMLR
     [Google Scholar]
  132. 132.
    Meng Q, Matthew J, Zimmer VA, Gomez A, Lloyd DF, et al. 2020.. Mutual information-based disentangled neural networks for classifying unseen categories in different domains: application to fetal ultrasound imaging. . IEEE Trans. Med. Imaging 40:(2):722–34
     [Crossref] [Google Scholar]
  133. 133.
    Bi Y, Jiang Z, Clarenbach R, Ghotbi R, Karlas A, Navab N. 2023.. MI-SegNet: mutual information-based US segmentation for unseen domain generalization. . In Medical Image Computing and Computer Assisted Intervention—MICCAI 2023, ed. H Greenspan, A Madabhushi, P Mousavi, S Salcudean, J Duncan, et al. , pp. 130–40. Cham, Switz:.: Springer
     [Google Scholar]
  134. 134.
    Fichtinger G, Troccaz J, Haidegger T. 2022.. Image-guided interventional robotics: lost in translation?. Proc. IEEE 110:(7):932–50
     [Crossref] [Google Scholar]
  135. 135.
    Khamis A, Li H, Prestes E, Haidegger T. 2019.. AI: a key enabler of sustainable development goals, part 1. . IEEE Robot. Autom. Mag. 26:(3):95–102
     [Crossref] [Google Scholar]
  136. 136.
    Haidegger T. 2019.. Autonomy for surgical robots: concepts and paradigms. . IEEE Trans. Med. Robot. Bionics 1:(2):65–76
     [Crossref] [Google Scholar]
  137. 137.
    Prestes E, Houghtaling MA, Gonçalves PJ, Fabiano N, Ulgen O, et al. 2021.. The first global ontological standard for ethically driven robotics and automation systems. . IEEE Robot. Autom. Mag. 28:(4):120–24
     [Crossref] [Google Scholar]
  138. 138.
    Haidegger T, Speidel S, Stoyanov D, Satava RM. 2022.. Robot-assisted minimally invasive surgery—surgical robotics in the data age. . Proc. IEEE 110:(7):835–46
     [Crossref] [Google Scholar]
  139. 139.
    Attanasio A, Scaglioni B, De Momi E, Fiorini P, Valdastri P. 2021.. Autonomy in surgical robotics. . Annu. Rev. Control Robot. Auton. Syst. 4::651–79
     [Crossref] [Google Scholar]
  140. 140.
    Karlas A, Pleitez MA, Aguirre J, Ntziachristos V. 2021.. Optoacoustic imaging in endocrinology and metabolism. . Nat. Rev. Endocrinol. 17:(6):323–35
     [Crossref] [Google Scholar]
  141. 141.
    Little C, Colchester R, Noimark S, Manmathan G, Rakhit R, Desjardins A. 2020.. Optical ultrasound (OpUS): a novel concept for intravascular imaging. . Eur. Heart J. 41:(Suppl. 2):ehaa946.2457
     [Crossref] [Google Scholar]
  142. 142.
    Zhang S, Lim CS, Zhang EZ, Beard PC, Desjardins AE, Colchester RJ. 2023.. Miniaturised all-optical ultrasound probe for thrombus imaging. . In Opto-Acoustic Methods and Applications in Biophotonics VI, ed. C Kim, J Laufer, V Ntziachristos, RJ Zemp , pp. 87–91. Bellingham, WA:: SPIE
     [Google Scholar]
  143. 143.
    Graham M, Assis F, Allman D, Wiacek A, Gonzalez E, et al. 2019.. In vivo demonstration of photoacoustic image guidance and robotic visual servoing for cardiac catheter-based interventions. . IEEE Trans. Med. Imaging 39:(4):1015–29
     [Crossref] [Google Scholar]
  144. 144.
    Shubert J, Bell MAL. 2017.. Photoacoustic based visual servoing of needle tips to improve biopsy on obese patients. . In 2017 IEEE International Ultrasonics Symposium (IUS). Piscataway, NJ:: IEEE. https://doi.org/10.1109/ULTSYM.2017.8091815
    [Google Scholar]
  145. 145.
    Hu H, Zhu X, Wang C, Zhang L, Li X, et al. 2018.. Stretchable ultrasonic transducer arrays for three-dimensional imaging on complex surfaces. . Sci. Adv. 4:(3):eaar3979
     [Crossref] [Google Scholar]
  146. 146.
    Wang C, Qi B, Lin M, Zhang Z, Makihata M, et al. 2021.. Continuous monitoring of deep-tissue haemodynamics with stretchable ultrasonic phased arrays. . Nat. Biomed. Eng. 5:(7):749–58
     [Crossref] [Google Scholar]
  147. 147.
    Wang C, Chen X, Wang L, Makihata M, Liu HC, et al. 2022.. Bioadhesive ultrasound for long-term continuous imaging of diverse organs. . Science 377:(6605):517–23
     [Crossref] [Google Scholar]
  148. 148.
    Hu H, Huang H, Li M, Gao X, Yin L, et al. 2023.. A wearable cardiac ultrasound imager. . Nature 613:(7945):667–75
     [Crossref] [Google Scholar]
  149. 149.
    Lin M, Zhang Z, Gao X, Bian Y, Wu RS, et al. 2023.. A fully integrated wearable ultrasound system to monitor deep tissues in moving subjects. . Nat. Biotechnol. https://doi.org/10.1038/s41587-023-01800-0
    [Google Scholar]
/content/journals/10.1146/annurev-control-091523-100042
Loading
Machine Learning in Robotic Ultrasound Imaging: Challenges and Perspectives
Annual Review of Control, Robotics, and Autonomous Systems 7, 335 (2024); https://doi.org/10.1146/annurev-control-091523-100042
/content/journals/10.1146/annurev-control-091523-100042
/content/journals/10.1146/annurev-control-091523-100042
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error