1932

Abstract

Histopathology plays a fundamental role in the diagnosis and subtyping of solid tumors and has become a cornerstone of modern precision oncology. Histopathological evaluation is typically performed manually by expert pathologists due to the complexity of visual data. However, in the last ten years, new artificial intelligence (AI) methods have made it possible to train computers to perform visual tasks with high performance, reaching similar levels as experts in some applications. In cancer histopathology, these AI tools could help automate repetitive tasks, making more efficient use of pathologists’ time. In research studies, AI methods have been shown to have an astounding ability to predict genetic alterations and identify prognostic and predictive biomarkers directly from routine tissue slides. Here, we give an overview of these recent applications of AI in computational pathology, focusing on new tools for cancer research that could be pivotal in identifying clinical biomarkers for better treatment decisions.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-cancerbio-061521-092038
2023-04-11
2024-06-23
Loading full text...

Full text loading...

/deliver/fulltext/cancerbio/7/1/annurev-cancerbio-061521-092038.html?itemId=/content/journals/10.1146/annurev-cancerbio-061521-092038&mimeType=html&fmt=ahah

Literature Cited

  1. Akinleye A, Rasool Z. 2019. Immune checkpoint inhibitors of PD-L1 as cancer therapeutics. J. Hematol. Oncol. 12:92
    [Google Scholar]
  2. Behjati S, Tarpey PS. 2013. What is next generation sequencing?. Arch. Dis. Child. Educ. Pract. Ed. 98:6236–38
    [Google Scholar]
  3. Binder A, Bockmayr M, Hägele M, Wienert S, Heim D et al. 2021. Morphological and molecular breast cancer profiling through explainable machine learning. Nat. Mach. Intell. 3:4355–66
    [Google Scholar]
  4. Bone D, Goodwin MS, Black MP, Lee C-C, Audhkhasi K, Narayanan S. 2015. Applying machine learning to facilitate autism diagnostics: pitfalls and promises. J. Autism Dev. Disord. 45:51121–36
    [Google Scholar]
  5. Campanella G, Hanna MG, Geneslaw L, Miraflor A, Werneck Krauss Silva V et al. 2019. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25:81301–9
    [Google Scholar]
  6. Chatterjee S. 2014. Artefacts in histopathology. J. Oral Maxillofac. Pathol. 18:Suppl. 1S111–16
    [Google Scholar]
  7. Chen RJ, Lu MY, Wang J, Williamson DFK, Rodig SJ et al. 2022. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans. Med. Imaging 41:4757–70
    [Google Scholar]
  8. Chen S, Xiang J, Wang X, Zhang J, Yang S et al. 2022. Pan-cancer computational histopathology reveals tumor mutational burden status through weakly-supervised deep learning arXiv:2204.03257 [cs.CV]
    [Google Scholar]
  9. Cifci D, Foersch S, Kather JN. 2022. Artificial intelligence to identify genetic alterations in conventional histopathology. J. Pathol. 257:4430–44
    [Google Scholar]
  10. Ciga O, Xu T, Martel AL. 2020. Self supervised contrastive learning for digital histopathology. Mach. Learn. Appl. 7:100198
    [Google Scholar]
  11. Cocco E, Scaltriti M, Drilon A. 2018. NTRK fusion-positive cancers and TRK inhibitor therapy. Nat. Rev. Clin. Oncol. 15:12731–47
    [Google Scholar]
  12. Collins GS, Reitsma JB, Altman DG, Moons KGM. 2015. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann. Intern. Med. 162:1W1–73
    [Google Scholar]
  13. Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M et al. 2018. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24:101559–67
    [Google Scholar]
  14. Coudray N, Tsirigos A. 2020. Deep learning links histology, molecular signatures and prognosis in cancer. Nat. Cancer 1:8755–57
    [Google Scholar]
  15. Drilon A, Laetsch TW, Kummar S, DuBois SG, Lassen UN et al. 2018. Efficacy of larotrectinib in TRK fusion-positive cancers in adults and children. N. Engl. J. Med. 378:8731–39
    [Google Scholar]
  16. Echle A, Ghaffari Laleh N, Quirke P, Grabsch HI, Muti HS et al. 2022. Artificial intelligence for detection of microsatellite instability in colorectal cancer—a multicentric analysis of a pre-screening tool for clinical application. ESMO Open 7:2100400
    [Google Scholar]
  17. Echle A, Grabsch HI, Quirke P, van den Brandt PA, West NP et al. 2020. Clinical-grade detection of microsatellite instability in colorectal tumors by deep learning. Gastroenterology 159:41406–16.e11
    [Google Scholar]
  18. Echle A, Laleh NG, Schrammen PL, West NP, Trautwein C et al. 2021a. Deep learning for the detection of microsatellite instability from histology images in colorectal cancer: a systematic literature review. ImmunoInformatics 3–4:100008
    [Google Scholar]
  19. Echle A, Rindtorff NT, Brinker TJ, Luedde T, Pearson AT, Kather JN. 2021b. Deep learning in cancer pathology: a new generation of clinical biomarkers. Br. J. Cancer 124:4686–96
    [Google Scholar]
  20. Farahani N, Parwani AV, Pantanowitz L. 2015. Whole slide imaging in pathology: advantages, limitations, and emerging perspectives. Pathol. Lab. Med. Int. 7:23–33
    [Google Scholar]
  21. FDA (US Food Drug. Admin.) 2021. Good machine learning practice for medical device development: guiding principles Web Resour. FDA Silver Spring, MD:
    [Google Scholar]
  22. Foersch S, Eckstein M, Wagner D-C, Gach F, Woerl A-C et al. 2021. Deep learning for diagnosis and survival prediction in soft tissue sarcoma. Ann. Oncol. 32:91178–87
    [Google Scholar]
  23. Foster KR, Koprowski R, Skufca JD. 2014. Machine learning, medical diagnosis, and biomedical engineering research—commentary. Biomed. Eng. Online 13:94
    [Google Scholar]
  24. Fu Y, Jung AW, Torne RV, Gonzalez S, Vöhringer H et al. 2020. Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis. Nat. Cancer 1:800–10
    [Google Scholar]
  25. Greenson JK, Huang S-C, Herron C, Moreno V, Bonner JD et al. 2009. Pathologic predictors of microsatellite instability in colorectal cancer. Am. J. Surg. Pathol. 33:126–33
    [Google Scholar]
  26. Havel JJ, Chowell D, Chan TA. 2019. The evolving landscape of biomarkers for checkpoint inhibitor immunotherapy. Nat. Rev. Cancer 19:3133–50
    [Google Scholar]
  27. Howard FM, Dolezal J, Kochanny S, Schulte J, Chen H et al. 2021. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nat. Commun. 12:4423
    [Google Scholar]
  28. Hu J, Cui C, Yang W, Huang L, Yu R et al. 2021. Using deep learning to predict anti-PD-1 response in melanoma and lung cancer patients from histopathology images. Transl. Oncol. 14:100921
    [Google Scholar]
  29. Huang S-C, Pareek A, Seyyedi S, Banerjee I, Lungren MP. 2020. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digit. Med. 3:136
    [Google Scholar]
  30. Jaafar H. 2006. Intra-operative frozen section consultation: concepts, applications and limitations. Malays. J. Med. Sci. 13:4–12
    [Google Scholar]
  31. Jain MS, Massoud TF. 2020. Predicting tumour mutational burden from histopathological images using multiscale deep learning. Nat. Mach. Intell. 2:6356–62
    [Google Scholar]
  32. Janowczyk A, Madabhushi A. 2016. Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J. Pathol. Inform. 7:29
    [Google Scholar]
  33. Jungherr A, Jürgens P, Schoen H 2012. Why the Pirate Party won the German election of 2009 or the trouble with predictions: a response to Tumasjan, A., Sprenger, T. O., Sander, P. G., & Welpe, I. M. “Predicting Elections With Twitter: What 140 Characters Reveal About Political Sentiment. .” Soc. Sci. Comput. Rev. 30:2299–34
    [Google Scholar]
  34. Kather JN, Heij LR, Grabsch HI, Loeffler C, Echle A et al. 2020. Pan-cancer image-based detection of clinically actionable genetic alterations. Nat. Cancer 1:8789–99
    [Google Scholar]
  35. Kather JN, Krisam J, Charoentong P, Luedde T, Herpel E et al. 2019a. Predicting survival from colorectal cancer histology slides using deep learning: a retrospective multicenter study. PLOS Med 16:e1002730
    [Google Scholar]
  36. Kather JN, Laleh NG, Foersch S, Truhn D. 2022. Medical domain knowledge in domain-agnostic generative AI. NPJ Digit Med 5:190
    [Google Scholar]
  37. Kather JN, Pearson AT, Halama N, Jäger D, Krause J et al. 2019b. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat. Med. 25:71054–56
    [Google Scholar]
  38. Kleppe A, Skrede O-J, De Raedt S, Liestøl K, Kerr DJ, Danielsen HE. 2021. Designing deep learning studies in cancer diagnostics. Nat. Rev. Cancer 21:3199–211
    [Google Scholar]
  39. Komura D, Ishikawa S. 2018. Machine learning methods for histopathological image analysis. Comput. Struct. Biotechnol. J. 16:34–42
    [Google Scholar]
  40. Korpanty GJ, Graham DM, Vincent MD, Leighl NB. 2014. Biomarkers that currently affect clinical practice in lung cancer: EGFR, ALK, MET, ROS-1, and KRAS. Front. Oncol. 4:204
    [Google Scholar]
  41. Kothari S, Phan JH, Stokes TH, Osunkoya AO, Young AN, Wang MD. 2014. Removing batch effects from histopathological images for enhanced cancer diagnosis. IEEE J. Biomed. Health Inform. 18:3765–72
    [Google Scholar]
  42. Krause J, Grabsch HI, Kloor M, Jendrusch M, Echle A et al. 2021. Deep learning detects genetic alterations in cancer histology generated by adversarial networks. J. Pathol. 254:70–79
    [Google Scholar]
  43. Laleh NG, Muti HS, Loeffler CML, Echle A, Saldanha OL et al. 2022a. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology. Med. Image Anal. 79:102474
    [Google Scholar]
  44. Laleh NG, Truhn D, Veldhuizen GP, Han T, van Treeck M et al. 2022b. Adversarial attacks and adversarial robustness in computational pathology. Nat. Commun. 13:5711
    [Google Scholar]
  45. Lazer D, Kennedy R, King G, Vespignani A. 2014. The parable of Google flu: traps in big data analysis. Science 343:61761203–5
    [Google Scholar]
  46. Leemans CR, Braakhuis BJM, Brakenhoff RH. 2011. The molecular biology of head and neck cancer. Nat. Rev. Cancer 11:9–22
    [Google Scholar]
  47. Levine AB, Peng J, Farnell D, Nursey M, Wang Y et al. 2020. Synthesis of diagnostic quality cancer pathology images by generative adversarial networks. J. Pathol. 252:2178–88
    [Google Scholar]
  48. Linardatos P, Papastefanopoulos V, Kotsiantis S. 2020. Explainable AI: a review of machine learning interpretability methods. Entropy 23:118
    [Google Scholar]
  49. Lipkova J, Chen TY, Lu MY, Chen RJ, Shady M et al. 2022. Deep learning-enabled assessment of cardiac allograft rejection from endomyocardial biopsies. Nat. Med. 28:3575–82
    [Google Scholar]
  50. Loeffler CML, Gaisa NT, Muti HS, van Treeck M, Echle A et al. 2022. Predicting mutational status of driver and suppressor genes directly from histopathology with deep learning: a systematic study across 23 solid tumor types. Front. Genet. 12:806386
    [Google Scholar]
  51. Louis DN, Gerber GK, Baron JM, Bry L, Dighe AS et al. 2014. Computational pathology: an emerging definition. Arch. Pathol. Lab. Med. 138:91133–38
    [Google Scholar]
  52. Lu MY, Chen RJ, Kong D, Lipkova J, Singh R et al. 2022. Federated learning for computational pathology on gigapixel whole slide images. Med. Image Anal. 76:102298
    [Google Scholar]
  53. Lu MY, Chen TY, Williamson DFK, Zhao M, Shady M et al. 2021a. AI-based pathology predicts origins for cancers of unknown primary. Nature 594:7861106–10
    [Google Scholar]
  54. Lu MY, Williamson DFK, Chen TY, Chen RJ, Barbieri M, Mahmood F. 2021b. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5:6555–70
    [Google Scholar]
  55. Ludwig JA, Weinstein JN. 2005. Biomarkers in cancer staging, prognosis and treatment selection. Nat. Rev. Cancer 5:11845–56
    [Google Scholar]
  56. Luo W, Phung D, Tran T, Gupta S, Rana S et al. 2016. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J. Med. Internet Res. 18:12e323
    [Google Scholar]
  57. Macenko M, Niethammer M, Marron JS, Borland D, Woosley JT et al. 2009. A method for normalizing histology slides for quantitative analysis. 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro1107–10. New York: IEEE
    [Google Scholar]
  58. Marcinkevičs R, Vogt JE. 2020. Interpretability and explainability: a machine learning zoo mini-tour. arXiv:2012.01805 [cs.LG]
  59. Mutasa S, Sun S, Ha R. 2020. Understanding artificial intelligence based radiology studies: What is overfitting?. Clin. Imaging 65:96–99
    [Google Scholar]
  60. Nigro V, Piluso G. 2012. Next generation sequencing (NGS) strategies for the genetic testing of myopathies. Acta Myol 31:3196–200
    [Google Scholar]
  61. Noseworthy PA, Attia ZI, Brewer LC, Hayes SN, Yao X et al. 2020. Assessing and mitigating bias in medical artificial intelligence: the effects of race and ethnicity on a deep learning model for ECG analysis. Circ. Arrhythm. Electrophysiol. 13:3e007988
    [Google Scholar]
  62. Pantanowitz L, Hartman D, Qi Y, Cho EY, Suh B et al. 2020. Accuracy and efficiency of an artificial intelligence tool when counting breast mitoses. Diagn. Pathol. 15:180
    [Google Scholar]
  63. Pinckaers H, Bulten W, van der Laak J, Litjens G. 2021. Detection of prostate cancer in whole-slide images through end-to-end training with image-level labels. IEEE Trans. Med. Imaging 40:71817–26
    [Google Scholar]
  64. Reinhard E, Adhikhmin M, Gooch B, Shirley P. 2001. Color transfer between images. IEEE Comput. Graph. Appl. 21:534–41
    [Google Scholar]
  65. Rolls GO, Farmer NJ, Hall JB. 2008. Artifacts in Histological and Cytological Preparations Deep Park, IL: Leica Microsyst.
    [Google Scholar]
  66. Saillard C, Schmauch B, Laifa O, Moarii M, Toldo S et al. 2020. Predicting survival after hepatocellular carcinoma resection using deep learning on histological slides. Hepatology 72:62000–13
    [Google Scholar]
  67. Saldanha OL, Quirke P, West NP, James JA, Loughrey MB et al. 2022. Swarm learning for decentralized artificial intelligence in cancer histopathology. Nat. Med. 28:1232–39
    [Google Scholar]
  68. Schirris Y, Gavves E, Nederlof I, Horlings HM, Teuwen J. 2022. DeepSMILE: contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer. Med. Image Anal. 79:102464
    [Google Scholar]
  69. Schmauch B, Romagnoni A, Pronier E, Saillard C, Maillé P et al. 2020. A deep learning model to predict RNA-Seq expression of tumours from whole slide images. Nat. Commun. 11:3877
    [Google Scholar]
  70. Schömig-Markiefka B, Pryalukhin A, Hulla W, Bychkov A, Fukuoka J et al. 2021. Quality control stress test for deep learning-based diagnostic model in digital pathology. Mod. Pathol. 34:122098–108
    [Google Scholar]
  71. Schulz S, Woerl A-C, Jungmann F, Glasner C, Stenzel P et al. 2021. Multimodal deep learning for prognosis prediction in renal cancer. Front. Oncol. 11:788740
    [Google Scholar]
  72. Sirinukunwattana K, Domingo E, Richman SD, Redmond KL, Blake A et al. 2021. Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning. Gut 70:3544–54
    [Google Scholar]
  73. Sounderajah V, Ashrafian H, Golub RM, Shetty S, De Fauw J et al. 2021. Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol. BMJ Open 11:6e047709
    [Google Scholar]
  74. Ström P, Kartasalo K, Olsson H, Solorzano L, Delahunt B et al. 2020. Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: a population-based, diagnostic study. Lancet Oncol 21:2222–32
    [Google Scholar]
  75. Tabibu S, Vinod PK, Jawahar CV. 2019. Pan-Renal Cell Carcinoma classification and survival prediction from histopathology images using deep learning. Sci. Rep. 9:10509
    [Google Scholar]
  76. Tellez D, Balkenhol M, Otte-Holler I, van de Loo R, Vogels R et al. 2018. Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks. IEEE Trans. Med. Imaging 37:92126–36
    [Google Scholar]
  77. van Treeck M, Cifci D, Laleh NG, Saldanha OL, Loeffler CML et al. 2021. DeepMed: a unified, modular pipeline for end-to-end deep learning in computational pathology. bioRxiv 10.1101/2021.12.19.473344. https://doi.org/10.1101/2021.12.19.473344
    [Crossref]
  78. Vanneman M, Dranoff G. 2012. Combining immunotherapy and targeted therapies in cancer treatment. Nat. Rev. Cancer 12:237–51
    [Google Scholar]
  79. Wang F, Oh TW, Vergara-Niedermayr C, Kurc T, Saltz J. 2012. Managing and querying whole slide images. Proc. SPIE Int. Soc. Opt. Eng. 8319:83190J
    [Google Scholar]
  80. Wang Y, Gu T, Tian X, Li W, Zhao R et al. 2021. A small molecule antagonist of PD-1/PD-L1 interactions acts as an immune checkpoint inhibitor for NSCLC and melanoma immunotherapy. Front. Immunol. 12:654463
    [Google Scholar]
  81. Warnat-Herresthal S, Schultze H, Shastry KL, Manamohan S, Mukherjee S et al. 2021. Swarm Learning for decentralized and confidential clinical machine learning. Nature 594:7862265–70
    [Google Scholar]
  82. Woerl A-C, Eckstein M, Geiger J, Wagner DC, Daher T et al. 2020. Deep learning predicts molecular subtype of muscle-invasive bladder cancer from conventional histopathological slides. Eur. Urol. 78:2256–64
    [Google Scholar]
  83. Xu H, Park S, Lee SH, Hwang TH. 2019. Spatial heterogeneity and organization of tumor mutation burden with immune infiltrates within tumors based on whole slide images correlated with patient survival in bladder cancer. J. Pathol. Inform. 13:100105
    [Google Scholar]
  84. Zeng Q, Klein C, Caruso S, Maille P, Laleh NG et al. 2022. Artificial intelligence predicts immune and inflammatory gene signatures directly from hepatocellular carcinoma histology. J. Hepatol. 77:1116–27
    [Google Scholar]
  85. Zhang H, Zhang F, Ren F, Wang Z, Rao X et al. 2019. Predicting tumor mutational burden from liver cancer pathological images using convolutional neural network. 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)920–25. New York: IEEE
    [Google Scholar]
/content/journals/10.1146/annurev-cancerbio-061521-092038
Loading
/content/journals/10.1146/annurev-cancerbio-061521-092038
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error