Deep Learning Exhibits ❯ RSNA 2019

Multidisciplinary approach for program development with artificial intelligence in pancreatic cancer: How we fit in

 

 

Multidisciplinary approach for program development with artificial intelligence in pancreatic cancer: How we fit in

Satomi Kawamoto, M.D., Seyoun Park, Ph.D. , Linda Chu, MD, Alejandra Blanco, M.D., Shahab Shayesteh, M.D., Saeed Ghandili, M.D., Daniel Fadaei Fouladi, M.D., Alan Yuille, Ph.D., Ralph Hruban, M.D., Kenneth Kinzler, Ph.D., Bert Vogelstein, M.D., Elliot K. Fishman, M.D.

The Russell H. Morgan Department of Radiology and Radiological Science, Department of Computer Science, Department of Pathology, and the Department of Cancer Research
Johns Hopkins University

 

Disclosure

Research supported by the Lustgarten Foundation
  • Satomi Kawamoto
  • Seyoun Park
  • Linda Chu
  • Alejandra Blanco
  • Shahab Shayesteh
  • Saeed Ghandili
  • Daniel Fadaei Fouladi
  • Alan Yuille
  • Ralph Hruban
  • Kenneth Kinzler
  • Bert Vogelstein
  • Elliot K. Fishman

 

Learning Objectives

  • To review roles of radiologists in development of artificial intelligence (AI) applications for early detection of pancreatic cancer on CT as a part of multidisciplinary team
  • To review tasks involving development of AI applications and challenges of each task.
  • To review roles of radiologist in each task involving development of AI applications

 

Introduction (1)

  • Recent rapid growth of artificial intelligence (AI) in radiology has been mostly driven by computer scientists, engineers, and commercial companies. Radiologists generally have much less direct participation.
  • However, radiologists can play a critical role in designing study, clinical applications and development of AI in medical imaging as a member of multidisciplinary team.
  • Multidisciplinary approach has advantages in development of AI by working closely with expertise of different fields and fine tuning the process in real time.
Rubin DL. Artificial intelligence in imaging: The radiologist’s Role. J Am Coll Radiol 2019;16:1309-1317.

 

Introduction (2)

  • Radiologists play an important role in designing study and clinical applications and provide expertise in assuring accuracy of image segmentation highlighting critical imaging features, data acquisition, data collection, reviewing the results by algorithm, providing feedback to improve deep learning algorithms, and determining evaluation metrics in multidisciplinary team.
  • We will review our experience of multidisciplinary collaboration in the development of successful AI program with the primary goal of early detection of pancreatic cancer.

 

Deep Learning Applications in Cancer Imaging

  • Artificial intelligence (AI) offers the opportunity to transform image interpretation from a purely qualitative and subjective task to effortless quantifiable and reproducible task.
  • Deep learning (DL), a subset of AI, uses training data and multiple layers of equations to develop a mathematical model that fits the data.
  • Convolutional neural network (CNN), a typical representative of deep learning, has been validated as an effective approach for radiological image classification in various studies.

 

Goals of Deep Learning Models

To create mathematical models that can be:
  • Trained: to create mathematical models that can be trained to produce useful outputs when input data are provided.
  • Tuned: to produce accurate predictions for the training data by an optimization algorithm
  • Generalized: to deliver correct predictions for new unseen data from their learned experience.
Lundervold AS, et al. Z Med Phys 2019;29:102-127.

 

Multidisciplinary Approach in Development of AI Algorithm for Detection of Early Pancreatic Cancer

Radiologists play a critical role in development of AI in medical imaging as a member of multidisciplinary team

Multidisciplinary Approach in Development of AI Algorithm for Detection of Early Pancreatic Cancer

 

Roles of Radiologists in Development of DL algorithms

Development of deep learning algorithms for early detection of pancreatic cancer

Roles of Radiologists in Development of DL algorithms

 

Image Acquisition

  • Proper data collection of large-set images is critical to the successful training of a deep convolutional neural network.
  • Radiology practice and is different in each institution and may change over time, e.g., patient populations, imaging equipment, imaging protocols.
  • These factors may change how well an AI algorithm performs on the images.
  • AI algorithm needs to be refined when data from different protocols, venders, and institutions are used to ensure its general applicability.
  • Radiologists need to assure the data were acquired properly.

 

Data Collection

  • One of the key elements of the success of CNN applied to natural images is the availability of very large-scale annotated datasets (e.g., 1.4 millions of images in ImageNet)
  • As data sets get bigger and computers become more powerful, the results achieved by deep learning will get better.
  • However, in medical images, the datasets are often very limited compared to natural images.
  • To overcome the problem of insufficient data, fine-tuned DNN from natural images (i.e., data augmentation by scaling, rotating, or elastic deformation, and data synthesis) may be used to increase the number of training sets.
Deng J, et al. IEEE Conference on. IEEE. 2009;248-255. Hinton G. JAMA 2018;320:1101.

 

Data Collection/Segmentation Process and Workflow

We developed an unique and reliable data collection and annotation process for normal abdominal structures using volumetric CT that can be used to train the deep learning network for automated recognition of normal abdominal organs.

Data Collection/Segmentation Process and Workflow

Park S, et al. Annotated normal CT data of the abdomen for deep learning. Diagnostic and Interventional Imaging 2019 [Epub ahead of print]

 

Image Segmentation

  • Segmentation of the internal structures, like body organs, in medical images is an essential task for many clinical applications.
  • Deep learning has largely relied on supervised learning approaches, i.e., large amounts of training data that composed of input images and ground truth annotations.
  • Radiologists play a critical role for accuracy of segmentation of target structures and annotation of pathology.
  • More than one radiologist could be engaged to review the images to have a better ground truth.

 

Image Segmentation (Normal CT)

Image segmentation: Partitioning an image into multiple regions that share similar attributes, enabling localization and quantification.

Manually segmented normal abdominal structures on Coronal CT image

Image Segmentation (Normal CT)

Park S, et al. Diagnostic and Interventional Imaging 2019 [Epub ahead of print]

 

2D vs. 3D Image Segmentation

  • Software tools can greatly speed up the segmentation process.
  • The ideal software functionality would not be limited to basic two-dimensional (2D) drawings, but would include efficient patient-list management, visualization of annotated results, convenient graphic user interfaces for segmentation including advanced 3D segmentation, and automatic backup. 
2D vs. 3D Image Segmentation

Park S, et al. Diagnostic and Interventional Imaging 2019 [Epub ahead of print]

 

Deep Network Predictions

  • Example of manual segmentation and DNN prediction of normal abdominal organs in 3D rendering.
  • In our series using 236 normal CT data, deep network prediction for the pancreas showed high fidelity with Dice similarity coefficient of 87.8 ± 3.1% and mean surface distances of 1.05 ± 0.65 mm.
Deep Network Predictions

Wang Y, et al. http://arxiv.org/abs/1804.08414
Park S, et al. Diagnostic and Interventional Imaging 2019 [Epub ahead of print]

 

Image Segmentation (Pancreatic Cancer)

  • We developed a data collection and annotation process of for pancreatic ductal adenocarcinoma (PDAC) and Pancreatic neuroendocrine tumor (PNET) using volumetric CT.
  • In this case, small PDAC in the uncinated process (allow) along with other abdominal organs were segmented.
  • Radiologist plays a critical role for accuracy of segmentation of target structures and annotation of pathology.
Image Segmentation (Pancreatic Cancer)

 

Supervised vs. Unsupervised Learning (1)

  • Accurate segmentation for supervised learning, especially by manual segmentation, requires experienced radiologists and is tedious and time consuming.
  • As the size of the imaging datasets has increased, the time and cost required for the labor intensive process of manual segmentation has become more difficult.
  • Interobserver and intraobserver variability are also an issue for manual segmentation.
  • Semi-supervised or unsupervised learning approaches have been used to circumvent the need for large number of fully annotated datasets.

 

Supervised vs. Unsupervised Learning (2)

  • Semi-supervised learning uses weakly annotated data, such as a bounding box of the target region. 
  • Unsupervised learning employs a combination of detection and segmentation from raw images in the absence of any manual annotation. 
  • However, semi-supervised and unsupervised approaches still require some annotated data for training. In addition, learning accuracy and convergences are clearly superior with supervised learning than with semi-supervised or unsupervised methods.
  • To maximize the possibility and utility of deep learning, it is therefore optimal to have high-quality annotated data.

 

Training and Testing in Deep Network

  • After algorithm is tested and the results are reviewed by multidisciplinary team.
  • Radiologists review the cases and provide feedback to the team to improve algorithms using their expertise.
    • To confirm if deep network prediction is correct.
    • To find out true lesion in false negative cases
    • To find out potential reasons for false positive prediction
    • To find out any potential annotation errors
  • From the beginning of this project, we had weekly meeting to constantly review the annotation, data collection and algorithms to fine tune our processes in real time.

 

Example of False Negative Case of Deep Network Prediction

Pancreatic neuroendocrine tumor in the body of the pancreas was not predicted by deep network.
Radiologists review the casesand provide feedback to the multidisciplinary team to improve algorithms.

Example of False Negative Case of Deep Network Prediction

 

Example of False Negative Case of Deep Network Prediction

Pancreatic neuroendocrine tumor in the head of the pancreas was not predicted by deep network.
Radiologists review the casesand provide feedback to the multidisciplinary team to improve algorithms.

Example of False Negative Case of Deep Network Prediction

 

Example of False Positive Case of Deep Network Prediction

Uneven focal fatty infiltration of the head of the pancreas might be related to false positive prediction
False positive PDAC prediction in the head of the pancreas

Example of False Positive Case of Deep Network Prediction

Wang Y, Zhou Y, Shen W, Park S, Fishman EK, Yuille AL. Abdominal multi-organ segmentation with organ-attention networks and statistical fusion. https://arxiv.org/abs/1804.08414

 

Deep Network Prediction for Detection of PNET Found Multiple Tumors: Need Confirmation of Multi-tumor Cases vs. Annotation/Segmentation Error

Pancreatic serous cystadenoma (blue arrow) in the head of the pancreas and neuroendocrine tumor (yellow arrow) in the tail of the pancreas. Radiologists confirmed this was a single PNET case with incidental other tumor by reviewing the CT and pathology data.

Deep Network Prediction for Detection of PNET Found Multiple Tumors: Need Confirmation of Multi-tumor Cases vs. Annotation/Segmentation Error

 

Evaluation of AI Tools

  • Choose appropriate evaluation metric for the algorithm.
  • Commonly used metrics are sensitivity, specificity, positive and negative predictive values.
  • A reliable gold standard (ground truth) is critical.
Evaluation of AI Tools

 

Evaluation of AI Tools

  • Receiver operating characteristic (ROC) curve: curvilinear graph generated by plotting true positive ratio as a function of false negative ratio.
  • A given AI algorithm has a particular ROC curve.
  • A particular point on ROC curve can be selected by AI developer to determine the output of the AL algorithm.
Evaluation of AI Tools

Rubin DL. J Am Coll Radiol 2019;16:1309-1317

 

Determining Clinical Needs

  • Need to consider clinical scenarios for which AI algorithms are used.
  • A performance threshold can be adjusted based on clinical scenario.
    • For example, to determine if high sensitivity (cancer detection) or specificity (ruling out an abnormality) is most important depending on the clinical scenarios
    • To determine the tolerance of the number of false positives and false negatives.
    • There is a trade-offs in performance metrics for AI algorithm.

 

Discussion (1)

  • Although generally, radiologists currently have little formal role in the development of AI tools other than being the targeted consumers, radiologists can have a active role in development of AI tools in medical imaging as a member of multidisciplinary team.
  • Multidisciplinary collaboration with expertise of different fields working closely has advantages in development of AI tools for early detection of pancreatic cancer.

 

Discussion (2)

  • Radiologists provide expertise in assuring accuracy of image segmentation and annotation, and highlighting critical imaging features.
  • Radiologists also play an important role in designing study and clinical applications, data acquisition, data collection, reviewing the results and providing feedback to improve deep learning algorithms.

 

Conclusion

  • Radiologist can have an active role in guiding and facilitating the development and implementation of AI tools in diagnostic radiology.
  • We believe that a multidisciplinary team approach is key to the success of this effort.

 

References

  • Rubin DL. Artificial intelligence in imaging: The radiologist’s Role. J Am Coll Radiol 2019;16:1309-1317.
  • Wang Y, Zhou Y, Shen W, Park S, Fishman EK, Yuille AL. Abdominal multi-organ segmentation with organ-attention networks and statistical fusion. http://arxiv.org/abs/1804.08414
  • Hinton G. Deep learning – A technology with the potential to transform health care. JAMA 2018;320:1101-1102
  • Deng J, et al. ImageNet: a large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009, IEEE Conference on. IEEE. 2009;248-255.
  • Lundervold AS, et al. An overview of deep learning in medical imaging focusing on MRI. Z Med Phys 2019;29:102-127.
  • Park S, et al. Annotated normal CT data of the abdomen for deep learning. Diagnostic and Interventional Imaging 2019 [Epub ahead of print]
  • Shin H, et al. Deep convolutional neural networks for computer-aided detection : CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 2016;35:1285–98.
  • Chu L, et al. Application of deep learning to pancreatic cancer detection lessons learned from our initial experience JACR 2019;9:1338
  • Zhu Z, Fishman EK, Yuille AL, et al. Multi-scale coarse-to-fine segmentation for screening pancreatic ductal adenocarcinoma. https://arxiv.org/pdf/1807.02941.
  • Liu F, Fishman EK, Yuille AL, et al. Joint Shape Representation and Classification for Detecting PDAC in abdominal CT scans. https://arxiv.org/pdf/1804.10684.
© 2020 Elliot K. Fishman, MD, FACR
All Rights Reserved.
www.CTISUS.com
CTisus CT Scanning CTisus CT Scanning CTisus CT Scanning CTisus CT Scanning CTisus CT Scanning CTisus CT Scanning CTisus CT Scanning CTisus CT Scanning CTisus CT Scanning