google ads

3D Imaging in Radiology: What A Long Strange Trip It's Been

3D Imaging in Radiology: What A Long Strange Trip It's Been

Elliot K. Fishman MD FACR
Department of Radiology
Johns Hopkins Hospital

 

“What a Long Strange Trip It’s Been” is a term often referring to the band the Grateful Dead but also is probably a good description of the road that 3D imaging has taken in Radiology. We have been part of this journey since the mid 1980’s and this exhibit describes some of the changes we have seen and looks forward to where the future might be.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

3D Imaging has and continues to be dependent on technology and the technologic advances in computer hardware and software. The hardware costs for state of the art 3D imaging in the 1980’s was $250-350,000 and now is usually under $10,000. Software development has matched the hardware advances and networking has allowed for the seamless and secure transfer of information. The other critical advance has been the CT scanners themselves where we have gone from datasets of under 100 slices to 3-5,000 slices. Scan slice thickness has gone from 3-4 mm to under 1mm. The introduction of Dual Source CT has also provided new capabilities for creating optimal datasets.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

What has not changed over the years perhaps is the “big picture” role of 3D imaging? When we consider its role in medical imaging we typically like to divide things into specific channels.

  • Optimize the detection of pathology on CT scans
  • Optimize the definition of the extent of disease processes including accurate staging of disease
  • Help with planning patient management and triage
  • Better selection of optimal treatment planning (surgical or laprascopic approach) with 3D being a tool for both the radiologist and the referring physician
  • Minimize medical error by providing a deeper understanding of the data as well as optimal display techniques which can help with lesion detection

 

 

3D medical imaging has always been closely aligned with 3D imaging or visualization in other domains. Whether the developments were in the world of movies (Lucasfilms, Pixar, Disney), video games (Microsoft, NVIDIA) or technology (Department of Defense, NASA) we shared a common need for "visualization" of ever increasing size and complex datasets. Two excellent quotes on visualization are provided and fit perfectly in the medical domain.

 

 

“ Scientific Visualization is concerned with exploring data and information in such a way as to gain understanding and insight into the data. The goal of scientific visualization is to promote a deeper level of understanding of the data under investigation and to foster new insight into the underlying processes, relying on the humans' powerful ability to visualize.”
Scientific Visualization, Techniques and Applications, R.A. Earnshaw (1992)

“Visualization is the process of transforming information into a visual form, enabling users to observe the information. The resulting visual display enables the scientist or engineer to perceive visually features which are hidden in the data but nevertheless are needed for data exploration and analysis.”
N. Gershon, From Perception to Visualization (1994)

 

1985-2017: What has changed?

  • CT Scan data (slice thickness, spatial resolution)
  • Network transfer of data
  • Computer hardware (cost and performance)
  • Computer software (including imaging algorithms)
  • Clinical need for interpretation of large datasets beyond axial imaging
  • Clinical applications that have been introduced (virt colon, coronary CTA, GI bleeding)
  • New technologies ( 3D modeling, deep learning)

 

3D Medical Imaging: The Changes Than Changed the Landscape

  • CT scan datasets ( 4 mm sections every 3 mm for 60-100 slices to .75 mm slices every . 5 mm for 800-2500 slices every case
  • Post processing speeds (decreased hardware costs by factor of 50X and speed increase by 1000X. We used to take 24 hours to process 60 slices and can process 1000 slice real time on an iPhone)
  • Reconstruction algorithmns available (Shaded Surface to Volume Rendering and MIP and now Cinematic Rendering)

 

3D Medical Imaging Hardware: Hopkins CT Experience

  • 1980-2017
  • Silicon Graphics O2 workstation
  • SGI O2 with VP500/VP1000 board
  • Dell 8100 with VolumePro 1000 board and/or NVIDIA G-Force 4
  • InSpace on Leonardo workstation (HP) with NVIDIA boards
  • iPads and iPhones with NVIDIA boards

 

3D Medical Imaging Hardware: Hopkins CT Experience

  • 1980-2017
  • Siemens Medical Systems: DEC PDP11
  • Pixar: Pixar I and II Image Computers
  • NeXT
  • Sun Workstations 3/160 thru Sparc 2
  • Silicon Graphics Onyx / Infinite Reality workstations

 

 

The Images tell the story of the journey. The 3D software did the processing on a DEC PDP-11 Computer which was part of the room full of computers for the CT scanner.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

 

The Pixar Image Computer was introduced in 1985-1986 and was the best performing system of its time doing an amazing 40 MIPS. The combination of Pixar Hardware and Software (VRT) began the era of great interest in CT post processing.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

 

LucasFilms 1985 (Pixar Image Computer). The original images which seem crude by current standards showed the use of both RGB and an Alpha channel and we could now have images that showed bone, muscle and soft tissue.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

 

3D imaging from Hopkins begins at 4:57 with other potential applications for the Pixar Image Computer also shown. This was from 1987. The CT reconstruction of 50-60 CT slices took over 24 hours to create 84 images that were then visualized as a video.

 

 

The introduction of the Pixar technology became front page news in many of the initial sites that were trying to develop medical applications including Johns Hopkins Hospital.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

“ Volumetric rendering differs from surface rendering in that all the information from the CT scans is preserved, not just surface boundaries. Object thickness and internal contours can be seen in the 3D projection.”
Volumetric Rendering Technique: Applications for Three-dimensional Imaging of the Hip
Fishman EK, Drebin RA, Ney DR et al.
Radiology 1987 Jun;163(3):737-738

3D Imaging in Radiology: What A Long Strange Trip It's Been

Volume Rendering was considered one of the key discoveries in the history of CT by Diagnosting Imaging in this timeline. The team at Hopkins was honored to share the credit with the Pixar team.

 

Image quality and speed on the Pixar computer produced some impressive images considering the datasets were 4 mm thick and reconstructed with 3 mm overlap. The 3D imaging began to play a critical role in orthopedic surgery.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

Here are some of the our original published work trying to document the role for 3D imaging in clinical practice. Some of this early work was crucial in getting reimbursement for 3D studies in practice.

  • Volumetric rendering techniques: applications for three-dimensional imaging of the hip.
    Fishman EK, Drebin B, Magid D, Scott WW Jr, Ney DR, Brooker AF Jr, Riley LH Jr, St Ville JA, Zerhouni EA, Siegelman SS. Radiology. 1987 Jun;163(3):737-8.
  • Three-dimensional imaging of acetabular trauma.
    Scott WW Jr, Magid D, Fishman EK, Riley LH Jr, Brooker AF Jr, Johnson CA. J Orthop Trauma. 1987;1(3):227-32.
  • Acetabular fractures: optimal imaging.
    Scott WW Jr, Fishman EK, Magid D. Radiology. 1987 Nov;165(2):537-9.
  • Three dimensional imaging in orthopedics: state of the art 1988.
    Fishman EK, Magid D, Ney DR, Kuhlman JE, Brooker AF Jr. Orthopedics. 1988 Jul;11(7):1021-6.
  • Three-dimensional reconstruction of the human body.
    Fishman EK, Drebin RA, Hruban RH, Ney DR, Magid D. AJR Am J Roentgenol. 1988 Jun;150(6):1419-20.

 

Here are some of the our original published work trying to document the role for 3D imaging in clinical practice. Some of this early work was crucial in getting reimbursement for 3D studies in practice.

  • Three-dimensional imaging and display of musculoskeletal anatomy.
    Fishman EK, Magid D, Ney DR, Drebin RA, Kuhlman JE. J Comput Assist Tomogr. 1988 May-Jun;12(3):465-7.
  • Interactive real-time multiplanar CT imaging.
    Ney DR, Fishman EK, Magid D, Kuhlman JE. Radiology. 1989 Jan;170(1 Pt 1):275-6
  • 2D and 3D computed tomography of the pediatric hip.
    Magid D, Fishman EK, Sponseller PD, Griffin PP. Radiographics. 1988 Sep;8(5):901-33.
  • The application of advanced CT imaging and computer graphics methods to reconstructive surgery of the hip.
    Robertson DD, Walker PS, Fishman EK, Mintzer CM, Poss R, Magid D, Granholm JW, Brooker AF, Essinger JR. Orthopedics. 1989 May;12(5):661-7.

 

 

Three-dimensional reconstruction of the Human Body
Fishman, EK, Drebin RA, Hruban RH et al.
AJR 1988 Jun;150(6):1419-1420

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

Over time Pixar left the computer hardware and medical business to concentrate on the movies (Toy Story) and led by Ed Catmull and Steve Jobs found great success and became part of Disney in the late 1990’s. Many other companies pursued the hardware for image processing with Silicon Graphics (SGI) being the most famous company for nearly a decade. We used many different platforms and it is amazing how we have gone from large computers like the Onyx (SGI) to the O2 (SGI) to the NeXT Computer and eventully to special purpose boards (Nvidia) with the interactive display now possible on iPads and iPhones.

3D Imaging in Radiology: What A Long Strange Trip It's Been

Silicon Graphics ONYX, and O2 and the NeXT Computer

 

"Server side rendering and processing allows manipulation of images to be done where the information is stored. The image current being viewed on the devise is the only one that is stored at that point in time. Because there is minimal information on the devise at any given time, the method of viewing radiology images is the most secure."
Cybersecurity in Radiology: Access of Public Hot Spots and Public Wi-Fi and Prevention of Cybercrimes and HIPAA Violations
Gerard P et al.
AJR 2013; 201:1186-1189

 

Review of 3D data on mobile devices can provide the ability to review images with patients at bedside or the clinic and well as provide low cost mobile solutions for radiologists and non-radiologists alike. Currently applications like Webviewer (Siemens Medical) are available in the Apple store and run on the iPad and iPhone. Numerous vendors provide similar products although widespread adoption has been slow.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

We have long predicted that 3D imaging would become mainstream in CT scanning (see article quotes). We still believe it will someday.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

3D Imaging has played a major role across imaging and its clinical applications continues to grow. Some of the applications include;

  • Orthopedic imaging (i.e. skeletal trauma)
  • Oncologic imaging (i.e. preoperative staging of pancreatic cancer and liver tumors)
  • Vascular imaging (i.e. planning aortic surgery and followup of patients post open surgery or grafts)
  • GI imaging (i.e. applications running from GI bleeding to virtual colonoscopy)
  • Renal imaging (i.e. detection and staging of renal masses and followup for tumor recurrence
  • Cardiac imaging (i.e. congenital anomalies to coronary artery imaging

 

Volume Rendering Technique (VRT)-Facts
  • Percentage classification technique
  • VRT assumes that a voxel can contain more than one tissue type and the amount of each tissue type is between zero and 100%
  • Each voxel is therefore accurately represented
  • The technique is implemented using a probabilistic classification involving a trapezoidal approximation
  • Each tissue type (bone, muscle,fat) is assigned a nominal value range that represents that tissue type
  • Trapezoids can be adjusted in real time
Maximum Intensity Projection Technique (MIP)-Facts
  • Considered to be a technique that falls between a thresholding technique and volume rendering
  • Evaluates each voxel along a line from the viewers eye through the volume of data and selects the maximum voxel value which is displayed
  • There are different implementation of MIP imaging so that there will be variations between different vendors
  • MIP has a number of known artifacts and potential pitfalls;
    • With MIP the brightest voxels always APPEAR to be in front, EVEN if they are in fact behind other structures
    • With MIP the brightest voxels always obscures all other structures behind or in front of them

 

The limitations of MIP vs VRT are illustrated in this potential renal donor where the location of the renal vein relative to the aorta can be a source of confusion

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

The information gained from going from axial CT to Multiplanar CT (MPR) to MIP and VRT is nicely shown in this case of active Crohns Disease.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

Do you see the duodenal mass which is a carcinoid tumor?

The 3D VRT in coronal plane nicely defines the 1 cm vascular lesion which was missed on the axial CT scan.

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

Angiodysplasia as a source of GI Bleed recognized on the 3D display

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

Interactive Analysis of 3D datasets is ideally done in real time by the radiologist combining axial, MPR and 3D imaging.

FinalDiagnosis is FNH (Focal Nodular Hyperplasia)

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

 

"Although these routine CT examinations may be diagnostic for pancreatic adenocarcinoma, they are inadequate for disease extent assessment given the lack of optimal multiphasic enhancement and use of thicker slice selection. These factors limit the ability to generate highly quality reformatted images and 3D reconstructions that are often necessary for accurate staging. It is therefore essential that these patients undergo MDCT angiogram using a dedicated pancreatic protocol."
Pancreatic Ductal Adenocarcinoma Radiology Reporting Template: Consensus Statement of the Society of Abdominal Radiology (SAR), and the American Pancreatic Association (APA)
Al-Hawary MH, Francis IR, Chari ST, Fishman EK et al.
Radiology 2014;270:248-260

 

Interactive Volume Rendering of the Pancreas and Arterial Map

3D Imaging in Radiology: What A Long Strange Trip It's Been

 

Main Duct IPMN with VRT

Interactive 3D analysis

 

Serous Cystadenoma (MIP vs VRT)

Interactive 3D analysis

 

Conclusions

"The present study showed that VRT and axial images produce almost identical values of CI, with the advantages of greater ease of execution and a time saving of almost 50% for 3D VRT images. In addition, VRT provides an integrated perspective that can better assist surgeons in clinical decision making and in operative planning, suggesting this technique as a possible standard method for CI measurement."
Value of three-dimensional volume rendering images in the assessment of the centrality index for preoperative planning in patients with renal masses.
Sofia C et al.
Clin Radiol 2017 Jan;72(1):33-40

 

Clear Cell RCC

Clear Cell RCC

 

 

"The genetic makeup of clear cell RCCs (ccRCCs) affects their imaging features at multidetector CT examinations. Multidetector CT imaging characteristics may help suggest differences at the cytogenetic level among ccRCCs."
Clear Cell Renal Cell Carcinoma: Multiphasic Multidetector CT Imaging Features Help Predict Genetic Karyotypes
Sauk SC et al.
Radiology 2011; 261:854-862

 

 

New rendering algorithms will help push 3D rendering forward. A novel technique called Cinematic Rendering seems to have some advantages over classic volume renderings. 3D Imaging in Radiology: What A Long Strange Trip It's Been

 

 

“In addition to the fact that photo-realistic volume renderings tend to be aesthetically more pleasing, it has been shown that realistic lighting contributes to 3D understanding and can improve depth-related task performance . With this work and the implementation that we have made available, we hope to contribute to the uptake of realistic illumination in interactive direct volume rendering applications.”
Exposure Render: An Interactive Photo-Realistic Volume Rendering Framework
Thomas Kroes et al.  
PLOS ONE 7(7): e38586. doi: 10.1371/journal.pone.0038586

 

 

“Cinematic rendering produces volume rendered images with photorealistic image quality. It uses a global illumination model, which takes direct and indirect illumination into account when constructing an image, to achieve rendering quality. The mathematical models that describe this visualization technology include complex integral equations that are solved numerically using the Monte Carlo integration. The result of the integration is a numerical rendering algorithm known as path tracing: thousands of light rays are traced to compute the resulting image.”
MDCT angiography with 3D rendering: A novel cinematic rendering algorithm for enhanced anatomic detail
Johnson PT, Schneider R, Lugo-Fagundo C, Johnson M, Fishman EK
AJR Am J Roentgenol. 2017 Aug;209(2):309-312

 

Cinematic Rendering of the Normal Pancreas

Cinematic Rendering of the Normal Pancreas

 

1 cm PNET of the Pancreas

1 cm PNET of the Pancreas

 

Trauma is a classic clinical application for 3D. Here is a mandibular fracture imaged with cinematic rendering

Trauma is a classic clinical application for 3D.  Here is a mandibular fracture imaged with cinematic rendering

 

Visualization of the neck with detailed vascular, soft tissue and muscle anatomy is often critical across a range of applications from trauma to infection to oncology

Visualization of the neck with detailed vascular, soft tissue and muscle anatomy is often critical across a range of applications from trauma to infection to oncology

 

3D Cinematic Rendering Can Provide Detail of the Vessels as well as Muscle and the soft tissue.

3D Cinematic Rendering Can Provide Detail of the Vessels as well as Muscle and the soft tissue.

 

New techniques in 3D can be used beyond the walls of the hospital

New techniques in 3D can be used beyond the walls of the hospital

 

3D imaging can be used as the basis for printing complex anatomy with 3D printing

3D imaging can be used as the basis for printing complex anatomy with 3D printing

 

3D mapping solves problems that may even be centuries old

3D mapping solves problems that may even be centuries old

 

Summary

The journey we are on changes every day and we look forward to new opportunities for improving the role of 3D in medical imaging. With a limited number of slides we could barely scratch the surface of the field and the changes over the years. We are optimistic that the best is yet to come.

"You can't connect the dots looking forward; you can only connect them looking backward. So you have to trust that the dots will somehow connect in your future. You have to trust in something - your gut, destiny, life, karma, whatever. This approach has never let me down, and it has made all the difference in my life."
Steve Jobs

Privacy Policy

Copyright © 2023 The Johns Hopkins University, The Johns Hopkins Hospital, and The Johns Hopkins Health System Corporation. All rights reserved.