Imaging Pearls ❯ Deep Learning ❯ Human Interface
|
-- OR -- |
|
- Let us be brutally honest with ourselves: in many technical aspects, the machines will win. We must not dismiss the power of AI or pretend it is just another iterative tool. It is a formidable, paradigm-shifting technology. Soon, AI will seamlessly synthesize trillions of data points, catch invisible clinical patterns, and compute probabilities with a precision humans simply cannot match. Faced with this reality, the era of expecting physicians to act as human algorithms, as the benchmark interpreter of clinical data and evidence-based guidelines, is coming to an end.
Irreplaceable: Five Enduring Roles of the Physician in the AI Era.
Lin S.
Fam Med. 2026;58(5):397-398 - So how do we prepare for this new paradigm of doctoring? The future of our profession remains in our classrooms and clinics. To prepare our learners for the age of AI, we must fundamentally change how we teach. If we continue the status quo, we are training them to compete in a race they have already lost. Instead, our training must evolve to select for and cultivate empathy, resilience, deep listening, and moral courage. We must teach them how to be guides through complex lives, anchors in the storm, motivators of the weary, advocates against broken systems, and healers with their hands.
Irreplaceable: Five Enduring Roles of the Physician in the AI Era.
Lin S.
Fam Med. 2026;58(5):397-398 - In other words, every physician needs to be more like a family physician. My colleagues, we stand at the threshold of an identity defining transformation. For too long, we have been forced to act like machines—staring at screens, clicking boxes, and typing away our sanity. If machines are here to take that work back, I say we let them. The AI can do the computational heavy lifting, so we can do the relational heavy lifting. It can and should be a beautiful partnership. So do not fear AI. Engage it. Guide it. The future of medicine is not automated. The future is deeply, profoundly human. And in that future, you are irreplaceable.
Irreplaceable: Five Enduring Roles of the Physician in the AI Era.
Lin S.
Fam Med. 2026;58(5):397-398 - * THE REAL-WORLD GUIDE
* THE HUMAN ANCHOR
* THE MOTIVATOR
* THE SYSTEM ADVOCATE
* THE HANDS-ON HEALER
Irreplaceable: Five Enduring Roles of the Physician in the AI Era.
Lin S.
Fam Med. 2026;58(5):397-398
- "Observing the digital footprints of clinicians’ current wayfinding activities will reveal where AI could optimize information navigation and decision-making. Doing so may enable discovery of more efficient and accurate diagnostic pathways. For instance, an AI algorithm may prioritize a lung nodule excision instead of the traditional intermediary steps of laboratory tests, additional imaging, or observation periods. This may appear illogical or anomalous, but the AI system, unconstrained by standard practices that make sense to clinicians, may uncover the safety and efficiency of alternative approaches that inform wayfinding guidance.”
Next-Generation Artificial Intelligence for Diagnosis From Predicting Diagnostic Labels to “Wayfinding”
Julia Adler-Milstein et al.
JAMA( Published online)December 9, 2021 - “To develop wayfinding AI tools, new types of data assets need to be generated. Alongside traditional patient-centric (clinical) information, new clinician-centric data are needed that capture clinicians’ actions during the diagnostic process (eg, what data clinicians typically review when evaluating a patient with low back pain) and the contextual factors that surround the clinician and patient during this process (eg, team structure, patient volume). Given the breadth of diagnostic scenarios, initial data sets should focus on characterizing the information processing and decision nodes in the diagnostic process for common symptoms.”
Next-Generation Artificial Intelligence for Diagnosis From Predicting Diagnostic Labels to “Wayfinding”
Julia Adler-Milstein et al.
JAMA( Published online)December 9, 2021 - • Natural language processing describes computer programming that aims to process or generate natural language data, and it is largely divided into symbolic (eg, rule-based) and statistical (eg, ma- chine learning) approaches.
• Common tasks useful in radiology report NLP include document classification, sentence classification, named entity recognition, relation extraction, automatic summarization, question answering, and image captioning.
Basic Artificial Intelligence Techniques Natural Language Processing of Radiology Reports
Jackson Steinkamp, Tessa S. Cook
Radiol Clin N Am 59 (2021) 919–931 - Natural language processing (NLP) is a subfield of computer science and linguistics. NLP involves the creation and study of computer programs that interact with human language data, including written, typed, or spoken language. Specific NLP applications might extract relevant pieces of information from language data, generate lan- guage data automatically, or change the form of language data (eg, translation, text-to-speech software).
Basic Artificial Intelligence Techniques Natural Language Processing of Radiology Reports
Jackson Steinkamp, Tessa S. Cook
Radiol Clin N Am 59 (2021) 919–931 - “DL radiology NLP approaches have become popular in recent years. The use cases are similar to those for symbolic and non-DL statistical NLP and include quantifying oncologic response, identifying pulmonary emboli, flagging critical findings, and detecting follow-up recommendations. Broader information extraction from radiology reports has also been demonstrated.”
Basic Artificial Intelligence Techniques Natural Language Processing of Radiology Reports
Jackson Steinkamp, Tessa S. Cook
Radiol Clin N Am 59 (2021) 919–931 - "Because of the inherent variety and variability in radiology reports, NLP has become a valuable technique for extracting discrete data and meaningful information from reports. Both symbolic and statistical approaches exist, and fundamental NLP tasks can often be applied to a radiology corpus. As with other kinds of ML, a great deal of attention must be paid to preparation of the data to be used. There are numerous use cases for radiology NLP, and ongoing efforts such as the development of standardized reporting tem- plates and common data elements will only augment more powerful DL-based techniques that continue to be developed.”
Basic Artificial Intelligence Techniques Natural Language Processing of Radiology Reports
Jackson Steinkamp, Tessa S. Cook
Radiol Clin N Am 59 (2021) 919–931
- “Briefly, deep learning systems for imaging use multilayer neural networks to transform input images into useful outputs. A deep learning system learns not only the mappings of image features to the outputs but also the image features them- selves. Example outputs include image categories (for image classification), object locations (for detection), and pixel labels (for segmentation). For image analysis, the fundamental architecture of deep learning systems is the convolutional neural network (CNN). A CNN designed for im- ages contains convolutional layers that compare overlapping rectangular patches of the input to small learnable weight matrices (termed kernels or filters) that encode features.”
Deep Learning: An Update for Radiologists
Cheng PM et al.
RadioGraphics 2021; 41:1427–1445 - “Training an effective CNN is dependent on labeled data. In classification, the data are images with category labels. In detection, the data are images and rectangular bounding box coordi- nates delimiting features of interest. In segmenta- tion, the data are images and image masks that provide labels for each pixel or voxel. Preparing medical image data for machine learning tasks is a complex process that has been reviewed in detail. For deep learning, it is critical to have training images that are representative of the task to be solved. Images from a single medical center may be insufficient to train a model for a given task or may be biased because of the sampled population. Multicenter datasets help to address these problems but introduce challenges related to privacy as well as standardization of image acquisition and labels.”
Deep Learning: An Update for Radiologists
Cheng PM et al.
RadioGraphics 2021; 41:1427–1445 - "With limited data, it is easy for a model to be trained to the point of predicting labels perfectly on the training data but poorly on new data; such a model is said to overfit the training set or to exhibit poor generalization. One common way to expand the training dataset to prevent overfitting is image augmentation. Simple methods of increasing the number of training images include random translations, rotations, flips, scalings, crops, and brightness and contrast adjustments. There has also been interest in generative adversarial networks (GANs) (discussed further in this article) to produce fake images that resemble real images.”
Deep Learning: An Update for Radiologists
Cheng PM et al.
RadioGraphics 2021; 41:1427–1445 - "Medical images need labels to be used for supervised learning, the most common form of machine learning, in which the goal is to predict labels for new inputs. Depending on the task, labels for classification may arise from radiology reports, expert reviews, or clinical or pathologic data. Labels for detection and segmentation tasks are more complicated and time-consuming to create compared with classification datasets. Dis- tributing the labeling task among more human labelers reduces the labeling burden on individuals but increases overall labeling work and raises consistency issues that may require averaged or consensus labels among several labelers.”
Deep Learning: An Update for Radiologists
Cheng PM et al.
RadioGraphics 2021; 41:1427–1445 
Deep Learning: An Update for Radiologists
Cheng PM et al.
RadioGraphics 2021; 41:1427–1445
- "Diagnostic error, though, has po- tential to become the area of AI in medicine of which patients become most cognizant. Therefore, the legal rami!cations should also be considered in the clinical implementation of AI. Like humans, AI is not perfect. The IBM Watson Health’s cancer AI algorithm (known as Watson for Oncology) was trained on a small number of synthetic cases with limited input from oncologists. Consequently, many output treatment recommendations were erroneous and potentially harmful. If a clinician errs based on "awed AI decision support, who is legally responsible? The black-box nature of several AI algorithms also complicates efforts to ascertain exact causes of errors. Medical AI systems are too new to have been involved in malpractice lawsuits, and it remains to be seen where responsibilities lie.”
Using Artificial Intelligence to Interpret CT Scans: Getting Closer to Standard of Care.
Weisberg EM, Chu LC, Fishman EK.
J Am Coll Radiol. 2021 Jun 17:S1546-1440(21)00461-0. doi: 10.1016/j.jacr.2021.05.008. - "For now, we can tell patients that AI is already being used in several stages of the patient experience, is in increasing use in the emergency setting of select in- stitutions, and although it has great potential in diagnosis, it is not in general use at this time but perhaps in the near future. Just when remains dif!cult to predict, but we are paying attention!”
Using Artificial Intelligence to Interpret CT Scans: Getting Closer to Standard of Care.
Weisberg EM, Chu LC, Fishman EK.
J Am Coll Radiol. 2021 Jun 17:S1546-1440(21)00461-0. doi: 10.1016/j.jacr.2021.05.008.
- “To prevent patient privacy compromise while promoting scientific research on large datasets that aims to improve patient care, the implementation of technical solutions to simultaneously address the demands for data protection and utilization is mandatory. Here we present an overview of current and next-generation methods for federated, secure and privacy-preserving artificial intelligence with a focus on medical imaging applications, alongside potential attack vectors and future prospects in medical imaging and beyond.”
Secure, privacy-preserving and federated machine learning in medical imaging
Georgios A. Kaissis et al.
Nat Mach Intell 2, 305–311 (2020) 
Secure, privacy-preserving and federated machine learning in medical imaging
Georgios A. Kaissis et al.
Nat Mach Intell 2, 305–311 (2020)- Key Points
• Successful implementation of AI in radiology requires collaboration between radiologists and referring clinicians.
• Implementation of AI in radiology is facilitated by the presence of a local champion.
• Evidence on the clinical added value of AI in radiology is needed for successful implementation.
Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors
Lea Strohm et al.
Eur Radiol (2020). https://doi.org/10.1007/s00330-020-06946-y - “Considering the great attention AI applications are receiving in radiology and other medical disciplines like pathology, un- derstanding the barriers of and facilitators for the implemen- tation of AI is important. One of the important facilitating factors is the presence of a “local champion,” an individual with a strong personal interest in AI applications who most often initiates and actively advances AI implementation in the organization.”
Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors
Lea Strohm et al.
Eur Radiol (2020). https://doi.org/10.1007/s00330-020-06946-y - "Among the most prominent hindering factors is the uncertain added value for clinical practice, which causes low acceptance of AI applications among adopters and complicates the mobilization of funds to acquire AI applications. Furthermore, the failure to include all relevant stakeholders in the planning, execution, and monitoring phase of the implementation of AI applications was found to be a major hindering factor.”
Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors
Lea Strohm et al.
Eur Radiol (2020). https://doi.org/10.1007/s00330-020-06946-y - “To increase the acceptance among adopters, more evidence of the added benefit of their AI applications in the clinical setting is needed. Also, all involved stakeholders (most notably radiologists and referring clinicians) should be included in the decisions for and the design of implementation processes of AI applications.”
Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors
Lea Strohm et al.
Eur Radiol (2020). https://doi.org/10.1007/s00330-020-06946-y - “In this study, we used artificial intelligence (AI) algorithms to integrate chest CT findings with clinical symp- toms, exposure history and laboratory testing to rapidly diagnose patients who are positive for COVID-19. Among a total of 905 patients tested by real-time RT–PCR assay and next-generation sequencing RT–PCR, 419 (46.3%) tested positive for SARS-CoV-2. In a test set of 279 patients, the AI system achieved an area under the curve of 0.92 and had equal sensitivity as compared to a senior thoracic radiologist. The AI system also improved the detection of patients who were positive for COVID-19 via RT–PCR who presented with normal CT scans, correctly identifying 17 of 25 (68%) patients, whereas radiologists classified all of these patients as COVID-19 negative. When CT scans and associated clinical history are available, the proposed AI system can help to rapidly diagnose COVID-19 patients.”
Artificial intelligence–enabled rapid diagnosis of patients with COVID-19
Xueyan Mei et al.
Nat Med (2020). https://doi.org/10.1038/s41591-020-0931-3
- “It is essential to develop technology that empowers doctors so that they can get back to doing what they trained for and love. It is equally important that we return to patients their doctors’ undivided attention. Accordingly, healthcare IT development should begin with a deep understanding of how clinicians need and want to work, then implement AI capabilities with the explicit goal of adapting to and supporting how they deliver care. Ambient clinical intelligence (ACI) is one promising approach.”
How AI in the Exam Room Could Reduce Physician Burnout
Michael Ash, Joe Petro, Shafiq Rab
Harvard Business Review
November 2019 - “As the name indicates, ACI is less a device than a set of capabilities as unobtrusively present and available as the light and sound in the exam room. The best way to picture ACI is to think of a typical exam room with a flat-screen display on the wall showing requested information. An inconspicuous array of microphones captures the patient interaction accurately regardless of speakers’ movements or positions. A computer isn’t needed in the exam room, because the computing and data entry takes place behind the scenes in back-end and cloud- based systems. ACI builds on the familiar speech recognition technology that doctors have used for the past 20 years. It also uses voice biometrics — in short, a way to identify individuals by voice — to authenticate clinical users, and other technologies to distinguish between the clinician, patient, and anyone else in the exam room. It also integrates conversational AI, machine learning, speech synthesis, natural language understanding, and cloud computing to provide diagnostic guidance and clinical intelligence.”
How AI in the Exam Room Could Reduce Physician Burnout
Michael Ash, Joe Petro, Shafiq Rab
Harvard Business Review
November 2019 - “Radiologists will remain ultimately responsible for patient care and will need to acquire new skills to do their best for patients in the new AI ecosystem. The radiology community needs an ethical framework to help steer technological development, influence how different stakeholders respond to and use AI, and implement these tools to make best decisions and actions for, and increasingly with, patients.”
Ethics of artificial intelligence in radiology: summary of the joint European and North American multisociety statement
J. Raymond Geis et al.
Insights into Imaging (2019) 10:101
- “When you ask a medical doctor why he diagnosed this or this, he’s going to give you some rea- sons,” he says. “But how come it takes 20 years to make a good doctor? Because the information is just not in books.”
Can we open the black box of AI? Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn. Davide Castelvecchi Nature Vol 538, Issue 7623 Oct 2016 - “Each of these tasks is amenable to automa- tion. Organs can be located by the computer using atlas- and landmark-based methods. Organ volume and shape can be assessed by finding the edges of the organs in three dimensions, a process known as segmentation. Lesions can be detected and segmented by assessing the patterns of Hounsfield unit intensities in the organs to identify anomalies. Example pat- terns include variations in intensities, texture, and shape. The quantitative measurements of these patterns are known as features.”
Progress in Fully Automated Abdominal CT Interpretation Summers RM AJR 2016; 207:67–79 - “In the other, generic features are used and a machine-learning algorithm is taught to distinguish disease from nondisease sites by being trained on labeled cases, without the need for handcrafted features. The latter approach, which is made feasible by recent advances in computer science known collo- quially as deep learning, is increasingly being used because it markedly increases the efficiency of image analysis development.”
Progress in Fully Automated Abdominal CT Interpretation Summers RM AJR 2016; 207:67–79 - “To perform fully automated abdominal CT image interpretation at the level of a trained ra- diologist, the computer must assess all the or- gans and detect all the abnormalities present in the images. Although this is a seemingly daunting task for the software developer, the numbers of organs and potential abnormalities are finite and can be addressed methodically .” Progress in Fully Automated Abdominal CT Interpretation Summers RM AJR 2016; 207:67–79
- “The purpose of this review is to discuss developments in computational image analysis tools for predictive modeling of digital pathology images from a detection, segmentation, feature extraction, and tissue classification perspective. We discuss the emergence of new handcrafted feature approaches for improved predictive modeling of tissue appearance and also review the emergence of deep learning schemes for both object detection and tissue classification. We also briefly review some of the state of the art in fusion of radiology and pathology images and also combining digital pathology derived image measurements with molecular "omics" features for better predictive modeling. The review ends with a brief discussion of some of the technical and computational challenges to be overcome and reflects on future opportunities for the quantitation of histopathology.”
Image analysis and machine learning in digital pathology: Challenges and opportunities. Madabhushi A1, Lee G2. Med Image Anal. 2016 Oct;33:170-5
- “The nationwide implementation of electronic medical records (EMRs) resulted in many unanticipated consequences, even as these systems enabled most of a patient’s data to be gathered in one place and made those data readily accessible to clinicians caring for that patient. The redundancy of the notes, the burden of alerts, and the overflowing inbox has led to the “4000 keystroke a day” problem and has contributed to, and perhaps even accelerated, physician reports of symptoms of burnout. Even though the EMR may serve as an efficient administrative business and billing tool, and even as a powerful research warehouse for clinical data, most EMRs serve their front-line users quite poorly. The unanticipated consequences include the loss of important social rituals (between physicians and between physicians and nurses and other health care workers) around the chart rack and in the radiology suite, where all specialties converged to discuss patients.”
What This Computer Needs Is a Physician: Humanism and Artificial Intelligence Abraham Verghese, MD1; Nigam H. Shah, MBBS, PhD1; Robert A. Harrington, MD JAMA (in press) doi:10.1001/jama.2017.19198 - “The lessons learned with the EMR should serve as a guide as artificial intelligence and machine learning are developed to help process and creatively use the vast amounts of data being generated in the health care system. Outside of medicine, the use of artificial intelligence in predictive policing, bail decisions, and credit scoring has shown that artificial intelligence can actually exaggerate racial and other bias.”
What This Computer Needs Is a Physician: Humanism and Artificial Intelligence Abraham Verghese, MD1; Nigam H. Shah, MBBS, PhD1; Robert A. Harrington, MD JAMA (in press) doi:10.1001/jama.2017.19198 - “Similar concerns around artificial intelligence predictive models in health care have been discussed: clearly, in the 3-step process of selecting a dataset, creating an appropriate predictive model, and evaluating and refining the model, there is nothing more critical than the data. Bad data (such as from the EMR) can be amplified into worse models. For example, a model might classify patients with a history of asthma who present with pneumonia as having a lower risk of mortality than those with pneumonia alone, not registering the context that this is an artifact of clinicians admitting and treating such patients earlier and more aggressively. Since machine learning presents no human interface and cannot be interrogated, even if its predictions are extraordinarily accurate, some clinicians are likely to view the “black box” with suspicion.”
What This Computer Needs Is a Physician: Humanism and Artificial Intelligence Abraham Verghese, MD1; Nigam H. Shah, MBBS, PhD1; Robert A. Harrington, MD JAMA (in press) doi:10.1001/jama.2017.19198 - “In the care of the sick, there is a key function played by physicians, referred to by Tinsley Harrison as the “priestly function of the physician.” Human intelligence working with artificial intelligence—a well-informed, empathetic clinician armed with good predictive tools and unburdened from clerical drudgery—can bring physicians closer to fulfilling Peabody’s maxim that the secret of care is in “caring for the patient.”
What This Computer Needs Is a Physician: Humanism and Artificial Intelligence Abraham Verghese, MD1; Nigam H. Shah, MBBS, PhD1; Robert A. Harrington, MD JAMA (in press) doi:10.1001/jama.2017.19198 - “It is likely that machine learning applications will soon transform some sectors of health care in ways that may be valuable but may have unintended consequences. Use of ML-DSS could create problems in contemporary medicine and lead to misuse. The quality of any ML-DSS and subsequent regulatory decisions about its adoption should not be grounded only in performance metrics, but rather should be subject to proof of clinically important improvements in relevant outcomes compared with usual care, along with the satisfaction of patients and physicians.”
Unintended Consequences of Machine Learning in Medicine. Cabitza F, Rasoini R, Gensini GF JAMA. 2017 Aug 8;318(6):517-518