Clinical Implementation of AI for Pulmonary Embolism Detection in over 30 000 CT Pulmonary Angiography Examinations
Shlomit Goldberg-Stein, Amir Gandomi, Matthew A Barish, Stuart L Cohen, David Hirschorn, Rakesh Shah, Ritesh D Patel, Elizabeth Y Rula, Jason Naidich, Pina C Sanelli
Purpose To quantify postimplementation concordance between a U.S. Food and Drug Administration-cleared artificial intelligence (AI) tool and AI-informed radiologists for pulmonary embolism (PE) detection at CT pulmonary angiography, with real-time adjudication of discordances. Materials and Methods A commercial PE AI tool was retrospectively implemented in the clinic across an integrated network (August 9, 2021-February 20, 2023). Adult CT pulmonary angiographic acquisitions underwent real-time AI analysis and radiologist interpretation. Radiologist-AI disagreements triggered adjudication by thoracic radiologists via the AI quality oversight process. Adjudicator diagnosis served as the reference standard for discordant cases. Concordance was measured and diagnostic performance of radiologists and AI was compared using adjudication for discordant cases. Results A total of 32 501 CT pulmonary angiographic acquisitions obtained from 29 492 patients (mean age, 62.4 years � 18.6 [SD], 17 424 female patients) were evaluated. PE positivity was 9.93% (3226 of 32 501). Overall concordance was 97.79% (95% CI: 97.62, 97.94) and was higher for AI-negative than for AI-positive examinations (98.18% vs 93.75%; P < .001). Expert adjudication favored the radiologist in 88.73% of discordances. The rate of unique diagnosis by the interpreting radiologist (483 of 3226 [14.97%]) was approximately 19 times that of the AI tool alone (26 of 3226 [0.81%]). Concordance varied by PE features: acute versus chronic (87.34% vs 60.12%; P < .001) and location (central, 95.79%; lobar and/or segmental, 83.81%; subsegmental, 58.62%; all P < .001). Conclusion In large-scale deployment, AI showed high concordance with radiologists and made meaningful contributions in discordant reviews while expert oversight confirmed complementary roles and highlighted scenarios of radiologist-AI divergence.