Closing the Trust Gap in Medical Imaging AI

AI-enabled medical devices are moving rapidly into clinical care, but confidence in their outputs remains uneven among clinicians and patients. Of all AI-enabled medical devices cleared by US FDA, 78% were cleared after 2019, while radiology accounts for 77% of current FDA-cleared AI-enabled medical devices. A 2026 perspective published in the American Journal of Roentgenology sets out a framework for explainable artificial intelligence in medical imaging, focused on making AI outputs more transparent, interpretable and clinically actionable. The framework identifies three requirements for clinical usefulness: technical robustness, adaptation to end users and alignment between the explanation and the clinical task. It also assigns responsibilities to developers, vendors and healthcare institutions as AI tools move from development into practice.

Technical Robustness Comes First
Clinical use requires XAI methods that accurately identify meaningful imaging features, explain how those features influence predictions and respond logically when inputs or models change. Saliency maps often indicate regions of interest, but evidence in medical imaging shows unreliable localisation of true abnormalities. In pneumothorax localisation, several common saliency methods did not outperform dedicated segmentation models and most did not beat a simple baseline that repeatedly highlighted the same average reference-standard location. Correct AI predictions can also produce visual explanations that mark areas influencing the model rather than the clinical finding itself, creating a gap between technical meaning and clinician interpretation.

Must Read: Explainable AI Improves Breast MRI Cancer Detection

Robustness also depends on clear communication of feature importance. Gradient-based approaches, sufficient features, necessary features and game-theoretic methods can convey different meanings. A highlighted pneumonia opacity may be sufficient when it alone preserves a prediction, or necessary when its removal changes the prediction. Sanity checks add another layer by testing whether explanations remain stable or change when expected. Some saliency maps remain visually similar for trained and randomly initialised neural networks, and a sham AI aneurysm model with near-zero sensitivity produced saliency maps that did not worsen radiologist performance compared with a high-performance model. Such outputs risk being clinically uninformative or actively misleading if localisation and stability are not demonstrated.

Adaptation Must Reflect Clinical Users
XAI serves human decision-makers whose training, cognitive styles and workflow pressures vary widely. Medical imaging users range from subspecialist radiologists to general practitioners, physician assistants and nurses. Even among radiologists, individual responses to AI assistance differ, which limits the value of a universal explanation format. Some users may benefit from visual cues such as saliency maps, while others may prefer structured text that resembles conventional radiology communication. Poor alignment can create confusion or diagnostic error when explanations are too simple, too detailed, poorly timed or inconsistent with clinical expectations.

Adaptive XAI offers a route beyond static one-size-fits-all displays. Reinforcement learning from human feedback and direct preference optimisation can support personalisation of explanation style, complexity or timing, and related techniques have improved image segmentation and factual accuracy in radiology output generation. Customisation is not required for every use case. Straightforward constrained tasks, such as pneumoperitoneum detection or aneurysm measurement, may only need a bounding box or anatomic segmentation. More complex tasks and broader user groups create stronger reasons to assess personalisation. Dynamic explanations also require continuous monitoring and regulatory structures such as predetermined change control plans. Developers provide feedback architecture, vendors organise data and support preference training, and healthcare institutions lead local implementation, feedback, validation and monitoring. Reliability must be rechecked as outputs evolve and local user feedback shapes preference tuning.

Task Matching Shapes Clinical Value
Explanations must also match the clinical task and the type of AI output. Medical imaging XAI includes localisation-based methods, quantitative methods, semantic methods and exemplar-based methods. These formats answer different questions about AI output: where the abnormality lies, what is abnormal and why the model reaches a particular result. A disease classifier with categories such as pneumonia, pneumothorax or normal may benefit from a saliency map or another method that highlights features supporting a category. A regression model for paediatric bone age needs a different format because it does not centre on a single abnormality to localise.

The framework links explanation choice to downstream clinical action. Localisation fits tasks involving visual search, targeted biopsy or surgical planning, where bounding boxes or segmentations can identify relevant anatomy. Quantitative and semantic explanations fit threshold-based triage or staging, such as ventricular volume or aortic dilation. Exemplar-based or counterfactual explanations fit holistic classification and regression tasks that depend on multiple findings, such as bone age estimation. Clinical leaders responsible for procurement need to understand these distinctions when selecting configurations for local workflows. Complex formulations may combine several explanation types. Lung cancer TNM staging may require localisation of the primary lesion, suspicious mediastinal lymph nodes and potential osseous metastases, quantitative measurements for tumour and lymph-node size, and semantic information on features such as pleural invasion.

The framework places technical robustness at the start of trustworthy XAI deployment, with task matching and end-user adaptation built on that foundation. Explanations that look persuasive but fail sanity checks, ignore the clinical task or disregard local workflow needs can undermine safe use. Developers, vendors and healthcare institutions therefore carry distinct but connected responsibilities across design, selection, implementation and monitoring. For medical imaging, the practical direction is clear: XAI needs to move away from generic displays and towards explanations that are robust, clinically aligned and usable in real-world practice.

Source: American Journal of Roentgenology

Image Credit: iStock