Chest radiography is a widely used method for diagnosing or ruling out thoracic diseases. This prevalence has led to the development of numerous artificial intelligence (AI) systems designed to detect specific abnormalities in chest X-rays, such as pneumonia, lung nodules, tuberculosis, or pneumothorax. Some AI products aim to identify a broad range of thoracic diseases and provide comprehensive reports to radiologists. AI systems can potentially improve the quality of chest radiograph interpretations and reduce radiologists' reporting time. Studies have focused on using AI to prioritise urgent cases, ensuring they are addressed first, and to reliably exclude normal radiographs from further evaluation. This exclusion could alleviate the workload on radiologists, which is especially important in the context of increasing demand, a shortage of physicians, and budgetary constraints in healthcare. Research suggests that AI can identify 28-36% of normal chest radiographs with high confidence, potentially reducing the radiologists' workload by 8-17%. However, the generalisability and performance of AI solutions at different operating points need further investigation. A recent study published in European Radiology aims to assess a commercially available AI system's ability to identify normal chest radiographs in a clinical series from two hospitals. It evaluates various operating points to estimate the proportion of normal radiographs that could be safely excluded from the clinical workflow.
Multicentre Study Overview and Methodology
The study involved a retrospective analysis of all consecutive chest radiographs and their reports from October 1 to October 14, 2016, at two hospitals in the Netherlands: Radboudumc (an academic hospital) and Jeroen Bosch Hospital (a large community hospital). It included radiographs from inpatients, outpatients, and emergency cases, but only the first exam during the study period for each patient to avoid bias. Exclusions were made for follow-up exams, bedside radiographs, incomplete images, imported radiographs, and pediatric radiographs. Three radiologists classified the findings from the radiographs into categories: normal, clinically irrelevant, clinically relevant, urgent, and critical. A commercial AI system analysed all radiographs, scoring ten chest abnormalities on a scale from 0 to 100. The AI's performance in detecting normal radiographs was measured using the area under the ROC curve (AUC). Sensitivity at both the default and a conservative operating point, negative predictive value (NPV) for urgent and critical findings, and potential workload reduction were calculated. Out of 2603 radiographs from 2141 patients, 1670 radiographs were included in the final analysis after exclusions.
Evaluation of AI Performance
Three chest radiologists categorised the findings into various classifications, including normal, clinically irrelevant, clinically relevant, urgent, and critical. The AI system processed all radiographs, scoring ten chest abnormalities on a 0-100 confidence scale. Categories included 479 normal, 332 clinically irrelevant, 339 clinically relevant, 501 urgent, and 19 critical findings. The AI system achieved an AUC of 0.92. Sensitivity for normal radiographs was 92% at default and 53% at the conservative operating point. At the conservative operating point, NPV was 98% for urgent and critical findings, and could result in a 15% workload reduction.
Exploring Operating Points for AI System
The study explored different operating points for the AI system, aiming to strike a balance between sensitivity to abnormal findings and the ability to accurately identify normal radiographs. The default operating point identified 92% of normal chest radiographs, potentially leading to a workload reduction of 36%. However, this operating point also resulted in more radiographs being classified as normal, potentially missing abnormalities. A more conservative operating point maintained sensitivity for urgent and critical findings while identifying a higher percentage (53%) of normal chest radiographs. This operating point aimed to minimise the risk of missing significant abnormalities but still resulted in some missed urgent findings. Comparison with previous studies revealed similar performance despite differences in population demographics. However, differences in AI system design and study methodology could impact results. Limitations of the study included reliance on expert opinion for the reference standard, exclusion of certain types of radiographs, lack of external validation for the AI system, and the current inability of the AI to stand alone in reporting.
Findings demonstrated the potential of AI to identify normal chest radiographs and reduce radiologists' workload effectively. Further research and validation on larger datasets, as well as prospective studies, are necessary to assess the generalisability and reliability of AI systems for chest radiography.
Source: European Radiology
Image Credit: iStock
