LLMs Classify Pneumonia Patterns from Radiology Reports

Large language models can classify patterns of usual interstitial pneumonia from radiology reports with substantial agreement against expert consensus, but their performance depends strongly on how they are prompted. A retrospective study in the Journal of Imaging Informatics in Medicine evaluated a range of open-source models using high-resolution computed tomography reports from a single academic medical centre. The strongest configuration achieved about 82% accuracy across four diagnostic categories. Structured reasoning improved some models but reduced performance in models with built-in reasoning capability, while extended thinking did not provide an overall advantage. 

Prompting Strategies Produce Different Results 

Usual interstitial pneumonia classification requires several imaging features to be considered together, including the distribution of abnormalities, fibrotic changes and findings that may suggest another diagnosis. Standardised categories can support clinical decision-making and provide labels for research, but these categories are not always explicitly recorded in routine radiology reports. 

The evaluation included 270 reports from patients assessed for suspected interstitial lung disease. Two experienced thoracic radiologists independently classified the reports according to Fleischner Society criteria, with disagreements resolved through consensus. The resulting classifications served as the reference standard for the language models. 

Ten open-source models from three architecture families were evaluated using three prompting approaches. The simplest provided classification criteria alone. A second approach added a structured clinical reasoning sequence, while a third also included representative examples. Models with native reasoning capability were additionally assessed with extended thinking enabled. 

Must Read: Structured Prompts Divide Radiology LLM Performance  

Most reports did not explicitly mention usual interstitial pneumonia or idiopathic pulmonary fibrosis. Classification therefore generally required models to interpret the described imaging findings rather than simply identify disease terminology. This made the task more demanding than extracting an isolated feature or recognised term from a report. 

Model Architecture Shapes Prompt Effectiveness 

The strongest four-category result came from a Llama model using structured reasoning supported by examples. However, the same prompting strategy did not consistently benefit other model families. Models without built-in reasoning generally improved when given a structured decision process, while reasoning-native Qwen and Gemma models performed better when supplied with the classification criteria alone. 

Extended thinking also failed to improve the strongest performance achieved by the reasoning-capable models. Under the conditions tested, enabling additional internal reasoning produced results that were similar to or worse than those obtained in standard mode. Greater reasoning complexity therefore did not consistently translate into better classification. 

Model size was also an unreliable predictor of performance. Increasing the size of a Llama model produced a substantial improvement initially, but further expansion offered little additional benefit. Within the Qwen family, a smaller model outperformed the largest version evaluated. These results indicate that model generation, training approach and prompt design can be at least as important as parameter count for this type of clinical classification. 

Additional biomedical pretraining also affected performance. MedGemma performed better than its base model when given criteria alone, but its performance declined more when structured reasoning was added. The optimal prompt therefore varied not only between model families but also between related models with different training histories. 

Errors Highlight Limits and Potential Uses 

The most frequent errors involved distinguishing indeterminate cases from reports suggesting an alternative diagnosis. These categories were also the main source of disagreement between the two expert radiologists, indicating that the models encountered difficulty in areas where human interpretation was also less consistent. Performance was stronger for more distinctive categories and weaker for intermediate classifications. 

Accuracy improved when the classification task was simplified. Combining categories into a binary decision produced accuracy of about 95%, while a three-category approach also performed better than the full four-category task. These findings support potential use in triage and decision support, particularly for identifying reports that may warrant specialist review. 

Automated extraction could also support research by converting information in free-text radiology reports into standardised labels. Such labels could assist with building larger datasets for model development, disease progression studies and retrospective analyses. 

Several limitations remain. The evaluation was conducted at a single tertiary referral centre, where disease prevalence, reporting style and terminology may differ from other institutions. The reference classifications were based on report text rather than independent review of the underlying images. The dataset was also unevenly distributed across diagnostic categories. External validation is therefore needed to determine whether the observed performance and differences between model–prompt combinations extend to other clinical settings. 

Open-source large language models can classify usual interstitial pneumonia patterns from radiology reports with substantial agreement against expert consensus, but increasing prompt complexity does not consistently improve performance. Structured reasoning benefited models without native reasoning capability while reducing performance in reasoning-oriented models, and extended thinking offered no clear advantage. Larger models were not consistently more accurate. The findings support testing each model together with its intended prompting strategy rather than assuming that greater scale or more elaborate reasoning will produce better clinical classification. 

Source: Journal of Digital Imaging Informatics in Medicine

Image Credit: iStock