Advancements in large language models (LLMs) have unlocked significant potential across numerous domains, including medical applications. Transforming unstructured radiology reports into structured data is crucial for retrospective analysis, quality management and artificial intelligence model training. However, achieving this efficiently requires models that can handle complex language with nuanced expressions. A recent review published by Radiology compares the performance of OpenAI’s GPT-4 with leading open-source LLMs in extracting findings from chest radiograph reports, highlighting their respective advantages and limitations.
Commercial LLMs vs. Open-Source Models
Commercial LLMs such as OpenAI’s GPT-4 are well-regarded for effectively managing free-text data. They have been trained on vast datasets and fine-tuned to handle various applications, including radiology report analysis. However, these proprietary models come with several limitations. The primary drawbacks include high costs associated with API access, privacy risks due to external server processing and dependencies on closed-source updates that are outside user control.
In contrast, open-source models like Meta’s Llama 2, Mistral by Mistral AI and Alibaba’s Qwen have been designed to offer flexible, scalable alternatives. These models eliminate the need for external servers, allowing users to deploy them locally, thus mitigating privacy concerns. Furthermore, open-source models enable users to maintain consistency and reproducibility, as they are not affected by remote updates. While smaller in size and perhaps less sophisticated in some aspects than GPT-4, these models have shown considerable promise in various applications, making them viable contenders in medical contexts such as radiology report classification.
Performance on Radiograph Reports
The comparative study utilised two independent datasets to evaluate the performance of these LLMs. The first dataset, the ImaGenome dataset, contained 500 manually annotated chest radiograph reports, while the second dataset included 540 reports from Massachusetts General Hospital. The LLMs tested included GPT-3.5 Turbo, GPT-4 and a range of open-source models, namely Llama 2–70B, Mistral-7B and Qwen1.5–72B, alongside the established models CheXbert and CheXpert-labeler.
The study employed two essential techniques: zero-shot and few-shot prompting. Zero-shot prompting involves providing the model only with instructions, whereas few-shot prompting offers a small set of examples alongside instructions. The findings indicated that GPT-4 performed slightly better in zero-shot classification tasks, achieving high accuracy in identifying key findings in radiograph reports. However, when few-shot prompting was utilised, open-source models such as Llama 2–70B came remarkably close to matching GPT-4’s performance. This suggests that with adequate examples, open-source models can close the gap in accuracy. On the institutional dataset, an ensemble model combining the predictions of three open-source models (Llama 2–70B, Mixtral-8 × 7B and Qwen1.5–72B) closely matched the performance of GPT-4 in both zero-shot and few-shot scenarios. This emphasises the potential of open-source models to achieve comparable results in specific contexts.
Key Benefits of Open-Source LLMs
Open-source LLMs offer several notable advantages that extend beyond just accuracy. Firstly, deploying these models locally ensures greater control over privacy and data security, which is particularly crucial in medical applications dealing with sensitive patient information. This avoids the need to transmit data to external servers, thereby reducing the risk of data breaches.
Secondly, open-source models eliminate the financial burden associated with proprietary models. Commercial models like GPT-4 often charge on a per-token basis, which can become prohibitively expensive when processing thousands of radiograph reports. In contrast, if backed by sufficient hardware, open-source solutions present a cost-effective alternative, making them ideal for large-scale analyses, such as those needed in clinical research or public health studies.
Finally, the consistency and reproducibility offered by the local deployment of open-source models are invaluable. Updates to proprietary LLMs may alter model behaviours unpredictably, complicating the replication of previous analyses. In contrast, open-source models remain under user control, ensuring stable and consistent performance over time. This stability makes them well-suited for long-term clinical studies or projects where repeatability is paramount.
The study comparing OpenAI’s GPT-4 with leading open-source LLMs reveals valuable insights into the capabilities of these models in labelling chest radiograph reports. While GPT-4 demonstrated slightly superior zero-shot classification performance, open-source models such as Llama 2–70B, Qwen1.5–72B and an ensemble approach showed comparable results with the addition of few-shot prompting. The benefits of using open-source models, including cost-effectiveness, privacy preservation and consistency, make them an appealing alternative for medical text analysis. In the future, open-source LLMs will likely offer even greater competition to commercial models, driving innovation and accessibility in medical research and practice.
Source: Radiology
Image Credit: iStock
