Benchmarking Dataset Tests LLM Symptom Detection

A benchmarking dataset and reproducible pipeline have been developed to assess how accurately large language models detect symptoms in clinical conversations. Published in JAMIA Open, a proof-of-concept analysis used publicly available synthetic patient–physician interviews, avoiding protected health information while enabling direct model comparisons. The dataset covers general symptom mentions and 16 individual symptoms, with annotations independently reviewed and disagreements adjudicated. Nine open- and closed-source models were tested using the same basic prompts and a consistent output structure. Performance differed across models and symptoms, while the resulting resources are intended to support further evaluation, optimisation and comparison before testing with real patient data. 

Must Read: Patient Bias Challenges LLM Health Consultation 

Building a Reliable Symptom Benchmark 

The dataset was created from 264 simulated patient–physician conversations in an objective structured clinical examination format. The transcripts were divided into 3,085 non-overlapping excerpts of fewer than 200 tokens to limit prompt size and reduce potential information loss. Of these excerpts, 2,087 contained at least one symptom. Simulated conversations were selected to minimise privacy concerns associated with the use of large language models, although no demographic information was available for assessing differences between patient groups. 

Annotations operated at two levels. The first recorded whether any symptom was present, while the second identified individual symptoms. An initial codebook covered 12 symptoms used in patient-reported outcomes at Dana-Farber Cancer Institute. Testing and review added headache, concentration problems, cough and fever, producing a final set of 16 symptoms plus an “other” category for additional symptom mentions. 

Each excerpt was independently reviewed by two of three coders. A third coder adjudicated disagreements, with uncertain cases discussed to reach consensus. Agreement was high for overall symptom presence and close to perfect for specific symptoms. Pain, cough and shortness of breath were the most frequent symptoms in the dataset. Other categories occurred much less often, including anxiety, concentration problems and trouble drinking fluids. The uneven distribution affected how confidently performance could be assessed for rarer symptoms, where sample sizes were limited. 

Testing Models with a Common Pipeline 

The evaluation pipeline was demonstrated with 9 models from OpenAI, Meta, DeepSeek and Moonshot AI. They included larger and smaller systems, as well as open- and closed-source options. The benchmarking dataset itself is not restricted to these models and can be applied to other systems and prompts. 

Two zero-shot prompts were used. One asked models to identify whether an excerpt contained any symptom, while the other asked them to identify specific symptoms. The same prompts were applied across all models, allowing rapid comparison without extensive prompt optimisation. This design means the reported performance reflects basic prompting rather than the strongest result potentially achievable through fine-tuning or more detailed instructions. 

A single structured output format was enforced to support automatic comparison with the annotated dataset. Models could retry when an output did not match the required structure, although all tested systems returned correctly formatted results. Performance was assessed through precision, recall, accuracy and F1 score, with additional analysis of false positives and false negatives. 

For general symptom detection, GPT-4.1 recorded the strongest overall result among the 9 models, with an F1 score of 0.88. Other models produced different balances between precision and recall. Llama-3.1, for example, captured nearly all symptom-containing excerpts but generated more false positive detections, while Llama-3.2 showed lower recall and overall performance. These differences demonstrate why several metrics are needed when comparing systems intended to identify clinical information. 

Performance Varies by Symptom and Error Type 

Results for individual symptoms varied more than results for general symptom detection. Across pain, cough and shortness of breath, F1 scores ranged from 0.66 to 0.88. Cough generally produced the strongest results, while pain and shortness of breath showed lower recall for several models. Performance was typically reduced for rare symptoms such as concentration problems and trouble drinking fluids, although small numbers limited evaluation. 

False negatives were more common than false positives for the three most frequent symptoms across nearly all models. Models therefore more often failed to capture a symptom than incorrectly assigned one. The main exception was Llama-3.1 when detecting cough. Manual review linked many errors to ambiguous or borderline language that the simple prompt did not address, including references to colds, coughing up blood or bringing up sputum. 

Prompt design and decision rules may affect symptom extraction. The pipeline was built primarily to provide a reproducible comparison method rather than optimise an individual model. Future work will examine prompt engineering and fine-tuning, adaptation to real clinical data and methods for combining excerpt-level results into conversation-level assessments. 

Further evaluation requires qualitative, mixed-method and task-based approaches, including clinician feedback. Quantitative scores cannot establish clinical utility or patient impact alone. The simulated conversations were largely derived from respiratory cases, while the absence of demographic data prevented assessment of bias and performance differences. Dividing transcripts into short excerpts may also have influenced results and requires testing across model context windows. 

The publicly available dataset and reproducible pipeline provide a structured way to compare large language models on symptom detection in clinical conversations. Strong agreement between human annotators supports the benchmark, while testing shows meaningful variation between models, symptoms and error patterns. General symptom detection performed better than several granular extraction tasks, with missed symptoms more common than incorrect detections. The use of simulated conversations supports preliminary evaluation without protected health information, but broader clinical data, demographic analysis, prompt optimisation and clinician-led assessment remain necessary before these methods can be judged for real-world clinical use. 

Source: JAMIA Open 

Image Credit: iStock