Large language model (LLM) embeddings can encode longitudinal electronic health record (EHR) data for clinical prediction without requiring access to private medical training datasets. A comparison in npj Digital Medicine assesses general-purpose LLM embeddings against specialised EHR foundation models and conventional baselines. Structured records are converted into plain-text descriptions of medical codes, then embedded for use in downstream prediction models. Across benchmark and external validation settings, the approach matches or modestly exceeds a domain-specific model on some tasks, while requiring substantially more computation. The central trade-off is between portability across coding systems and the efficiency of models trained directly on EHR data.
Text-Based EHR Encoding Supports Prediction
Electronic health records contain information on visits, diagnoses, procedures, medications, laboratory values and other clinical events. Their structure can make prediction modelling difficult. Records may include irregular sampling intervals, missing entries, variable-length sequences, noisy information and hierarchical medical concepts. Foundation models trained on EHR data can address some of these barriers, but code-based models are limited by site-specific coding practices and fixed vocabularies learned during pretraining.
Must Read: Semantic Models Improve EHR Risk Transfer
The evaluated approach works around those limits by replacing structured medical codes with natural-language descriptions. Each patient record is then serialised as plain text. The default format is a newline-separated list of event descriptions, using the most recent occurrence of each code and adding values and units when available. These text records are embedded by LLMs and passed to logistic regression classifiers.
The comparison focuses on representation quality rather than complex downstream modelling. Qwen3-Embedding-8B receives task-specific instructions and processes records within a defined token limit. It is compared with CLMBR-T-Base, a specialised EHR foundation model trained on de-identified Stanford Medicine records, as well as encoder-only biomedical language models and count-based models. The aim is to test whether general-purpose language models can represent clinical histories without institution-specific pretraining or a fixed medical vocabulary.
Benchmarks Show Similar Predictive Strength
Primary analyses used EHRSHOT, a benchmark based on records from adult patients treated at Stanford Health Care and Lucile Packard Children’s Hospital. The benchmark covers clinical prediction tasks across operational outcomes, laboratory-test prediction, new diagnosis assignment and chest X-ray findings. External validation used UK Biobank data, allowing performance to be assessed across a different healthcare dataset and coding environment.
On EHRSHOT, Qwen3-Embedding-8B matched CLMBR-T-Base when all available training and validation examples were used. Qwen3-Embedding-8B performed slightly better in laboratory prediction, new diagnosis assignment and chest X-ray prediction. Task-level testing showed significant gains for thrombocytopenia, hyponatraemia and hyperkalaemia. CLMBR-T-Base performed significantly better on anaemia and hypoglycaemia. Most other task differences were small within statistical uncertainty.
Combining Qwen3-Embedding-8B and CLMBR-T-Base embeddings improved overall performance, suggesting that the two representations capture complementary information. A strengthened count-based model also remained competitive when all data were available. This shows that conventional baselines can still perform well in high-data settings. In few-shot settings, Qwen3-Embedding-8B maintained slightly higher performance than CLMBR-T-Base for new diagnoses and chest X-ray tasks across most shot settings.
Portability Comes with Computational Cost
External validation in UK Biobank showed a modest overall advantage for Qwen3-Embedding-8B over CLMBR-T-Base. Significant improvements appeared on a subset of tasks, with no significant differences for the remaining tasks. The result was partly linked to vocabulary coverage. CLMBR-T-Base uses a fixed vocabulary, while the processed UK Biobank cohort contained many medical codes outside that vocabulary.
Only a minority of UK Biobank codes could be mapped to the CLMBR-T-Base vocabulary, and these accounted for a limited share of medical events. When Qwen3-Embedding-8B was restricted to the same mappable codes, performance decreased. This placed it between the full Qwen3-Embedding-8B model and CLMBR-T-Base. The sensitivity analysis indicates that both broader vocabulary coverage and model generalisation contributed to the external validation gains.
The flexibility of the LLM-based approach comes with substantially higher computational cost. CLMBR-T-Base encoded the EHRSHOT benchmark in minutes, while Qwen3-Embedding-8B required many hours on the same computing setup. Input design also mattered. Keeping the most recent occurrence of each code performed better than keeping the first occurrence. Date and time information did not improve results, and handcrafted Markdown did not improve overall performance compared with the simple list format.
General-purpose LLM embeddings offer a flexible way to encode structured EHRs by translating medical codes into natural-language descriptions. They can match a specialised EHR foundation model on an in-domain benchmark and show modest gains in an external cohort where coding coverage limits fixed-vocabulary models. The approach is most relevant as an evaluation of representation quality, portability and data independence. Its main limitation is computational cost, which remains far higher than for specialised EHR models. The key finding is a trade-off between efficiency and transfer across coding systems.
Source: npj Digital Medicine
Image Credit: iStock
References:
Hegselmann S, von Arnim G, Rheude T et al. (2026) Large language models are powerful electronic health record encoders. npj Digit. Med.
