Integrating vision-language models into radiology requires a way to send two-dimensional image evidence without stripping away the viewing context that shapes interpretation. A technical development in European Radiology Experimental centres on state-based export inside the diagnostic viewer. The architecture keeps model interaction within a split DICOM viewer, with a diagnostic grid for image review and an integrated chat sidebar for staging evidence and submitting questions. Instead of relying on screenshots or external uploads, capture requests become explicit viewer-state descriptors that can be replayed by an offscreen renderer. Benchmarks under controlled conditions show reproducible, low-latency evidence generation for multislice and multiseries workflows.
State-Based Capture Replaces Screenshots
Traditional interaction between radiologists and vision-language models can sit outside the diagnostic viewer. Text-only prompting requires radiologists to translate imaging findings into free text without sharing visual evidence. Manual export and upload can preserve some visual content, but often reduces a volumetric, interactive examination to a small set of static images. Critical details can be lost, including slice location, imaging plane, window and level settings, slab configuration, camera orientation and visible overlays or annotations. The architecture addresses this bottleneck by treating volumetric context capture as a viewer workflow rather than a separate export task.
An exportable view is represented as a serialised viewer state. That state can encode the series, viewing mode, slice or plane intent, window and level settings, slab parameters, camera when needed and overlay policy. Viewing modes include two-dimensional slice navigation and multiplanar reconstruction in axial, coronal and sagittal orientations, with optional slab rendering. Interactive viewports can serialise their current state, while batch actions can generate sequences such as sampled slices or orthogonal plane sets. A separate offscreen viewer executes each queued state using shared in-memory imaging data, while the diagnostic viewer remains available for interactive interpretation. Each exported frame can be linked to a replayable state, supporting later reproduction and audit of the evidence sent to the model.
Integrated Staging Supports Model Interaction
The workspace combines a diagnostic grid for interactive image review with an assistant sidebar for navigation, evidence staging and chat. Capture can begin from an active viewport or from a series list. The sidebar exposes capture actions for current views, sampled slice stacks, orthogonal multiplanar reconstruction and annotated bookmarks. When capture creates multiple frames, the generated images are staged before transmission so users can review the evidence attached to a message. Placeholders appear immediately and are replaced by thumbnails as frames complete, allowing question drafting while export continues. The current implementation targets one operating ecosystem, while the underlying approach uses framework-agnostic state serialisation and offscreen rendering.
The capture pipeline renders to a graphics processing unit-backed target and exports a pixel buffer for optional overlay compositing and compression. Frames can be encoded in model-compatible formats, including JPEG or high-efficiency image container format, in the viewer’s display orientation. Provider-specific model wiring is isolated to an adapter layer that packages selected images and parses responses. Model responses can stream into the chat interface as text. Overlay handling remains policy controlled because overlays may carry clinical meaning but may also contain identifying text. Exported images exclude DICOM metadata, and deployments must ensure identifiers do not appear in pixels or filenames. Automated burned-in-text detection is not part of the current implementation, so prevention relies on overlay configuration and institutional policy.
Must Read: Imaging Biomarker Catalogue Aims to Improve Reuse
Benchmarks Show Reproducible Export
Benchmarking used an offscreen capture pipeline at a fixed output resolution under controlled conditions. The evaluation ran on a single device with the source series resident in shared memory. Test data were one axial magnetic resonance imaging series from the UPenn-GBM collection via The Cancer Imaging Archive. In the determinism preset, 10 two-dimensional viewer states were each captured 5 times with overlays disabled. Repeated captures from identical serialised states produced no final hash mismatches across the 50 exported frames, indicating pixel-identical output within the tested run. Reproducibility across heterogeneous deployment environments was not assessed.
A mixed performance preset captured two-dimensional and multiplanar reconstruction frames with overlays enabled. Capture times remained in the millisecond range across logged view types, with median values from 4.49 ms for axial reconstruction to 16.69 ms for sagittal reconstruction. Coronal and sagittal reconstruction took longer than axial reconstruction because the source series was acquired in the axial plane, making orthogonal reslicing computationally heavier. Encoding showed a clear size and latency trade-off. High-efficiency image container output was smaller than JPEG at the selected encoder setting, with median sizes of 27.9 kB and 79.4 kB respectively, but encoding took longer. Performance may differ with larger datasets, other modalities, cold-cache conditions or networked environments, and workflow effects require evaluation under realistic clinical reading conditions.
State-based offscreen rendering offers a practical route for connecting vision-language models to radiology viewing without reducing examinations to ad hoc screenshots or separate upload steps. By capturing explicit viewer context and staging evidence inside the diagnostic environment, the architecture supports repeatable, auditable and curated model interaction. The benchmark results show deterministic export and interactive-speed capture in the tested setting, while also identifying limits around deployment variability, memory use, protected health information controls and the need for workflow evaluation before routine clinical use.
Source: European Radiology Experimental
Image Credit: iStock
References:
Albera M, Colarieti A, Albera I et al. (2026) Technical architecture for integrating vision-language models with DICOM viewers. Eur Radiol Exp; 10, 103.
