Clinical decision-making is increasingly supported by artificial intelligence systems that generate recommendations across diagnosis, treatment and care planning. Although these tools are often positioned as neutral aids, their outputs can reflect value choices that shape what is recommended and what is deprioritised. Those choices may relate to financial incentives, institutional priorities or differing ethical commitments rather than purely clinical reasoning. When value orientations remain implicit, recommendations can appear objective while subtly steering decisions in particular directions. A proposed governance approach focuses on making these underlying values visible to clinicians and patients, so that recommendations can be interpreted with clearer context and used more safely in everyday care.
Must Read:Prompting and Thresholds Shift GPT TBI Mortality Triage
Hidden Incentives in Clinical Recommendations
Many clinical decisions involve trade-offs between competing priorities, and AI systems can resolve these trade-offs differently depending on how they are designed or deployed. A simplified scenario involves a 60-year-old man with lower back pain. One system shaped by a fee-for-service hospital recommends magnetic resonance imaging, presented as a way to rule out rare but serious conditions while also generating income. Another system tuned by an insurer advises watchful waiting, suggesting physical therapy could speed recovery. Both recommendations may be clinically defensible, yet each embodies a distinct value framework, one leaning towards revenue generation and the other towards cost containment. The practical challenge is that the governing values can be hard to detect at the point of use.
A second scenario illustrates value conflict in a higher-stakes setting. A young woman is hospitalised for cardiac complications of anorexia, refuses oral supplements and has not improved food intake over 24 hours, while her weight remains stable since admission. The decision is whether to place a feeding tube against her will. Clinicians were split, with 47% emphasising the duty to protect autonomy and 53% emphasising the duty to prevent imminent harm. When 14 large language models (LLMs) were asked the same question, responses were also split, ranging from forced intervention to deferring to autonomy or watchful waiting, with some invoking liability concerns. These divergences underscore that value alignment cannot be assumed from technical descriptions alone.
Value conflicts also appear in routine operational choices, including prioritising appointments and setting thresholds for tests and referrals. Even when stakes seem modest, decisions can influence downstream care patterns, costs and patient experience. Existing legal, ethical and clinical frameworks may define principles such as autonomy, beneficence and fairness, yet they do not reveal which values are embedded in specific AI systems and how those values are applied in recommendations.
Beyond More Data: Building Value-Sensitive Benchmarks
One response to concerns about clinical AI is to train models on larger electronic health record (EHR) datasets, assuming that more data will improve decision support. The proposed approach challenges that assumption by noting that documented practice can reflect institutional incentives, workflow pressures and imperfect knowledge. EHRs may also capture uncertainty or incomplete understanding of patient preferences. Training AI systems on these records risks reproducing misaligned patterns rather than correcting them, particularly when the drivers are organisational or financial rather than patient-centred.
A complementary approach is to test systems against clinical scenarios designed to place values in tension. Examples include situations where a patient prefers a treatment with a low likelihood of success while expected effectiveness points elsewhere, or where individual preferences intersect with broader public health considerations. The objective is not to force a single correct answer, but to reveal how a system behaves when decisions hinge on contested trade-offs rather than straightforward optimisation.
To make such testing credible, a large library of clinical vignettes would be developed and validated with diverse stakeholders. Inputs would include clinicians across specialties, patients from different backgrounds, healthcare leaders, insurers, public health experts, ethicists and policy makers. The resulting distributions would provide reference points for how different groups resolve value conflicts. AI systems could then be evaluated against these benchmarks, with results summarised by stakeholder category. A model might align closely with majority choices among insurers while diverging from majority choices among patients. Making those divergences visible supports better selection, oversight and context-aware use, while recognising that individual patient preferences may differ from any group majority.
Values In the Model and Practical Oversight
A Values In the Model (VIM) framework has been proposed as a transparent labelling system that accompanies AI tools deployed in clinical care. The intention is to extend typical model documentation by adding an alignment component focused on value-laden decisions. Rather than describing only technical performance, a VIM label would document how a system behaves when tested against the value-sensitive vignette library, making the system’s practical orientation legible to those using it.
A VIM label could indicate whether a system tends towards aggressive intervention or conservative management, prioritises autonomy over harm prevention or leans towards resource stewardship. It could also disclose whether a system was purposefully aligned, including through reinforcement learning, to recommend more interventions, or whether a system prompted for capitation is prone to cost cutting. Some components would come from developers, describing whether a system was designed to offer recommendations from an explicit viewpoint, such as acting as a provider aligned to maximise fee-for-service. Other components would come from benchmark testing, summarising the preferences the system exhibits under standardised value conflicts.
The framework also distinguishes between allocative decisions linked to business or payment models and decisions made on behalf of individual patients. Some recommendations relate to organisational incentives, including preferential use of specific services. Others concern individual trade-offs, weighing patient preferences alongside factors such as age, disease severity, reversibility of pathology and quality of life-years remaining. The label is intended to address both categories while clarifying when recommendations incorporate considerations about sparing resources or moderating risks for other patients.
Ongoing oversight would also require transparency mechanisms that capture how systems behave under real-world conditions. The MEDLOG system has been proposed as a way to log actions of AI systems with full clinical context, creating a record of model behaviour. Benchmark libraries would need periodic refreshing because widely used benchmarks are likely to be incorporated into training data for frontier models over time, which could weaken their value for independent evaluation.
Clinical AI does not remove value judgements from healthcare but can scale them rapidly when embedded in workflows. The central risk is not only technical error but the quiet propagation of hidden priorities that may reflect hospitals, insurers or developers rather than patients and clinicians. A Values In the Model approach aims to shift those commitments from implicit to explicit, using scenario-based benchmarks and transparent labels to show how systems resolve ethical and incentive trade-offs. Paired with robust logging of model behaviour, this supports more accountable deployment and more informed tool selection. For healthcare decision-makers, visible value alignment clarifies what a system tends to prioritise and where clinical judgement and patient preference must remain decisive.
Source: New England Journal of Medicine
Image Credit: iStock
