Generative AI for Comprehensive Medication Management: A Call for Framework

Agentic artificial intelligence (AI) requires domain-specific benchmarking and evaluation. This piece discusses relevant features for how to evaluate the performance of AI as it conducts medication-related decision-making.

Introduction 

Comprehensive medication management (CMM) poses a specialised and unique domain for the application of generative artificial intelligence (AI). CMM is defined as “a patient-centred approach to optimise medication use and improve patient health outcomes…that ensures patients’ medications are individually assessed.” As such, CMM is a cognitive service involving domain-specific, expert-level clinical reasoning with the goal to individualise medication therapy in a way that maximises benefit and minimises harm (Health OMfb). Multiple interprofessional societies have endorsed recommendations affirming the importance of CMM (Kleinpell et al. 2021; Lat et al. 2020; Clemmons et al. 2018; Dunn et al. 2015; Sikora et al. 2025).

Despite the impressive feats of health AI including high performance on diagnostic reasoning tasks (Brodeur et al. 2026), AI has also demonstrated notable reasoning gaps, including Apple’s seminal paper ‘The Illusion of Thinking’ (Shojaee et al. 2026). In the space of medication decision-making, large language models (LLMs) have shown striking vulnerability to interpreting medication names, differentiating between medications and non-medication entities [including failing to see the difference between drugs and the popular Pokémon characters] (Sikora et al. 2026; Henry et al. 2026), and completing tasks at which rules-based software excel [e.g., drug-drug interaction identification, drug-dose matching] (Blotske et al. 2026; Blotske et al. 2025; Zhao et al. 2025; Chase et al. 2025; FDA 2025). However, even small errors (a period in the wrong place, an extra zero, missing two interacting agents) can lead to harmful and even fatal medication events.

The purpose of this commentary is to provide a review of studies that have aimed to probe LLM capabilities and failure modes in the CMM space and propose frameworks for high-quality evaluation. This work is in line with recent guidance from the United States (U.S.) Food and Drug Administration draft guidance regarding “context of use,” which supports defining an AI model’s context of use and developing specific evaluation around that context (FDA 2025).

The Potential for Generative AI and CMM 

Preventing medication-related harm remains a persistent challenge in modern healthcare, and the high rates of medication errors reflect not just individual human error but also the limitations of the systems clinicians rely on to support complex medication decisions (Rothschild et al. 2005). Clinical decision support systems (CDSS) are used to assist healthcare professionals with clinical decision-making and are now embedded across many phases of medication use (Kane-Gill et al. 2017). In selected domains, these systems have reduced error rates and supported safer prescribing when used alongside clinician judgement (Puxty et al. 2025; Prgomet et al. 2017). However, traditional rules-based CDSS have notable limitations when applied to the full scope of CMM. These systems are often difficult to customise to specific practice settings, patient populations, and local workflows. They may also fail to account for nuanced patient context, evolving clinical priorities, and competing risks across multiple conditions and medications. As a result, clinicians are frequently presented with alerts that are technically correct but clinically irrelevant or poorly prioritised, contributing to alert fatigue, high override rates, and clinician burnout, with potential downstream effects on patient safety (Wong et al. 2018; Nanji et al. 2018; Wong et al. 2023; Newton et al. 2026). Although targeted optimisation can reduce low-value alerts, such efforts are time- and resource-intensive, difficult to scale across practice settings and institutions, and unable to fully address all limitations of rules-based systems (McCoy et al. 2022; Bakker et al. 2024).

Challenges with rules-based CDSS have prompted interest in generative AI as a more flexible form of clinical decision support that may better support CMM activities. Generative AI tools are increasingly being incorporated into electronic health records (EHRs) for tasks such as clinical documentation and chart, with the potential to produce efficiency gains and reduce administrative burden (Silberlust et al. 2025; Olson et al. 2025; Everson et al. 2025). However, in CMM, their potential extends beyond administrative efficiency. Generative AI could help synthesise structured and unstructured patient data, support medication reconciliation, identify possible drug therapy problems, assess drug-drug and drug-disease risks, interpret patient data, evaluate and recommend therapy in relation to guidelines and patient characteristics, draft monitoring and follow-up plans, and help clinicians prioritise medication-related concerns, all while integrating more information than a clinician is capable of processing in time-constrained workflows.

Notably, modern generative AI models still demonstrate critical gaps in clinical reasoning and practice-based performance as well as poor uncertainty awareness (Gong et al. 2025; Rao et al. 2026). CMM differs substantially from lower-risk administrative applications because errors in medication-related outputs may directly influence therapeutic decisions and contribute to inappropriate therapy, missed safety concerns, preventable adverse drug events, and patient harm (FDA 2025). Accordingly, the most likely role for generative AI in CMM is as a human-supervised support tool rather than an autonomous medication management system. In this human-in-the-loop model, clinicians remain responsible for verifying inputs, interpreting outputs, contextualising recommendations, and making final care decisions. The near-term value of these systems is therefore likely to come from helping clinicians make more informed, timely, and personalised decisions rather than replacing clinician judgement and accountability. The central question, then, is not simply whether a model is more accurate than a clinician, but whether a clinician using the model consistently makes safer, higher-quality medication decisions than a clinician working without it (Goh et al. 2025).

Why CMM is a Unique Domain for Generative AI 

Despite the enticing future potential for generative AI in healthcare, CMM poses unique challenges across the AI lifecycle. Unlike more discrete clinical tasks, CMM requires dynamic, patient-specific, medication-level reasoning under conditions of uncertainty and high consequence for error. Patients receiving CMM may have multiple interacting disease processes, competing therapeutic priorities, and rapidly changing physiology and pharmacokinetic and pharmacodynamic determinants, requiring generative AI systems to reason over a moving target. Medication regimens themselves are often large, complex, and dynamic (Cullen et al. 1997). Safe decision-making requires attention to not just the medication itself but other critical features that describe medication exposure, including dose, route, formulation, frequency, timing, and duration. Complex regimens require reasoning across numerous therapies while accounting for drug-drug and drug-disease interactions as well as cumulative risks such as bleeding, QT prolongation, or nephrotoxicity. These characteristics create significant challenges for training and evaluation. High-quality datasets capturing the full complexity of medication decision-making remain limited, and available data sources often incompletely represent features critical to medication-related decision-making (Sikora et al. 2024). Real-world data, literature, and drug information resources may also be incomplete, outdated, contradictory, erroneous, or poorly applicable to the patient at hand, and the massive amount of healthcare data generated by each patient represents a challenging noise-to-signal ratio for systems to manage (Manor-Shulman et al. 2008).

CMM is further complicated by the absence of a single correct answer for many medication-related tasks. Multiple plans may be clinically acceptable depending on context, and this paradigm is further complicated by heterogeneous practices and the role of provider preference or local practice patterns in decision-making (Sutherland and Levesque 2020; Sirovich et al. 2014). Generative AI systems must distinguish unsafe choices from acceptable alternatives, explain rationale and trade-offs, and avoid presenting preference-driven recommendations as universally correct. They must also balance patient-level and population-level consequences, since reasonable recommendations for individual patients may contribute to population-level concerns such as antimicrobial overuse, increased costs, worsening polypharmacy, or reinforcement of inequitable prescribing patterns when scaled across health systems (Haug and Drazen 2023; Seyeed-Kalantari et al. 2021).

These challenges are compounded by known generative AI failure modes, including hallucination, overconfidence, poor uncertainty calibration, and sycophancy. Fluent but incorrect rationales may be difficult to detect and may influence high-stakes decisions. Because CMM is interprofessional and workflow-dependent, tools must provide transparent, auditable, prioritised outputs at the appropriate level of detail for the right user and context. Taken together, these features make CMM a uniquely challenging domain for generative AI.

The Gap: Not Knowing How to Grade or Characterise Generative AI for CMM 

While the FDA has recently put forth a draft guidance on using AI in certain medication-related applications, nothing has been put forward regarding generative AI that makes medication recommendations and very little has been put forth in the space of AI interacting in the medication domain. This is particularly salient as the state of Utah sanctioned a pilot programme allowing for AI-based medication refills after a relatively short validation period and a majority of hospitals are incorporating generative AI into their health systems (Gerke et al. 2026). While this document represents an important step forward for regulation, its narrow focus, draft status, and lack of teeth with regard to external validation, bias assessment, context drift monitoring, and requirement for real-world evidence makes it just the beginning (Blotske and Murray 2026). Former FDA Commissioner Robert Califf excellently summarised the problem faced: “I have looked far and wide, I do not believe there’s a single health system in the United States that’s capable of validating an AI algorithm that’s put into place in a clinical care system” (Angus et al. 2025).

Examples of Problems with Generative AI and CMM 

To illustrate the issues of generative AI and CMM, a few use-cases are presented. While not comprehensive, these examples represent cutting-edge techniques published in high-tier journals that nonetheless miss the mark when it comes to the safe performance of medication tasks.

1. MedAgentBench & Potassium:MedAgentBench was published in NEJM AIand was proposed to serve as a virtual EHR with a series of proposed benchmark tasks (Jiang et al. 2025). Seven tasks were included, with most tasks either related to information retrieval, documentation, or test-referral-medication ordering. For the medication ordering task, the LLM was instructed to review the patient’s most recent potassium level and if below 3.5mEq/L, provide replacement according to dosing instructions where for every 0.1mEq/L increase desired, replace by 10mEq of potassium. While this may very well be an appropriate rule for many hospitalised patients, it fails to capture the true complexity of hypokalaemia management, which includes different goals for potassium repletion based on disease state (e.g., a patient with a cardiac history may require a more aggressive repletion goal while a patient currently receiving targeted temperature management (TTM) may require a less aggressive goal, given that upon rewarming potassium will exit the cell), different calculations for appropriate repletion based on disease state and past medical history (e.g., patients with renal disease would likely correct more than 0.1mEq/L per every 10mEq repletion received), and different considerations for route and timing of repletion (e.g., a patient with electrocardiogram changes may require intravenous repletion at a faster rate compared to a stable patient with moderately low potassium who would be more appropriate for oral repletion). Potassium disorders can result in lethal cardiac arrhythmias (indeed, potassium chloride is often part of the drug cocktail used in capital punishment cases): while these disorders are commonly managed in the acute care setting, they are no less complex or dangerous. The simplification of MedAgentBench’s “potassium replacement” to a single rule is a dangerous one that verges on clinically irrelevant. For perspective, a recent report using hypokalaemia cases with more complex medical history (e.g., renal replacement therapy, TTM, acute kidney injury, refeeding syndrome, myocardial infarction history) revealed that GPT-5-Chat reported high confidence for all cases but 100% of responses contained a medication error. As measured by the Harm Associated with Medication Error Classification [HAMEC]) classified as minor to severe, a third of cases reporting a medication error of “serious” or “severe” (Blotske et al. 2026). 

2. DrugGPT and DDI-Corpus:DrugGPT trained on existing medical literature has published results for its accuracy in answering medication-related questions and conducting medication-related tasks (Zhou et al. 2026; Herrero-Zazo). For management of drug-drug interactions (DDIs), DrugGPT was tested using DDI-Corpus, a manually annotated dataset of 5,028 DDIs. However, this dataset was created by extracting DDIs from published scientific articles and was reviewed by pharmacists who ensured that each DDI was listed as “no effect, mechanism, or advise” to describe the type of interaction. Therefore, they reviewed what the article said, but they did not include any clinical interpretation. This resulted in significant duplicates or contradictions as a DDI may be described by multiple papers with a different mechanism (e.g., meropenem and valproic acid are listed as mechanism but clinically this combination should be avoided since meropenem reduces valproic acid concentrations to near-zero). Additionally, many of the DDIs include drugs that are no longer on the market or used in clinical practice, so while this appears to be a robust dataset, it lacks both clinical validation and relevance and may not be an appropriate resource to train an LLM. 

3. Physician Reasoning Tasks:A recent article published in Science reviewed the ability of OpenAI o1 against hundreds of physicians to complete various “reasoning” tasks (Brodeur et al. 2026). However, the majority of these tasks included diagnostic accuracy on text rather than clinical care (Sikora and Henry 2026). Many tasks included listing every possible diagnosis on the differential diagnosis list or every possible laboratory test that could be performed; however, this is not evaluating an LLM’s ability to actually treat a patient. Additionally, these tasks did not compare true patient outcomes with care by an LLM versus a physician, and thus, the claim that the LLM outperformed physician baselines may be an overstatement without further evaluation of tasks that include true reasoning.

Recent work on LLMs for medication safety has begun to move beyond general drug-information queries toward more task-specific forms of medication decision support, including DDI identification, medication reconciliation and deprescribing support, prescription or dosing review, and pharmacovigilance (Ong et al. 2025). For example, Pais et al. (2024) developed MEDIC (Medication Direction Copilot) to reduce direction-related near-miss events by using LLMs. Similarly, Ong et al. (2025) evaluated a retrieval-augmented LLM clinical decision support system for medication chart review and found that it improved detection of serious drug-related problems when used as a pharmacist co-pilot. However, important limitations have also been identified for LLMs. In hospitalised patient data, ChatGPT’s DDI was sensitive to prompt wording and showed limited agreement with pharmacist assessment (Radha et al. 2024). For interventions in hospitalised renal dysfunction patients, ChatGPT performed poorly despite the apparently rule-based nature of dose adjustment in renal dysfunction (van Nuland et al. 2024). Comparative clinical pharmacy studies have also found that although ChatGPT may perform well on drug counselling tasks, it remains weaker than clinical pharmacists for prescription review, patient education, adverse drug reaction recognition, and causality assessment (Huang et al. 2024).

Benchmark Evaluations of LLMs for Medication Tasks in the CMM Process

Our recent work has focused on building clinician-validated benchmarks for real medication tasks in comprehensive medication management (CMM). These studies focus on whether an LLM can correctly and safely complete specific medication tasks, such as identifying DDIs, checking medication routes and formulations, adjusting doses for kidney function, generating medication orders, and recognising fake medication names (Sikora et al. 2026; Henry et al. 2026; Blotske et al. 2026; Blotske et al. 2025; Zhao et al. 2025; Chase et al. 2025; Sikora and Henry 2026). Across these studies, LLM performance was uneven and task-dependent: a model could perform well on one medication task but poorly on another, and some models gave inconsistent answers across repeated runs. CMM requires decisions that are precise, consistent, and safe. Together, these findings show why LLMs need medication-specific, clinician-validated benchmarks before they can be trusted for medication-related clinical decision support. Medication management is not only about knowing drug facts but about applying those facts in a complex environment of competing risks and concerns (e.g., severe infection vs acute kidney injury). Next steps include benchmarks that evaluate both the final answer and whether the model’s reasoning process is safe. 

Automated Benchmarking: MedMatch and Medication Standards

Medication errors account for roughly 50% of all preventable medication related harm worldwide (Hodkinson et al. 2020). These errors can occur at any point in the medication use process (e.g., ordering, verification, administration) and at any point within the medication order (e.g., dose, frequency, route). The ability of an LLM to evaluate medication orders for errors or for an automated evaluation of LLM performance requires some element of standardisation: without this, LLMs have been shown to view ‘q8h’ and ‘every 8 hours’ differently, despite the same semantic meaning, or may miss the nuance of ‘every 8 hours’ vs ‘three times per day’ (which can be very different in the inpatient vs outpatient settings) among many other more overt error types. To date, no standardisation of medication orders has been attempted due to the complex nature of language structure and variability.

Our team has begun this work by evaluating lexical overlap patterns for medication order structure and clinician preferences (Blotske et al. 2026). We designed a standardised medication order format (MedMatch) in alignment with human clinician expectations. We believe this standardisation supports future automated performance evaluation and safety benchmarking of LLMs within the treatment space. These MedMatch medication standards serve as a tool for evaluating surface-level coherence by forcing key contextual components required for safe medication ordering (e.g., drug, dose, frequency, route). Though coherence evaluations do not replace factual evaluation or fix hallucinations, it is a complementary tool if used in tandem to assess an LLM's ability to reason while remaining structurally coherent across prompting. These standards are proposed in Table 2. Utility exists in providing the LLMs with structured output formats and with the ability to automate checking of outputs vs answer keys, instead of relying on resource-intensive clinician validation. 

Beyond the Surface: A Framework for CMM 

Given the potential of medication errors to cause patient harm or even death, performance evaluation constructs are essential for evaluating AI outputs that miss or endorse medication errors. A standardised grading framework for performance evaluation of medication management tasks has been proposed, drawing on four domains: safety, factuality, clinician preference, and completeness. These domains are used to evaluate the three key tasks of CMM, collecting patient information, designing medication regimens, and analysing information. This framework was drawn from the American Society of Health-System Pharmacy (ASHP} Required Competency Areas, Goals, and Objectives for Postgraduate Year Two (PGY2) Critical Care Pharmacy Residencies and previous studies that have implemented a multi-part clinician grading scale (Zhou et al. 2026; AHSP 2016). Additionally, the Harm Associated with Medication Error Classification (HAMEC) was incorporated into the framework to aid in categorisation of potential medication-related harm as part of the generative AI evaluation (Gates et al. 2019). This score ranges from 0 to 4 (Table 3) (Gates et al. 2019). The framework also includes automatic failure criteria, when the LLM either hallucinates information or proposes something that could cause serious or severe harm (HAMEC 3 or 4). 

Conclusion 

Generative AI holds great potential for improving the safety and efficacy of medication treatment. However, this potential is balanced by equally great potential for errors and harm. Generative AI works differently than traditional rules-based approaches and requires different evaluation approaches. The task-specific nature of this framework, combined with automatic failure criteria, clinician preference and alignment domains, and scaled approach, has potential to be adapted to other domains. Next steps include the testing, application, and clinical validation of such a framework for real-world medication decision-making.

Funding

Funding through Agency of Healthcare Research and Quality for Dr Sikora was provided through R01HS029009.

Conflict of Interest

None.