Governance at the Centre of Clinical AI Deployment

Clinical AI governance is becoming a practical requirement as high-risk systems move from technical validation into routine care. A framework developed at the Institut de Cancérologie de Lorraine in France and published in npj Digital Medicine shows how EU AI Act obligations could be translated into hospital processes. The approach combines central oversight, risk-based project assessment, lifecycle monitoring, reporting of AI-related adverse events and role-specific training. It treats AI deployment as an organisational as well as a technical challenge, with human judgement, data governance, workflow integration and continuous monitoring positioned as essential parts of safe adoption.

Governance Brings Regulatory Duties into Practice 

The framework places a multidisciplinary Data/AI Steering Committee at the centre of institutional oversight. It brings together clinical departments, information systems, data protection, project management and institutional leadership. The committee considers both strategic and technical issues, including priorities, risks, regulatory requirements, feasibility, data availability, interoperability and resources. It acts as a single entry point for AI and data initiatives and can approve, defer, reformulate or reject projects. It also maintains a registry of candidate and deployed AI tools.

Must Read: Ambient AI Scribe Supports Emergency Care Workflows

A shared AI doctrine sets the main principles for use. Human clinical judgement takes priority over algorithmic output, while AI remains a decision-support technology rather than a decision authority. Expected benefits must be balanced against risks, and use must remain aligned with regulatory and ethical requirements. 

Data governance is also central. The framework distinguishes health from non-health data and separates development and testing environments from production use. Patients are informed that their data may be used for AI applications and can opt out.

These arrangements are intended to turn EU AI Act duties into practical hospital responsibilities. High-risk clinical AI may also fall under the Medical Device Regulation when software supports diagnosis, treatment decisions or patient triage. Hospitals therefore need structures for monitoring, human oversight, transparency, staff competency and formal clinical documentation.

Risk Stratification and Lifecycle Oversight 

Risk stratification is based on two factors: the type of data involved and how close the AI function is to clinical decision-making. Governance becomes more demanding as systems move closer to decisions that directly affect patient care. Classification depends on what the tool does, not on the professional using it. 

Lower-risk transversal tools can support documentation, communication or administrative work, particularly when they use non-health or anonymised data. Clinical-organisational tools may use health data for pathway coordination, reporting, quality monitoring or research without directly changing individual clinical decisions. High-risk systems supporting diagnosis, treatment selection or treatment adaptation require stronger controls, including data readiness, workflow assessment and formal lifecycle monitoring.

Before routine use, projects pass through staged evaluation. This can include prototyping, controlled testing, assessment of feasibility and usability and review of workflow integration. Only systems with acceptable technical performance, safe integration and sufficient governance readiness move to wider deployment. 

Monitoring continues after deployment. Indicators can include usage patterns, agreement between AI outputs and clinical decisions, user feedback and signs of performance drift. Predefined conditions can lead to recalibration, suspension or withdrawal.

The framework also introduces structured reporting of AI-related adverse events. Events are grouped into five categories: performance failure, automation bias, workflow disruption, data-related incidents and human–AI interaction failure. Each event is graded by severity and by confidence that AI contributed to it. Reports then follow a dedicated institutional route for review, corrective action and escalation.

Training and Early Implementation 

Human oversight is treated as a practical competency requirement. Training differs according to role. End users need to understand intended use, known limitations, uncertainty and common failure modes. Professionals involved in AI design, evaluation or governance require knowledge of data governance, regulation and lifecycle monitoring. Institutional leaders need understanding of strategic, ethical and compliance responsibilities. 

The framework goes beyond lectures and e-learning by adding proficiency testing. Test cases can come from de-identified institutional scenarios, vendor challenge sets or deliberately modified AI outputs. Users are assessed on whether they recognise an error and whether they respond correctly by correcting or overriding it, escalating it and starting an adverse-event report where appropriate. 

For generative and other non-deterministic systems, a single fixed answer may not be enough. Assessment therefore focuses on error-detection rates across sampled outputs and ongoing quality control rather than one-time validation. Proficiency testing is intended to become a recurring part of professional development. 

Early use at the cancer centre remains limited. The steering committee began structured sessions in 2025 and reviewed around ten projects in its first six months. Only some moved into pilot implementation. Others were deferred or revised because of data governance issues, uncertain regulatory status, limited interoperability, resource constraints or funding considerations.

Implementation also revealed practical challenges, including unclear responsibilities, difficulty recruiting combined clinical and data-science expertise and the scale of organisational change required. The adverse-event reporting system and proficiency-testing programme have been designed, but neither has yet produced formal operational data. 

The framework places institutional readiness at the centre of clinical AI deployment. Governance covers project selection, risk assessment, data controls, ongoing monitoring, adverse-event reporting and staff competency while keeping human clinical judgement in control. Early implementation shows that the approach can be put into practice, but its effect on safety, efficiency and clinical outcomes has not yet been established. The model was developed in a single French cancer centre, so its wider transferability, reporting performance and proficiency-testing impact still require prospective evaluation. 

Source: npj Digital Medicine 

Image Credit: iStock