Machine learning tools for emergency care can generate early predictions, but clinical value depends on whether staff can use them within daily work. A 2026 publication in the International Journal of Medical Informatics assessed the feasibility and early implementation of a machine learning hospitalisation prediction tool in the emergency department of Leiden University Medical Center, a Dutch tertiary care centre with about 23,000 annual visits. The implementation was evaluated with the Medical Research Council framework and focused on adoption, staff perceptions, implementation barriers and contextual factors, rather than clinical effectiveness. The tool produced continuously updated admission and discharge probabilities through a web-based dashboard. Four months of implementation showed limited use, despite initial interest, and revealed a gap between predictive performance and operational value.
Low Use Despite Early Interest
The emergency department receives ambulance arrivals, walk-ins and referrals from general practitioners and specialists. Trained nurses use the Manchester Triage System, while admission decisions are made by emergency physicians in collaboration with the receiving speciality. Nurses do not have admission authority. During the implementation period, there was no centralised bed coordinator with mandate, and patient flow was affected by exit block, including delays before and after admission decisions. The setting therefore combined varied patient arrivals with shared but unevenly distributed admission responsibilities across staff groups.
The intervention combined a previously validated machine learning algorithm with a web-based interface and an educational campaign. Three multidisciplinary focus groups involving emergency physicians, nurses and consulting specialists shaped the interface and implementation procedures. A Delphi process with 22 stakeholders informed training needs and supported presentations, posters and hands-on support during early implementation.
The tool became available to all emergency clinicians and was intended for discretionary use during routine care. Predictions started shortly after patient arrival, were updated every five minutes and showed patient-level probabilities only. No automated recommendations, predefined actions or mandatory workflow points were attached to the output, reflecting clinicians’ preference to avoid extra decision rules at this stage.
Workflow Barriers Limited Practical Value
After four months of exposure, 25 staff members responded to a staff experience survey, including 11 physicians and 14 nurses. Around half reported rarely using the tool, while 17% never used it. Use was mainly logistical, including identifying the number and identity of likely admissions, and less often supported clinical decision-making. Nurses reported slightly higher use for monitoring admissions and discharges. Physicians mainly used the tool for communication with colleagues.
The main barriers were practical and organisational. The dashboard had to be opened separately from the electronic health record at the start of each shift, which created an extra step in an already pressured clinical environment. Staff also reported that predictions were not linked to operational actions, limiting their effect on patient flow. A high probability of admission did not automatically trigger provisional bed allocation, early transfer coordination or other structured responses.
Must Read: Reducing Hospitalisations Through Risk Prediction
Role differences also shaped perceived utility. Nurses could recognise planning value but could not act independently on admission predictions. Physicians retained admission decision-making authority and often waited for diagnostic results, specialist agreement or supervisory consultation before acting. Capacity constraints in receiving departments further reduced the usefulness of early prediction. As a result, the tool supplied information but did not alter the workflows that determine when patients leave the emergency department.
Perceptions Shift After Implementation
Staff perceptions of the overall hospitalisation process remained broadly stable before and after implementation. Confidence in admission or discharge decisions, views on communication with colleagues and perceptions of admission complexity did not substantially change. However, views on artificial intelligence became less positive in specific areas after direct exposure to the tool.
Agreement declined significantly with the view that a real-time dashboard could support daily work. Perceived usefulness of artificial intelligence for decisions about ward type, such as intensive care or general ward placement, also declined. The reduction appeared more pronounced among nurses than physicians, consistent with nurses’ closer involvement in patient flow and more limited ability to act on predictions.
Concern that artificial intelligence would replace clinical roles declined after implementation. Staff also considered insight into the key predictive factors of the model less important after using the tool. These shifts suggest that experience with the dashboard reduced both optimism about its practical value and anxiety about professional replacement. The model itself retained good predictive performance and calibration during the implementation period, but accurate prediction did not translate into routine use. The central issue was not whether the tool could estimate admission probability, but whether the department and downstream services could act on that information.
Operational outcomes changed between the pre-implementation and implementation periods, but the changes cannot be attributed to the tool because adoption was low and the patient mix differed. Emergency department length of stay above four hours increased from 19.8% to 26.1%, and hospitalisation rose from 31.8% to 33.3%. Adjusted analysis found no significant association with length of stay, while differences in hospitalisation reflected contextual variation. The experience shows that machine learning tools need electronic health record integration, clear action protocols, role-specific responsibilities and alignment with hospital-wide capacity processes before predictive output can support meaningful operational change.
Source: International Journal of Medical Informatics
Image Credit: iStock
References:
Raven W, de Hond A, Vermeire J et al. (2026) From prediction to practice: early implementation of a machine learning–based hospitalization prediction tool in the emergency department. International Journal of Medical Informatics, 215:106446.
