HealthManagement, Volume 19 - Issue 2, 2019

AI-based prediction in clinical settings: can we trust it?

share Share

AI is proving its worth in delirium prediction

An AI model used for prediction and prevention of delirium in elderly patients is reaping results and gaining the confidence of users. 

The challenge


Users in healthcare are facing the situation that it is becoming increasingly difficult for them to incorporate information available for their decisions in a limited amount of time.


Therefore it is necessary to provide relevant information for the respective decision situation – a context-sensitive presentation of a medical history. In a further step, then clinical decision support can be given.

Appropriate algorithms collect and link available information, they derive indications and recommendations for decision-makers (doctors, nursing staff, therapists, patients) and make it available at the point of care in real time.


You may also like: Digital transformation and the subconscious


But scepticism and mistrust of users have to be adressed: They must be enabled to gain confidence in such systems and trust their information by evident benefits (especially for the patients) and intuitive integration in their user interface.


An AI case already in clinical practice


In this case, a prediction for delirium is offered as decision support, based on statistical and machine learning models ("controlled" and "qualitatively evaluated"). These models are derived from extensive data from the hospital operating company KAGes with approximately 2.1 million longitudinal medical histories from the Austrian province of Styria with approximately 1.2 million inhabitants. In this area about 90% of the acute care beds are operated by KAGes.


Delirium is a highly relevant syndrome with elderly patients. Delirium means an acute, organically induced impairment of the brain. Affected patients respond inappropriately to environmental stimuli, seem "confused" and unable to orient themselves.


In up to 44% of cases, delirium is considered preventable. If left untreated, however, it will lead to a significantly higher risk of complications during hospitalisation or even death. According to studies, the hospital stay is considerably prolonged by a delirium suffered (Dowal 2015).


In a supervised learning process, models were generated with several algorithms. In order to avoid overfitting, a training (75%) and a test dataset (25%) were used. The models were optimised exclusively on the training data set. The performance was validated with the test data set. The different algorithms yielded very similar results. The Random Forest as the best algorithm in this use case achieved an AUROC of 0.90 (“area under the curve“ as statistical performance indicator).


Based on this model, the probability of suffering from delirium is calculated for each newly-admitted patient with data extracted from his medical history. In case a warning is displayed at the clinical work station, the explanatory component can then be activated by clicking on the warning symbol. This then shows the data of the patient which led to the calculated probability, eg the patient has a certain elevated laboratory value, a certain prediagnosis, an accumulation of disease events in the recent past, difficult social conditions, etc.

The attending physician or nurse can then take preventive measures to prevent the impending delirium if possible.




The benefit for a hospital group such as KAGes can be seen in the case of widespread use as follows (estimated conservatively):

  • Approx. 100,000 inpatients in the risk population (60+) per year
  • With an assumed (low) incidence of 3%: 3,000 patients with delirium per year (assuming some potentially preventable delirium would have been detected early enough to prevent delirium)
  • Average 7 days longer stay for preventable delirium, assumed €500 per day: additional costs of €10,500,000
  • Predictive approach detects > 80% of delirium patients early and 25% (low estimate) are considered preventable.
  • Cost savings potential per year: € 2,100,000

Further use cases are in development, eg prediction of avoidable readmissions, prediction of the intensive care necessity of surgical patients to improve the disposition of expensive intensive care beds, etc.


Trust in AI


The prediction of delirium was compared with the estimation of the delirium risk by staff members in a nine-month evaluation project proving the high predictive quality of the model in practice. The acceptance of the users was evaluated as well. This approach has allowed users to gain confidence in using this form of artificial intelligence. This very context-sensitive support in the explanatory component makes work easier for the users and increases patient safety. As a result, the senior physicians and nursing staff of the pilot departments have adopted the solution and have recommended it for wide use in KAGes.




Further application modules are being developed and there is also interest of other hospital providers to use these models, which are based on an extensive body of data from longitudinal medical histories. There are numerous publications, which show good statistical metrics of their prediction models. But the trust of doctors and care staff proved by real clinical use is hardly found. The "last mile" to clinical use has obviously not been made yet. KAGes has already succeeded and is optimistic it will add more use cases.

User scepticism and mistrust have to be adressed and they must be enabled to gain confidence in such systems

In up to 44% of cases, delirium is considered preventable but, if untreated, will lead to complications


Key Points

  • Healthcare users are experiencing information overload
  • Algorithms provide point-of-care information on indications and reccomendations
  • Machine learning is predicting and preventing delirium in an Austrian hospital
  • The potential annual cost savings are €2,100,000
  • Success has enabled user trust in AI
Recommended articles:
For universal healthcare, quality is not the icing, it's the cake 
How tech improved healthcare hiring process
Non-traditional stakeholders making waves in healthcare

Next Article: Artificial Intelligence: a next way forward for healthcare


«« Sustainable healthcare: The Danish model that's streets ahead

Experts and novices: getting the leadership balance right »»


Dowal S et al. (2015) Effectiveness of Multicomponent Nonpharmacological Delirium Interventions A Meta-analysis, JAMA Internal Medicine 175/4:512

Related IssueArticles

Quality Assurance as a function of healthcare business profitability and efficiency   Healthcare needs to invest in quality... Read more

Social Media is gaining significant popularity amongst the medical and allied health professionals. It enables a platform... Read more

Aspects & challenges - Approaches & tools Head-up-tilt testing (HUTT) and autonomic function testing are essential diagnostic... Read more

AI prediction, Predictive, Delerium, User confidence An AI model used for prediction and prevention of delirium shows how it is reaping results and gaining the confidence.

No comment

Please login to leave a comment...

Highlighted Products