Machine learning (ML) is a branch of artificial intelligence in which computers identify patterns in large data sets to make predictions or classifications. The rise of digital health data and electronic health records (EHRs) provides opportunities to use ML to optimise healthcare through improved diagnosis, prognosis, treatment pathways, medication dosing, healthcare efficiency, and risk identification for adverse outcomes.
AutoML, a sub-field of ML, automates the creation and validation of ML models, reducing the time data scientists spend on these tasks. Auto ML platforms streamline processes like data preparation, feature engineering, model optimisation, training, and validation. This automation supports data democratisation, making data handling more accessible to non-experts and reducing human error and bias.
AutoML has applications in medical imaging, disease diagnosis, and EHR data analysis. Popular AutoML platforms include commercial options like DataRobot and Azure AutoML and open-source programs like H2O AutoML and Auto-Sklearn. Selecting the right AutoML platform for a healthcare organisation involves assessing its compatibility with the organisation's digital infrastructure and governance.
This article published in Jamia Open discusses developing a checklist to evaluate Auto ML platforms for a multi-hospital service. The chosen platform would predict the dosing of intravenous heparin in patients with acute thrombotic disorders, aiming to improve therapeutic dosing accuracy and reduce complications compared to current weight-based nomograms.
Methodology for Selecting AutoML Platforms in Healthcare: Insights from Metro South Hospital
A rigorous three-step methodology was employed in selecting an AutoML platform for Metro South Hospital and Health Services in southeast Queensland, Australia. Step 1 involved identifying key requirements through collaboration among clinical informatics experts, data scientists, clinicians, and stakeholders from the Metro South Clinical Artificial Intelligence Working Group. This process included a literature review and stakeholder meetings to establish 21 functional and six non-functional criteria essential for platform selection, categorised by importance.
Step 2 consisted of a market scan using the Gartner Magic Quadrant for Data Science and Machine Learning Platform March 2021, identifying 20 potential vendors. Two vendors meeting local regulatory requirements and demonstrating application to relevant use cases were shortlisted (#9 and #12). These vendors were invited to submit detailed applications addressing all criteria and participate in live demonstrations.
Step 3 involved a panel assessment by four experts including the Director of Clinical Informatics, Director of Pharmacy, Director of Internal Medicine, and a senior data scientist. Each vendor's written responses were independently evaluated against the criteria, categorised as fully compliant, partially compliant, or non-compliant, and scored qualitatively from 0 to 10. Final scores were averaged across panellists for each criterion, contributing to a total maximum score of 270 across functional and non-functional criteria.
Adjustments were made to criteria based on vendor feedback, and all panellists attended live demonstrations before finalising their assessments. This structured approach ensured the selection of an AutoML platform aligned with Metro South's digital environment and healthcare needs.
Evaluating AutoML Platforms in Healthcare
The selection criteria for an AutoML platform at Metro South Hospital and Health Services were structured into 27 criteria, categorised as mandatory (14), highly desirable (11), and desirable (2), reflecting core health service needs. This approach, involving input from a diverse team of clinicians, data scientists, and stakeholders, aimed to ensure transparency, objectivity, and comprehensiveness in platform evaluation. Cost considerations were excluded from the criteria to focus solely on utility assessment.
While AutoML platforms offer efficiency and scalability advantages over traditional ML techniques, challenges such as data quality, interoperability of EHR systems, and transparency of black-box models have been noted. Despite these limitations, AutoML tools show promise in improving model performance and accessibility, albeit requiring oversight to ensure appropriate use and interpretation of results.
The methodology acknowledges limitations such as small vendor sample size and potential influence of organisational cost considerations on platform selection. However, strengths include criteria refinement through vendor feedback and practical validation via a specific healthcare use case involving patient safety implications.
Description of criteria and rating categories
Functional criteria | Category |
1. Support the handling of multiple data sources (ie, structured, unstructured) | Mandatory |
2. Perform feature engineering on imported data sets | Mandatory |
3. Data cleansing and data transformation capabilities | Highly desirable |
4. Conduct supervised learning | Mandatory |
5. Apply ensemble models or blend deep learning models with rule-based models | Mandatory |
6. Apply multiple models on given data sets to determine best fit-for-purpose | Mandatory |
7. Update libraries used in developing data models | Mandatory |
8. Customize for additional tuning/optimisation of developed models | Mandatory |
9. Conduct unsupervised, reinforcement and deep learning | Highly desirable |
10. Perform natural language processing on given data sets | Desirable |
11. Apply different metrics when developing a model | Mandatory |
12. Generate documentation detailing model findings with exportability in MS Word or PDF format | Mandatory |
13. Have a graphing function and plotting performance of a developed model | Highly desirable |
14. Allow for multiple models to be deployed into production | Mandatory |
15. Scoring pipelines to make predictions on newly acquired data | Mandatory |
16. Export developed models for use outside the Auto ML platform | Highly desirable |
17. Encompass model monitoring capability to identify model drift | Highly desirable |
18. Have action logs and the ability to audit historic user modelling/system interactions | Mandatory |
19. Provide explanation for predictions made through scoring pipelines | Mandatory |
20. Conduct optimisation process to the input variables to achieve specific outcome | Highly desirable |
21. Customizable benefits calculation for a developed model | Desirable |
Looking forward, the checklist provides a foundational tool for selecting AutoML platforms across healthcare settings, adaptable for different organisational contexts and evolving with advances in AutoML technology.
Source & Image Credit: JAMIA OPEN
