Artificial intelligence (AI) and its latest offering, ChatGPT, is increasingly used in medicine to enhance disease diagnosis treatment and reduce unnecessary patient screenings. However, an international task force, including a bioethicist from the University of Rochester Medical Center, has cautioned that AI medical devices could potentially harm patients and exacerbate health disparities if not thoughtfully designed, rigorously tested, and responsibly employed.
Dr Jonathan Herington was a member of the AI Task Force within the Society for Nuclear Medicine and Medical Imaging. This task force recently issued recommendations for the ethical development and utilisation of AI medical devices, published in two papers in the Journal of Nuclear Medicine. In essence, the task force emphasised the need for increased transparency regarding the accuracy and limitations of AI and outlined strategies to ensure that AI medical devices are accessible to all, regardless of their race, ethnicity, gender, or socioeconomic status.
While the responsibility for proper design and testing primarily falls on AI developers, healthcare providers are ultimately accountable for the responsible utilisation of AI and should avoid over-reliance on AI predictions in making patient care decisions.
Dr Herington emphasised the importance of having a human presence in the decision-making process, stating, "There should always be a human in the loop." Clinicians should view AI as an input into their decision-making process rather than replacing their clinical judgment.
This necessitates that doctors fully understand the intended use of a specific AI medical device, its performance in that context, and any associated limitations. They must then communicate this knowledge to their patients. Additionally, doctors must weigh the risks of false positives and false negatives for a given situation, all while considering structural inequalities.
Transparency is crucial for the developers of these systems, as they must make accurate information about the intended use, clinical performance, and limitations of their medical devices readily available to users. One recommended approach is incorporating alerts within the device or system to inform users about the uncertainty associated with AI predictions.
To reduce uncertainty, developers should carefully define the data used to train and test AI models and employ clinically relevant criteria for evaluating their performance. Mere validation of algorithms is insufficient; AI medical devices should undergo "silent trials," where researchers assess their performance on real patients in real-time, but the predictions are not used for clinical decision-making.
Moreover, developers should ensure that AI models are designed to be effective and accurate in all deployment contexts. There is concern that high-tech, expensive systems might be predominantly used in well-resourced hospitals, potentially favouring better-off patients, while under-resourced or rural hospitals might lack access to such systems or use ones ill-suited for their needs.
AI medical devices are trained on datasets with inadequate representation of Latino and Black patients, making them less reliable for patients from these demographic groups. To prevent further deepening of health disparities, developers should calibrate AI models to account for variations across all racial and gender groups by training them with representative datasets.
Dr Herington stressed the urgency of establishing an ethical and regulatory framework as AI technologies rapidly advance.
Source: University of Rochester Medical Center
Image Credit: iStock
