Artificial intelligence is increasingly shaping clinical decisions in triage, deterioration alerts and diagnostic support, but biased inputs can reproduce existing inequities in care. Evidence from the United States includes findings on pulse oximetry and a commercial care-management algorithm, both showing how apparently objective technologies can disadvantage Black patients. Historical data can embed unequal testing, spending and access into models, while inaccurate measurements can affect downstream decision tools. Nurses are particularly well placed to identify discrepancies because bedside observations may reveal when an automated output does not match a patient’s clinical condition, making clinical oversight an important part of equitable technology use.
How Bias Enters Clinical AI
Artificial intelligence systems identify statistical patterns in historical records and use those patterns to guide future decisions. If training data reflect unequal testing, healthcare spending or access to specialists, those inequalities can become part of the model. The resulting output may appear neutral because it is expressed numerically, even when it directs resources away from patients who have historically received less care.
A care-management algorithm examined in 2019 shows how bias can enter through the choice of proxy variables. The commercial tool used previous healthcare spending as a substitute for illness severity. Because less had historically been spent on Black patients with comparable levels of illness, the algorithm classified those patients as healthier than equally sick White patients. Fewer Black patients were consequently identified for additional case-management resources.
Must Read: Healthcare AI Regulation Needs Lifecycle Oversight
Replacing cost with a more direct measure of illness substantially changed the outcome. After the model was corrected, the share of Black patients identified for extra care nearly tripled. The example shows that bias can arise from model design rather than from an unavoidable limitation of artificial intelligence. It also shows that an apparently neutral score can conceal unequal treatment and that identifying the source of bias can allow the model to be changed without abandoning the underlying technology. The design choice therefore matters.
Bias Can Travel Through Clinical Systems
Bias does not originate only in algorithms. It can also enter clinical decision systems through the measurements on which those systems depend. Pulse oximetry provides an example because readings can feed into hospital early-warning systems and deterioration algorithms.
A 2020 investigation found that pulse oximeters were more likely to miss dangerously low blood oxygen levels in Black patients than in White patients. The devices tended to overestimate oxygen saturation in people with darker skin tones. In rapidly changing situations such as respiratory decline, a falsely reassuring reading can delay supplemental oxygen or escalation of care.
When a measurement feeds several systems, the effect can extend beyond the original device. An inaccurate oxygen saturation value can move into other tools that use the reading as an input, including deterioration alerts used by clinical teams. A single measurement problem can therefore influence later automated recommendations rather than remaining isolated at the bedside.
These examples challenge the assumption that numerical outputs are necessarily objective. Historical patterns, proxy measures and device performance can all shape the information presented to clinicians. Bias may therefore appear as a normal-looking score, alert or measurement, making bedside assessment especially important when technology produces an output that does not fit the condition of the patient or the wider clinical picture.
Clinical Oversight and AI Governance
Nurses are positioned to identify discrepancies between automated recommendations and what is observed at the bedside. Their role in addressing bias can extend from individual care decisions to the governance of artificial intelligence systems.
The 2025 Bias Elimination for Fair and Responsible Artificial Intelligence in Healthcare framework, known as BE FAIR, sets out a specific role for nurses in AI governance. It calls for nurses to participate in governance committees, question outputs associated with unexplained disparities in care and advocate for training data that represent the communities being served.
Fairness assessment is therefore not confined solely to data science teams. Nurses can contribute clinical knowledge about how technologies perform in practice and whether outputs correspond with patient observations. They do not need to become programmers or statisticians to evaluate whether a tool has been tested on relevant patient groups, whether performance differs between groups and what occurs when the tool produces an incorrect result.
The examples involving pulse oximetry and care management also show why clinical scrutiny matters after implementation. When bias is identified, it can sometimes be corrected without sacrificing a tool’s overall usefulness. Transparency, representative data and attention to unequal outcomes remain central as artificial intelligence becomes more embedded in triage, monitoring and care coordination throughout clinical practice.
Artificial intelligence can support clinical decision-making while also carrying forward inequities embedded in historical data, design choices or underlying measurements. The examples of pulse oximetry and care management show how bias can influence both bedside information and the allocation of additional support. Clinical observation remains important when automated outputs do not align with a patient’s condition. Nurses can also contribute beyond individual encounters through governance, scrutiny of unequal outcomes and advocacy for representative training data. As AI becomes more integrated into care, equity depends on recognising and addressing bias throughout its use.
Source: HealthData Management
Image Credit: iStock
