Early detection and response to abnormal laboratory values is crucial in critical care. However, traditional electronic medical record (EMR) interfaces mark laboratory tests as abnormal based on standard reference ranges derived from healthy, middle-aged adults. This yields many false positive alerts with subsequent alert-fatigue when applied to complex populations like hospitalised, critically ill patients. For example, a haemoglobin of 9.5 mg/dl may raise concerns at the family physician’s visit in a healthy person, but be fully acceptable in non-bleeding patients in the ICU. Such a traditional, “one size fits all” approach therefore can prevent the recognition of true abnormalities.
Novel EMR interfaces using adjusted reference ranges customised for specific patient populations may ameliorate this problem. In a study published by PLoS One, researchers sought to compare the accuracy of abnormal laboratory value indicators between two EMR systems (novel vs. traditional) using as a reference standard the judgment of board certified critical care physicians about which data they would like to be alerted given patients’ particular clinical situations.
Research Method
The study was conducted at a tertiary care academic medical centre. Laboratory data from patients consecutively admitted to any of four ICUs (medical, surgical, mixed and trauma) during a two-day period were recorded. For each patient, researchers simultaneously took screenshots of the two EMR interfaces’ laboratory sections at a random time point during the day of admission. A document was created for each participant containing all available lab results and the patient's problem list, omitting information on "interface-origin(s)" and "abnormal indicators" of either interface.
The patient documents were sent to two mutually blinded critical care experts, who marked the values about which they would like to be alerted. All disagreements were resolved by an independent super-reviewer. Based on this gold standard, the investigators calculated and compared the sensitivity, specificity, positive and negative predictive values (PPV, NPV) of customised versus traditional abnormal value indicators.
Patient demographics including age, gender, admission diagnosis, severity of illness (APACHE III), ICU/hospital length of stay and mortality along with mechanical ventilation status were queried from a previously validated ICU database (Multidisciplinary Epidemiology and Translational Research in Intensive Care [METRIC] ICU Datamart).
Descriptive patient data were summarised as median and inter-quartile ranges (IQR) or percent (number) as appropriate. For each EMR interface, the authors calculated sensitivity, specificity and predictive values of abnormal value indicators as compared to the gold standard. For each measure of accuracy, the authors determined exact binomial confidence intervals and used two-sample proportion tests to compare them across the two EMR interfaces.
Results and Discussion
The study included 37 patients with a total of 1,341 laboratory values reported in the traditional EMR interface, of which 754 values (56 percent) were also displayed in the novel interface. The percentage of laboratory values indicated as abnormal was 42 percent (559/1341) in the traditional interface compared to 26 percent (195/754) in the novel interface.
Experts had fair agreement (kappa = 0.39) classifying 71 of all 1,341 (5.3 percent) values as abnormal. One of these 71 (1.4 percent) “truly abnormal” values was not displayed in the novel interface (serum Hydroxybutyrate). Compared to the traditional interface, the novel interface had a similar sensitivity (77 percent vs 85 percent, P = 0.22) and negative predictive value (97.1 percent vs 98.6 percent, P = 0.06) but had a higher specificity ((79 percent vs 61 percent, P<0.001) and positive predictive value (8 percent vs 11 percent, P<0.001).
The novel interface selectively omitted almost half of all available laboratory data and amongst all displayed values marked 38 percent less values as abnormal compared to the traditional interface. The main effect was a substantial reduction in the number of values indicated as abnormal by the interface but considered normal by critical care experts (false positive alerts), which was reflected by the higher specificity and positive predictive value of the novel interface.
The finding that the standard reference ranges in the traditional interface produce a large fraction of false positive notifications (37 percent) is consistent with reports from other studies evaluating alerts in ICU settings more generally.
Study Conclusion
Reference ranges for laboratory values adjusted towards a population of critically ill patients based on expert consensus decreased false positive alerts (increased specificity and positive predictive value) without affecting the fraction of false negative alerts (unchanged sensitivity and negative predictive value). This may reduce alert fatigue among healthcare providers.
Substantial disagreement among critical care experts about which laboratory values should be indicated as abnormal is a major limitation for both developing and testing such customised reference ranges.
Image Credit: Wikipedia
Novel EMR interfaces using adjusted reference ranges customised for specific patient populations may ameliorate this problem. In a study published by PLoS One, researchers sought to compare the accuracy of abnormal laboratory value indicators between two EMR systems (novel vs. traditional) using as a reference standard the judgment of board certified critical care physicians about which data they would like to be alerted given patients’ particular clinical situations.
Research Method
The study was conducted at a tertiary care academic medical centre. Laboratory data from patients consecutively admitted to any of four ICUs (medical, surgical, mixed and trauma) during a two-day period were recorded. For each patient, researchers simultaneously took screenshots of the two EMR interfaces’ laboratory sections at a random time point during the day of admission. A document was created for each participant containing all available lab results and the patient's problem list, omitting information on "interface-origin(s)" and "abnormal indicators" of either interface.
The patient documents were sent to two mutually blinded critical care experts, who marked the values about which they would like to be alerted. All disagreements were resolved by an independent super-reviewer. Based on this gold standard, the investigators calculated and compared the sensitivity, specificity, positive and negative predictive values (PPV, NPV) of customised versus traditional abnormal value indicators.
Patient demographics including age, gender, admission diagnosis, severity of illness (APACHE III), ICU/hospital length of stay and mortality along with mechanical ventilation status were queried from a previously validated ICU database (Multidisciplinary Epidemiology and Translational Research in Intensive Care [METRIC] ICU Datamart).
Descriptive patient data were summarised as median and inter-quartile ranges (IQR) or percent (number) as appropriate. For each EMR interface, the authors calculated sensitivity, specificity and predictive values of abnormal value indicators as compared to the gold standard. For each measure of accuracy, the authors determined exact binomial confidence intervals and used two-sample proportion tests to compare them across the two EMR interfaces.
Results and Discussion
The study included 37 patients with a total of 1,341 laboratory values reported in the traditional EMR interface, of which 754 values (56 percent) were also displayed in the novel interface. The percentage of laboratory values indicated as abnormal was 42 percent (559/1341) in the traditional interface compared to 26 percent (195/754) in the novel interface.
Experts had fair agreement (kappa = 0.39) classifying 71 of all 1,341 (5.3 percent) values as abnormal. One of these 71 (1.4 percent) “truly abnormal” values was not displayed in the novel interface (serum Hydroxybutyrate). Compared to the traditional interface, the novel interface had a similar sensitivity (77 percent vs 85 percent, P = 0.22) and negative predictive value (97.1 percent vs 98.6 percent, P = 0.06) but had a higher specificity ((79 percent vs 61 percent, P<0.001) and positive predictive value (8 percent vs 11 percent, P<0.001).
The novel interface selectively omitted almost half of all available laboratory data and amongst all displayed values marked 38 percent less values as abnormal compared to the traditional interface. The main effect was a substantial reduction in the number of values indicated as abnormal by the interface but considered normal by critical care experts (false positive alerts), which was reflected by the higher specificity and positive predictive value of the novel interface.
The finding that the standard reference ranges in the traditional interface produce a large fraction of false positive notifications (37 percent) is consistent with reports from other studies evaluating alerts in ICU settings more generally.
Study Conclusion
Reference ranges for laboratory values adjusted towards a population of critically ill patients based on expert consensus decreased false positive alerts (increased specificity and positive predictive value) without affecting the fraction of false negative alerts (unchanged sensitivity and negative predictive value). This may reduce alert fatigue among healthcare providers.
Substantial disagreement among critical care experts about which laboratory values should be indicated as abnormal is a major limitation for both developing and testing such customised reference ranges.
Image Credit: Wikipedia
References:
Kilickaya O, Schmickl C, Ahmed A, Pulido J, Onigkeit J, Kashani K, Gajic
O, Herasevich V, Pickering B (2014) Customised Reference Ranges for
Laboratory Values Decrease False Positive Alerts in Intensive Care Unit
Patients. PLoS One. 2014; 9(9): e107930. PMCID: PMC4169437
Latest Articles
laboratory, EMR, value, interface
Early detection and response to abnormal laboratory values is crucial in critical care. However, traditional electronic medical record (EMR) interfaces mar...
