Clinician performance and prominence of diagnoses displayed by a clinical diagnostic decision support system.

Clinician Performance and Prominence of Diagnoses Displayed by a Clinical Diagnostic Decision Support System Eta S. Berner, EdD, Richard S. Maisiak, PhD, MSPH, Gustavo R. Heudebert, MD and K. Randall Young, Jr., MD University of Alabama at Birmingham, Birmingham, AL ABSTRACT Clinical decision support systems (CDSS) can impact both diagnostic and therapeutic decision-making, but physicians sometimes fail to heed the appropriate CDSS advice, or become influenced in a negative way by the CDSS. This study examined the relationships among clinicians' prior diagnostic accuracy, theperformance of a diagnostic CDSS, and how the CDSS influenced the accuracy of the clinician's subsequent diagnoses. Results showed that (1) clinicians who already were considering the correct diagnosisprior to using the CDSS were more likely to get the CDSS to produce the correct diagnosis in a prominent position than those not considering it initially; (2) physicians are strongly anchored by their initial diagnoses prior to using the CDSS; and (3) changes in the clinicians' diagnoses after using the CDSS are related to presence or absence of the correct diagnosis in the top 10 diagnoses displayed by the CDSS. INTRODUCTION Clinical decision support systems (CDSS) have been shown to decrease medication errors and improve diagnostic decision-making. 1-3 Groups such as Leapfrog and others have advocated the use of reminder and alerting systems and interest in computerized provider order entry with clinical decision support is increasing. 4 Although these systems are clearly capable of influencing physicians in a positive manner, there are also reports in the published literature that physicians may fail to alter their original wrong decisions, or worse, may change correct decisions to incorrect ones after using CDSS. 2'5'6 For instance, Teich et al. found that some decisions were easier to influence than others. In particular, they commented that it is difficult to get physicians to change plans they had already made. 5 Galanter et al. published an article describing a situation where a CDSS provided an alert of a medication overdose that the attending clinician repeatedly ignored. 6 On the other hand, Friedman et al, in an article documenting the effectiveness of two diagnostic decision support systems mentions that sometimes their subjects switched from the correct diagnosis to an incorrect one after using the system. 3 Although Friedman et al.'s subjects improved more when the CDSS provided the correct diagnosis, it was not immediately clear what factors related to what Friedman et al. termed "negative consultations." It is possible that the CDSS provided misleading information that caused the subjects to change, or that the subjects did not attend to or understand what they were viewing in spite of accurate CDSS information. Similarly, in other studies where the CDSS provided useful information that was not acted upon, it is not clear whether the subjects were anchored to a prior course of action or whether there was some other reason for failing to heed the suggestions. As CDSS move out of the research arena and into practice settings, it becomes important not just to document when they are effective, but also to systematically examine the situations in which they do not work a~ intended. Such data are important for improving the CDSS as well as for instructing users on the most appropriate ways to utilize the CDSS output. This paper provides an analysis of the impact of CDSS performance on diagnostic decision-making. Studies of the impact of diagnostic CDSS have tended to examine how the physician performs without examining in detail the what the CDSS actually provided, other than the correct diagnosis being present on the CDSS list.2'3 Only by examining the CDSS performance can we begin to understand how it influences clinician performance. In the present study we examined the effect on physician diagnostic performance of the presentation of the correct diagnosis in a prominent or less prominent position in the output of a diagnostic CDDS. Although we focused on a diagnostic system, the issue of the factors that lead a CDSS to change, or fail to change, clinician decision-making is important for all kinds of CDSS. AMIA 2003 Symposium Proceedings − Page 76 METHODS The subjects were 70 internal medicine residents who took part in a study examining the effects of training and practice on use of a diagnostic decision support system. The diagnostic decision support system was Quick Medical Reference (QMWM), version 3.8.5 marketed by First DataBank. It was available via the Internet on a server running Citrix Metaframe, version 1.8 and Windows NT, version 4.0. These residents had all been given two to four hours of training on the use of QMR, had QMR available online for use for a two-month period and then used QMR to assist them with the diagnosis of four diagnostically challenging paper and pencil cases. The cases were selected from a previous sample of diagnostically challenging cases. 7 Although all of the cases were rare and/or atypical, the correct diagnosis (based on definitive test results) was in QMR's knowledge base and, with optimal use of QMR, appeared within the top 20 of QMR's suggestions. Subjects reviewed the case, prepared their own differential diagnosis, used QMR in any manner they chose, and then revised their differential diagnosis. A maximum of 20 diagnoses were allowed. For each case, subjects saved the last QMR screen they were looking at as a text file. This permitted determination of whether the correct diagnosis was either being actively considered by the resident and/or or presented to them as a possibility by QMR. Thus, there were a total of 280 cases analyzed. For eight cases, the screens were inadvertently not saved and those cases are excluded from further analyses, making the total analyzed 272. Final diagnosis-For each case, this was scored correct if the correct diagnosis was included on the resident's final differential after using the CDSS and incorrect if the correct diagnosis was not included on the differential. For each case we determined the resident's unaided diagnosis, the position of the correct diagnosis on the last QMR screen saved for that case, and the resident's final diagnosis. Table 1 Unaided diagnosis--For each case, this was scored correct if the correct diagnosis was included on the resident's initial differential prior to using the CDSS and incorrect if the correct diagnosis was not included. Position of the correct diagnosis on the last screen s a v e d - R a n k of the correct diagnosis as displayed o n QMR's screen. The top diagnosis was rank 1. We grouped the ranks into strata of five as shown in Tables 2 and 3. We considered display within the top two strata (e.g, first ten diagnoses on the list) as a prominent position. Previous research testing QMR performance in a more artificial situation has shown that when QMR displays the c (...truncated)