The ABCs of DKA: Development and Validation of a Computer-Based Simulator and Scoring System

The ABCs of DKA: Development and Validation of a ComputerBased Simulator and Scoring System Catherine H. Y. Yu, MD FRCPC MHSc1, Sharon Straus, MD FRCPC MSc1, and Ryan Brydges PhD2 1 St. Michael’s Hospital, Toronto, ON, USA; 2Department of Medicine, University of Toronto, Toronto, ON, USA. BACKGROUND: Clinical management of diabetic ketoacidosis (DKA) continues to be suboptimal; simulation-based training may bridge this gap and is particularly applicable to teaching DKA management skills given it enables learning of basic knowledge, as well as clinical reasoning and patient management skills. OBJECTIVES: 1) To develop, test, and refine a computerbased simulator of DKA management; 2) to collect validity evidence, according to National Standard’s validity framework; and 3) to judge whether the simulator scoring system is an appropriate measure of DKA management skills of undergraduate and postgraduate medical trainees. DESIGN: After developing the DKA simulator, we completed usability testing to optimize its functionality. We then conducted a preliminary validation of the scoring system for measuring trainees’ DKA management skills. PARTICIPANTS: We recruited year 1 and year 3 medical students, year 2 postgraduate trainees, and endocrinologists (n=75); each completed a simulator run, and we collected their simulator-computed scores. MAIN MEASURES: We collected validity evidence related to content, internal structure, relations with other variables, and consequences. KEY RESULTS: Our simulator consists of six cases highlighting DKA management priorities. Real-time progression of each case includes interactive order e n t r y, l a b o r a t o r y a n d c l i n i c a l d a t a , a n d individualised feedback. Usability assessment identified issues with clarity of system status, user control, efficiency of use, and error prevention. Regarding validity evidence, Cronbach’s α was 0.795 for the seven subscales indicating favorable internal structure evidence. Participants’ scores showed a significant effect of training level (p < 0.001). Scores also correlated with the number of DKA patients they reported treating, weeks on Medicine rotation, and comfort with managing DKA. A score on the simulation exercise of 75 % had a sensitivity and specificity of 94.7 % and 51.8%, respectively, for delineating between expert staff physicians and trainees. CONCLUSIONS: We demonstrate how a simulator and scoring system can be developed, tested, and refined to determine its quality for use as an assessment modality. Our evidence suggests that it can be used for formative assessment of trainees’ DKA management skills. KEY WORDS: medical education; assessment/evaluation, medical education; clinical skills training, medical education; computer/webbased training, medical education; instructional design, medical education; simulation. Published online July 15, 2015 J Gen Intern Med 30(9):1319–32 DOI: 10.1007/s11606-015-3273-y © The Author(s) 2015. This article is published with open access at Springerlink.com BACKGROUND Diabetic ketoacidosis (DKA) accounts for an estimated 115,000 hospital discharges per year in the USA.1 Clinical management is suboptimal; in a single-centre chart audit of 55 patients admitted with DKA to a large teaching hospital, the mean time to insulin initiation (a key component of therapy) was 207 min, and 75 % were placed on an inappropriate hyperglycemia protocol that did not address the other metabolic derangements of DKA.2 DKA is a medical emergency necessitating hourly assessment of a myriad of dynamic clinical parameters, resulting in numerous critical decision-making points, which are further complicated by the complex interplay between management actions.3 While clinical knowledge is necessary, clinical reasoning and management skills are critical for successful patient management. One before-after study examined the effect of resident education on DKA knowledge4. Fifty-one residents undertook a web-based test consisting of 12 multiple-choice questions before and 6 months after the intervention. In addition to receiving test feedback and links to further reading, they attended two 1-hour didactic lectures and case-based discussion. The authors reported no change in resident knowledge between the two time points. How best to improve residents’ clinical reasoning and management skills related to DKA has yet to be studied fully. In contrast to passive delivery of content (i.e., didactic lectures), research has shown that trainees acquire skills and develop expertise through deliberate practice. Ericsson5, 6 describes deliberate practice as a set of “…activities that have been found most effective in improving performance,” consisting of nine elements: highly motivated learners, welldefined learning objectives, appropriate levels of difficulty, focused repetitive practice, reliable measurements, informative feedback, monitoring and error correction, evaluation and performance, and advancement to the next task.7 A meta-analysis comparing simulation-based training in which trainees followed deliberate practice principles to traditional clinical medical education found 14 studies (6 randomized trials, 3 cohort, 1 case-control, and 4 pre-post studies), which addressed procedural, auscultation, and life support skills in medical students and residents.7 All studies favored 1319 1320 Yu et al.: Development and Validation of a DKA Simulator simulation-based training with deliberate practice over traditional education, with an overall effect size correlation of 0.71 (95 % CI 0.65–0.76, p<0.001). Thus, deliberate practice has strong potential as a framework for designing the training and assessment of clinical skills, including medical students’ and residents’ DKA management skills. These previous studies on deliberate practice have not clarified which of the nine elements are most responsible for the observed performance improvements. In order to optimize the effectiveness of educational interventions employing deliberate practice, a rigorous understanding of its key elements and the contribution of each is central. For example, Pusic et al. have demonstrated that repetitive practice, one of the key elements of deliberate practice, is essential for trainees to develop expertise.8 In a prospective cross-sectional study, 18 pediatric residents were asked to classify whether 234 cases of ankle radiographs were normal or abnormal. Learning was greatest between cases 21 to 50, highlighting the importance of repetitive practice in gaining expertise. Given the high number of repetitions required to gain expertise, Pusic et al. suggest that computer simulation is an ideal medium for tracking the development of deliberate practice and for clarifying which of its nine elements are most useful.9 Two of the key elements of deliberate practice are that informative feedback be provided from educational sources and that assessment scores are available to produce a mastery standard.7 T (...truncated)