Nudging clinical supervisors to provide better in-training assessment reports

Show and Tell Perspect Med Educ (2020) 9:66–70 https://doi.org/10.1007/s40037-019-00554-3 Nudging clinical supervisors to provide better in-training assessment reports Valérie Dory · Beth-Ann Cummings · Mélanie Mondou · Meredith Young Published online: 17 December 2019 © The Author(s) 2019 Abstract Introduction In-training assessment reports (ITARs) summarize assessment during a clinical placement to inform decision-making and provide formal feedback to learners. Faculty development is an effective but resource-intensive means of improving the quality of completed ITARs. We examined whether the quality of completed ITARs could be improved by ‘nudges’ from the format of ITAR forms. Methods Our first intervention consisted of placing the section for narrative comments at the beginning of the form, and using prompts for recommendations (Do more, Keep doing, Do less, Stop doing). In a second intervention, we provided a hyperlink to a detailed assessment rubric and shortened the checklist section. We analyzed a sample of 360 de-identified completed ITARs from six disciplines across the three academic years where the different versions of the ITAR were used. Two raters independently scored the ITARs using the Completed Clinical Evaluation Report Rating (CCERR) scale. We tested for differences The research was conducted at McGill University, Montreal (QC), Canada V. Dory () Department of Medicine and Centre for Medical Education; Faculty of Medicine, McGill University, Montreal, QC, Canada B.-A. Cummings Undergraduate Medical Education, Department of Medicine, and Institute of Health Sciences Education; Faculty of Medicine, McGill University, Montreal, QC, Canada M. Mondou · M. Young Department of Medicine and Institute of Health Sciences Education; Faculty of Medicine, McGill University, Montreal, QC, Canada 66 Nudging for better assessment reports between versions of the ITAR forms using a one-way ANOVA for the total CCERR score, and MANOVA for the nine CCERR item scores. Results Changes to the form structure (nudges) improved the quality of information generated as measured by the CCERR instrument, from a total score of 18.0/45 (SD 2.6) to 18.9/45 (SD 3.1) and 18.8/45 (SD 2.6), p = 0.04. Specifically, comments were more balanced, more detailed, and more actionable compared with the original ITAR. Discussion Nudge interventions, which are inexpensive and feasible, should be included in multipronged approaches to improve the quality of assessment reports. Keywords Workplace-based assessment · Faculty development · Feedback Background Competency-based medical education relies heavily on workplace-based assessment to guide learning and inform decisions about learners’ attainment of competence [1]. Workplace-based assessment is traditionally documented at the end of a clinical rotation in an in-training assessment report (ITAR—previously referred to as in-training evaluation report or ITER)[2]. Although the shift to competency-based medical education is leading to more frequent documentation of specific assessment events, ITARs continue to play a role in synthesizing assessments for decision-making and for providing formal feedback to learners [2]. To effectively support decision-making and feedback, ITARs must meet quality standards [3]. During a reform of our undergraduate medical education program at McGill University, the committee responsible for curriculum renewal in clerkship (i.e. the clinical phase of the curriculum, in the third and fourth years Show and Tell of a 4-year curriculum) identified several issues with our ITARs. Specifically, numeric grades appeared inflated (with average ratings in the ‘exceeds expectations’ range), and comments were considered generic and uninformative by both course directors and student representatives. Faculty development has been championed as a means of ensuring quality of workplace-based assessment in general [4] and of ITAR completion in particular [5]. However, organizing effective faculty development initiatives that reach the large numbers of clinical supervisors completing ITARs is resourceintensive. Nudge theory proposes that small, lowcost changes to the ‘environment’ in which decisions are made can increase the likelihood that individuals will behave in desired ways without coercion [6]. Examples of nudges include presumed consent for organ donation with means for individuals to register refusal (i.e. using default options to influence behaviour), providing the estimated number of calories burned on gym equipment (i.e. providing feedback to influence behaviour), or stating the rate of tax compliance in a letter to tax-payers (i.e. using social norms to influence behaviour)[6]. Nudges have proved effective in influencing a variety of behaviours in diverse domains from nutrition (e.g. providing smaller plate sizes or portions, modifying food labelling) [7] to the environment (e.g. providing social comparisons in electricity bills, making the default energy provider an environmentally responsible one) [8]. This project examined whether nudges, i.e. changes to the environment in which clinical supervisors provide assessment data, specifically changes to the structure of ITAR forms used for undergraduate clinical placements, could improve the quality of the data generated from ITARs. Development and implementation of the nudge interventions Nudge interventions (Tab. 1) As part of our curriculum reform, the program shifted to pass-fail grading in an effort to encourage students to prioritize learning over competition for grades [9]. This implied concurrent changes to the ITARs, aimed at encouraging supervisors to provide more narrative comments, more balanced comments (i.e. including both strengths and areas for improvement), and more actionable comments (i.e. with specific recommendaTable 1 Overview of the format of in-training assessment forms used tions to learners about how to improve, not just what to improve), which learners could use to direct their learning. Our original locally developed ITAR form had 8 checklist items and 12 5-point rating scales items, followed by a single free-text comment box, and an overall rating item. In 2015 (Intervention 1), we moved the comments section to the beginning of the form to ‘nudge’ supervisors to provide more narrative comments. We also split the single comment box into four distinct comment boxes (Keep doing, Do more, Do less, Stop doing), to ‘nudge’ supervisors to write more balanced and actionable comments. Specific items (n = 27) were all in checklist format (attained course objectives/has not yet attained course objectives/not observed) to reflect the program’s shift to pass-fail grading. In 2016 (Intervention 2), we addressed feedback (provided by clinical supervisors to course directors) about the comment boxes being cumbersome, by replacing our previous four free-text boxes with two boxes: one to describe performance and one to provide recommendations (whic (...truncated)