Workplace-based assessments of entrustable professional activities in a psychiatry core clerkship: an observational study

Pinilla et al. BMC Medical Education (2021) 21:223 https://doi.org/10.1186/s12909-021-02637-4 RESEARCH ARTICLE Open Access Workplace-based assessments of entrustable professional activities in a psychiatry core clerkship: an observational study Severin Pinilla1,2* , Alexandra Kyrou3, Stefan Klöppel1, Werner Strik3, Christoph Nissen3 and Sören Huwendiek2 Abstract Background: Entrustable professional activities (EPAs) in competency-based, undergraduate medical education (UME) have led to new formative workplace-based assessments (WBA) using entrustment-supervision scales in clerkships. We conducted an observational, prospective cohort study to explore the usefulness of a WBA designed to assess core EPAs in a psychiatry clerkship. Methods: We analyzed changes in self-entrustment ratings of students and the supervisors’ ratings per EPA. Timing and frequencies of learner-initiated WBAs based on a prospective entrustment-supervision scale and resultant narrative feedback were analyzed quantitatively and qualitatively. Predictors for indirect supervision levels were explored via regression analysis, and narrative feedback was coded using thematic content analysis. Students evaluated the WBA after each clerkship rotation. Results: EPA 1 (“Take a patient’s history”), EPA 2 (“Assess physical & mental status”) and EPA 8 (“Document & present a clinical encounter”) were most frequently used for learner-initiated WBAs throughout the clerkship rotations in a sample of 83 students. Clinical residents signed off on the majority of the WBAs (71%). EPAs 1, 2, and 8 showed the largest increases in self-entrustment and received most of the indirect supervision level ratings. We found a moderate, positive correlation between self-entrusted supervision levels at the end of the clerkship and the number of documented entrustment-supervision ratings per EPA (p < 0.0001). The number of entrustment ratings explained 6.5% of the variance in the supervisors’ ratings for EPA 1. Narrative feedback was documented for 79% (n = 214) of the WBAs. Most narratives addressed the Medical Expert role (77%, n = 208) and used reinforcement (59%, n = 161) as a feedback strategy. Students perceived the feedback as beneficial. Conclusions: Using formative WBAs with an entrustment-supervision scale and prompts for written feedback facilitated targeted, high-quality feedback and effectively supported students’ development toward self-entrusted, indirect supervision levels. Keywords: Entrustable professional activities, Entrustment, Workplace-based assessment, Undergraduate medical education, Clerkship * Correspondence: 1 University Hospital of Old Age Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland 2 Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland Full list of author information is available at the end of the article © The Author(s). 2021 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Pinilla et al. BMC Medical Education (2021) 21:223 Background Introducing entrustable professional activities (EPAs) to competency-based undergraduate medical education (UME) has led to new approaches for the design of workplace-based assessments (WBAs). EPAs are observable clinical tasks and serve as units of assessment that are often based on entrustment-supervision scales [1–6]. WBAs (i.e., any type of structured assessment done in the workplace such as the Mini-clinical exercise or clinical work sampling) serve multiple purposes [7, 8]. In a low-stakes context, they are intended to create opportunities for structured observation, feedback, and to support the achievement of competency-based learning goals (assessments for learning) [7, 9–11]. In the context of linking clinical UME and graduate medical education (GME) curricula [12, 13]— and short clerkship rotations—the emphasis should be on maximizing the value of formative WBAs, as described in GME [14]. This value depends on the context, content, and quality of the feedback resulting from the WBAs [15–21]. While a number of studies have explored the potential of WBAs for generating high-quality narrative feedback in GME [22–25], little is known about the relationship between WBAs based on an entrustment-scale, their narrative feedback output, and the perceived need for supervision (i.e., self-entrustment) in early-stage clinical students. In particular, changes in self-entrustment can be used as an indicator of self-efficacy [26–29]. Thus, developing higher levels of self-entrustment is relevant for selfregulated learning in clinical workplaces. WBAs have become a central part of many graduate training programs [20, 30–33] and are increasingly used to assess EPAs and competencies in undergraduate clinical training programs as well [2, 4, 34–37]. Typically, they are used to support the direct or indirect observation of trainees’ clinical activities and to provide assessment information for both low- and high-stakes purposes. Despite the potential of WBAs to provide formative feedback and their key role within assessment programs [8, 11], major feasibility issues have been identified in workplaces. These include a lack of understanding regarding the purpose of WBAs from both trainers and trainees, time constraints, and a lack of training within faculties [21, 32]. Duijn et al. [21] identified specific criteria for meaningful feedback taken from students’ perspectives on EPAs. These corresponded to the general feedback quality criteria described by Lefroy et al. [19], which included reinforcement, key point identification, strategy development, and whether feedback is actionable. The requirements and complexity involved in aligning valid learning goals such as EPAs, rating scales, and feedback narratives from WBAs has been explored in GME [22, 38] and UME [21, 34, 39]. However, we are Page 2 of 11 unaware of any studies that have addressed the content and quality of narrative feedback resulti (...truncated)