Pre-clerkship EPA assessments: a thematic analysis of rater cognition

(2022) 22:347 Meyer et al. BMC Medical Education https://doi.org/10.1186/s12909-022-03402-x Open Access RESEARCH Pre‑clerkship EPA assessments: a thematic analysis of rater cognition Eric G. Meyer1* , Emily Harvey2,3 , Steven J. Durning3   and Sebastian Uijtdehaage3    Abstract Background: Entrustable Professional Activities (EPAs) assessments measure learners’ competence with an entrustment or supervisory scale. Designed for workplace-based assessment EPA assessments have also been proposed for undergraduate medical education (UME), where assessments frequently occur outside the workplace and may be less intuitive, raising validity concerns. This study explored how assessors make entrustment determinations in UME, with additional specific comparison based on familiarity with prior performance in the context of longitudinal studentassessor relationships. Methods: A qualitative approach using think-alouds was employed. Assessors assessed two students (familiar and unfamiliar) completing a history and physical examination using a supervisory scale and then thought-aloud after each assessment. We conducted a thematic analysis of assessors’ response processes and compared them based on their familiarity with a student. Results: Four themes and fifteen subthemes were identified. The most prevalent theme related to “student performance.” The other three themes included “frame of reference,” “assessor uncertainty,” and “the patient.” “Previous student performance” and “affective reactions” were subthemes more likely to inform scoring when faculty were familiar with a student, while unfamiliar faculty were more likely to reference “self” and “lack confidence in their ability to assess.” Conclusions: Student performance appears to be assessors’ main consideration for all students, providing some validity evidence for the response process in EPA assessments. Several problematic themes could be addressed with faculty development while others appear to be inherent to entrustment and may be more challenging to mitigate. Differences based on assessor familiarity with student merits further research on how trust develops over time. Keywords: Competency based assessment, Entrustable professional activities, Rater cognition Introduction Competency-based, time-variable education is a hotly debated topic in medical education [1, 2]. In response to criticisms that current models for assessment in competency-based medical education (CBME) are too reductionist and onerous, [3, 4] ten Cate and Scheele introduced a synthetic assessment framework based *Correspondence: 1 Department of Psychiatry, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Rd, Bethesda, MD 20814, USA Full list of author information is available at the end of the article on entrustable professional activities (EPAs) that are assessed with trust [5]. EPAs are “professional activities that together constitute the mass of critical elements that operationally define a profession” [5]. An EPA assessment is operationalized by how much supervision an assessor believes the learner requires to safely execute the activity. Studies suggest that a single EPA assessment, combined with narrative feedback, can serve a formative purpose, helping a learner understand their current performance and driving improvement [6]. Furthermore, studies have found that a robust collection of EPA assessments for multiple tasks, in conjunction with other assessment © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativeco mmons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Meyer et al. BMC Medical Education (2022) 22:347 data, can be employed in a program of assessment [7] and inform high-stake advancement determinations [8]. EPA assessments are typically done in the clinical workplace and resemble the decisions supervisors make frequently regarding their trainees [9]. They are intended to align with how clinicians think, fit in to the daily work flow [10], and thus seem to ask “the right questions, in the right way, about the right things” [11]. EPA assessments are supposed to reflect a trainee’s ability to do a task, but it has been theorized that in workplace-based assessment (WBA) they are also influenced by factors beyond the control of the trainee: the characteristics of the assessor, the context in which the trainee was observed, the task itself, and the relationship between trainee and assessor [12, 13]. It has been further theorized that such trust itself develops overtime – from presumptive and initial trust to grounded trust [12, 13]. These characteristics not only re-enforce the complex nature of rater cognition [14], but also hint at the intricacies of trust, which has historically been conceptualized as a multidimensional concept [15, 16]—an intuition, and perhaps even a gut feeling [17]. While initially intended for residency training, there has also been interest in using EPAs in early medical training [18]. To that end, EPA tasks for entering clerkship [19] and an entrustment scale specific to undergraduate medical education (UME) have been developed [20], both of which were met with skepticism [21]. Moreover, given that opportunities for workplace-based assessments in the pre-clerkship phase of medical training are often simulated, infrequent, or absent altogether - it is unclear how assessors arrive at an entrustment rating in those circumstances. All of this raises questions about the validity evidence of the decisions that are based, in part, on EPA assessments. To assess the validity of an assessment, it has been proposed that a series of four step-wise inferences must be supported by evidence: scoring (translating an observation into a score or scores); generalization (interpreting the score or scores as a reflection of test performance); extrapolation (interpreting the score or scores as a reflection of real-world performance), and implications (making an advancement decision based on the score or scores) [22, 23]. The first inference, sco (...truncated)