Reference letters for subspecialty medicine residency positions: are they valuable for decision-making? Results from a Canadian study

Chopra et al. BMC Medical Education (2020) 20:350 https://doi.org/10.1186/s12909-020-02270-7 RESEARCH ARTICLE Open Access Reference letters for subspecialty medicine residency positions: are they valuable for decision-making? Results from a Canadian study Deepti Chopra1, Mala Joneja1, Gurjit Sandhu2, Christopher A. Smith1, Catherine M. Spagnuolo1 and Lawrence Hookey1,3* Abstract Background: The letter of recommendation is currently an integral part of applicant selection for residency programs. Internal medicine residents will spend much time and expense completing sub-specialty away electives to obtain a letter of recommendation. The purpose of this study was 1) to examine a large sample of reference letters in order to define essential components of a high-quality letter, and 2) to elucidate the relationship between quality of reference letter and the letter writer. Methods: We conducted a two-phase study. In phase one, a large sample of letters of recommendation was examined using an audit tool as a coding framework. A 5-point composite endpoint of high-quality letter components was subsequently developed. In phase two, program director letters were compared to non-program director home institution and non-home institution elective letters based on inclusion of components of the 5-point composite endpoint using Chi square testing. Results: 715 letters were examined (398 non-program director home institution letters, 201 program director letters, and 116 non-home institution elective letters). High-quality letter components were: nature of relationship, duration of relationship, In Training Evaluation Report information, research involvement and comments on areas for improvement. Program director letters had a significantly higher proportion (10.4%) of all 5 high-quality components, compared to 0% in both non-program director home institution letters and elective letters (p < 0.001). A significantly higher proportion of program director letters had 4–5 high-quality components (62.5%) compared to 2% of non-program director home institution letters and 0% of elective letters (p < 0.0001). Conclusions: Letters of recommendation from elective rotations are of the poorest quality and such rotations should not be pursued for the sole purpose of obtaining a letter. The low quality of elective letters leads to the recommendation that writers should decline to write them, programs should not require them and trainees should not request them. Program directors write the highest quality letters and should be a resource for faculty development. Clinical supervisors can use the 5-point composite endpoint as a guide when writing letters for applicants. Keywords: Reference letter, Letter of recommendation, Residents, Medicine subspecialty match * Correspondence: 1 Department of Medicine, Queen’s University, Kingston, ON, Canada 3 Division of Gastroenterology, Hotel Dieu Hospital, 166 Brock Street, Kingston, Ontario K7L 5G2, Canada Full list of author information is available at the end of the article © The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Chopra et al. BMC Medical Education (2020) 20:350 Background The Letter of Recommendation (LOR) is an integral part of applicant selection for internal medicine subspecialty training programs. LORs are considered one of the most important factors in ranking candidates to postgraduate subspecialty programs and may be helpful in predicting residency retention rates [1–5]. However, the writing of LORs has also been described as a process that very few understand [6] and the role of the LOR has been deemed worthy of further examination. LORs have been found to lack a meaningful comparison of the applicant to peers and often fail to include concrete examples demonstrating applicant performance [7]. LORs will often include “lengthy reiterations of already available objective data” and render all applicants “excellent,” making it extremely difficult to discriminate between them [8, 9]. In response to these noted shortcomings, program directors and medical educators are often asked to comment on what could make LORs more helpful. Researchers have examined, in a limited fashion, the role of the letter writer [10], key elements and phrases in letters and correlations between LORs with objective data [11, 12]. Our previous research which included a survey of Canadian internal medicine program directors has provided some insight into the challenges with letter writing and interpretation. Program directors (PDs) felt LORs lacked a common vocabulary and format and that referees often used variable rating scales or similar statements that did not differentiate between candidates. One example of this is the observation that a significant proportion of residents appeared to be in the top 5% of applicants [13]. The Canadian internal medicine PDs also suggested that there may be certain reference letter features that are more useful than others, and this led us to explore LORs further [13]. We partnered with the Canadian Resident Matching Service (CaRMS) to evaluate LORs for internal medicine residents applying to subspecialty training programs. The study objectives were to: 1a) elicit the key components of letters which affect quality through a large audit of LORs to subspecialty internal medicine programs in Canada between 2011 and 2014 and b) subsequently define the components of a high-quality letter; and 2) compare PD letters to non-PD home institution and non-home institution elective letters (elective letters) based on the inclusion of high-quality components. Methods Ethics approval was obtained from the Queen’s University Health Sciences & Affiliated Teaching Hospitals Research Ethics Board and the Canadian Resident Matching Service (CaRMS) Board. This study was designed in two phases. Page 2 of 7 Phase 1: creation of composite endpoint for a highquality LOR The objective of phase one was to identify common components (...truncated)