How are "teaching the teachers" courses in evidence based medicine evaluated? A systematic review

BMC Medical Education, Sep 2010

Background Teaching of evidence-based medicine (EBM) has become widespread in medical education. Teaching the teachers (TTT) courses address the increased teaching demand and the need to improve effectiveness of EBM teaching. We conducted a systematic review of assessment tools for EBM TTT courses. To summarise and appraise existing assessment methods for teaching the teachers courses in EBM by a systematic review. Methods We searched PubMed, BioMed, EmBase, Cochrane and Eric databases without language restrictions and included articles that assessed its participants. Study selection and data extraction were conducted independently by two reviewers. Results Of 1230 potentially relevant studies, five papers met the selection criteria. There were no specific assessment tools for evaluating effectiveness of EBM TTT courses. Some of the material available might be useful in initiating the development of such an assessment tool. Conclusion There is a need for the development of educationally sound assessment tools for teaching the teachers courses in EBM, without which it would be impossible to ascertain if such courses have the desired effect.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

http://www.biomedcentral.com/content/pdf/1472-6920-10-64.pdf

How are "teaching the teachers" courses in evidence based medicine evaluated? A systematic review

BMC Medical Education How are teaching the teachers courses in evidence based medicine evaluated? A systematic review Jacek Walczak Anna Kaleta Elbieta Gabry Krzysztof Kloc Shakila Thangaratinam Gemma Barnfield Susanne Weinbrenner Berit Meyerrose Theodoros N Arvanitis Andrea R Horvath Gianni Zanrei Regina Kunz Katja Suter Bernard Burnand Chantal Arditi Katrien Oude Rengerink Gee Harry Ben WJ Mol Khalid S Khan Background: Teaching of evidence-based medicine (EBM) has become widespread in medical education. Teaching the teachers (TTT) courses address the increased teaching demand and the need to improve effectiveness of EBM teaching. We conducted a systematic review of assessment tools for EBM TTT courses. To summarise and appraise existing assessment methods for teaching the teachers courses in EBM by a systematic review. Methods: We searched PubMed, BioMed, EmBase, Cochrane and Eric databases without language restrictions and included articles that assessed its participants. Study selection and data extraction were conducted independently by two reviewers. Results: Of 1230 potentially relevant studies, five papers met the selection criteria. There were no specific assessment tools for evaluating effectiveness of EBM TTT courses. Some of the material available might be useful in initiating the development of such an assessment tool. Conclusion: There is a need for the development of educationally sound assessment tools for teaching the teachers courses in EBM, without which it would be impossible to ascertain if such courses have the desired effect. - Background Evidence-based medicine (EBM), is defined as the integration of the best research evidence with patients values and clinical circumstances for clinical decision making [1]. However, for EBM to become a reality for practice, studies have shown that it needs to be taught by competent clinical trainers, who help trainees to learn efficiently by exploiting on-the-job learning opportunities [2]. Trainers who have received formal EBM training are more likely to teach application of EBM in a clinical setting [3]. Teaching EBM should not only equip practitioners with knowledge and skills but also foster their attitudes and encourage the practice of EBM. Critical appraisal and EBM teaching that is integrated into clinical practice appears to be effective in * Correspondence: 1CASPolska, 32-400 Mylenice, ul. Mickiewicza 40, Poland Full list of author information is available at the end of the article improving substantial outcomes, including behavioural changes [4]. One of the objectives of TTT courses should be to bring about a positive change in the teachers knowledge, skills and attitude towards EBM teaching in a clinical setting. For professional development in this field, many trainers attend teaching the teachers (TTT) courses for EBM, which are widespread. The Leonardo da Vinci programme, part of the European Commissions Lifelong Learning Programme, has funded a EU project involving collaboration of 10 partners within Europe to design an e-learning curriculum for continuing professional development (CPD) on how to teach EBM in a clinical setting http://www.ebm-unity. org[5,6]. Through this project, we have developed a European qualification in Teaching Evidence-Based Medicine which aims to improve transparency across the European healthcare sector [7]. Often teachers assess their own learning through their students feedback. However, this may be naive as it is well known that there is no direct relationship between feedback and learning achievement [8,9]. The use of assessment tools if any to evaluate such courses is unknown. There are currently many courses that train the trainer in teaching EBM skills. A review of assessment tools used for such courses can help us to gauge how TTT courses are evaluated. We aimed to systematically review existing assessment tools used to evaluate TTT courses in EBM. Methods The systematic review has been conducted by a well designed protocol using robust methodology [10,11] to allow us to identify educationally sound assessments of EBM TTT courses. We searched the following databases from database inception to November 2008: PubMed, EmBase, BioMed, Cochrane Library and Eric (Education Resources Information Centre, a collection of bibliographic records of education literature). Search strategies for electronic databases were developed based on the review of known relevant literature, and combined MeSH and free text terms, as shown in Additional File 1. We handsearched relevant papers identified through the reference list of retrieved articles, examined our Consortiums personal files, explored for related articles in Pubmed and searched citations in Google Scholar. No language restrictions were applied. Target audiences of the course (teachers) were defined as any health care professional with teaching responsibilities. The definition of a TTT course was based on three characteristics: the aim of the course, the type of participants, and the presence of an assessment strategy. We selected the courses that were related to learning teaching skills in EBM. We included papers that reported development, validation or implementation of any assessment tool or methods to evaluate an EBM TTT course. Due to the specificity of the subject matter we did not use restrictive inclusion criteria related to study quality. All titles and abstracts were screened by two reviewers for studies that met the inclusion criteria. Any disagreement between reviewers was resolved by consensus. We extracted data on the course objectives, duration, target audience, teaching methods, assessment strategy and key outcomes. Results Our search identified 1230 primary articles, of which five were included in the review (Figure 1). The summary of all TTT courses, which have been included in this systematic review is shown in Additional File 2, providing details of course objectives, target audience, teaching methods, duration of course, assessment methods and key outcomes. The target audience of the EBM TTT courses identified in the review were faculty members in family practice residences who help residents develop skills in the use of informatics and evidence based medicine [12], podiatric medical educators [13], clinical teachers, and librarians [14-16]. Diverse teaching methods such as instructional workshops, facilitated discussions, handson exercises and small group sessions were used in the courses. We did not identify a clinically integrated course teaching application of EBM in routine clinical practice. The total duration of these courses varied from 1 day to 24 months. The courses used various assessment tools independent of the teaching method to evaluate the effectiveness of the course. The most popular method of course assessment was pre-and post-course appraisal as observed in three of the five chosen articles [12,13,15]. One course surveyed the participants only once, after the completetion of the course. The most detailed assessment method to study the effectiveness of the course was by Scherrer et al [16]. It was a collaboration between librarians and medical faculty to train librarians and health professionals to teach principles of EBM. EBM principles and skills were presented and appropriate teaching methods and techniques were considered. As a method of evaluation, a questionnaire with twenty questions was created and designed to be completed by a twenty-minute telephone interview. After a pilot project, the final survey contained twenty-three questions that took about fifteen minutes to complete. Respondents were asked to state if the course resulted in increased teaching of EBM concepts by the participants and methods used when they returned to their institutions. It explained the influence of the course on the participants in teaching EBM principles, their understanding of EBM principles and confidence in teaching EBM concepts. One of the questions aimed to identify the barriers faced by the participants in teaching EBM. The components of this questionnaire may constitute a basis for the development of an assessment tool to evaluate teaching the teachers courses for EBM [16]. Discussion This review has revealed variations in the content of EBM courses, their duration, methods of teaching and course assessment. The reviewers have identified a substantial gap in assessment methods for the evaluation of EBM TTT courses. Very few studies in this field had any detail of course assessments. Existing assessment methods are very few and of poor quality. Moreover the identified assessment tools were intended for teacher courses focused on courses conducted traditionally (workshops, one-to-one consultation, lectures) rather than clinically integrated courses. Total number of citations identified by search in electronic databases Articles retrieved for detailed evaluation n= 34 Articles excluded after review of citations n= 1196 Reasons for exclusion Inappropriate target population n= 15 No assessment method described n=3 Not specific to EBM teaching n=8 Full text not available n=3 Articles included in the review n= 5 Figure 1 Flow chart of study selection articles included in the review. Our review has been the first one to systematically search for assessment tools for teaching the teachers courses in EBM. It was systematically conducted with rigorous methodology and has comprehensively searched, identified and described existing tools for assessing EBM TTT courses. The findings of the review are limited by the small number of published reports on the assessment methods used in this area. EBM can only improve care if it is integrated in clinical practice [17]. This can be achieved by effective application of EBM skills in the clinical environment.To evaluate attainment of this objective teaching the teachers courses need to employ appropriate assessment tools. The TTT EBM curricula is widely based on the model developed in McMaster University, Sacketts books on how to teach EBM and Users guide to medical literature [18]. But little seems to be reported or known about EBM teaching evaluation tools, with much of this knowledge observational [19]. Until now several tools for assessment are known to be used in different kinds of courses: Research SelfEfficacy Scale (RSES), the stages of change (SOC) model and Kirkpatricks model [9,20,21]. RSES scale has good internal consistency but it has been designed to assess research self-confidence. Thus it lacks face validity for teaching the teachers courses which focus on helping with EBM teaching not on conducting or teaching how to research. The SOC model suffers the same deficiency as it examines learners attitudes, intentions and actions towards research, not EBM teaching. Moreover, self assessments of this type suffer in the area of criterion validity as there is a loose link between self-perception and objective assessment [22]. The most commonly known tool is the Kirkpatricks model, which is used frequently as a tool for assessing individual trainee performance but also for measuring the effectiveness of teaching the teacher courses [20,21]. Evaluation of educational interventions must concern at least four dimensions embedded in Kirkpatricks hierarchy, such as participants satisfaction, learning (both knowledge and skills), behavioral change (understood as a transfer of knowledge and skills to workplace), and outcomes which are mainly considered as impact on patients [17]. However none of included articles described and made use of this method. Furthermore, change in knowledge may not equate to change in behaviour. Successful teaching depends not only on good teaching, but also on willingness and effort by the learner, a supportive social environment with teaching and learning opportunities [8,23]. For evaluation of a teachers performance, can we rely on assessments which utilise trainees evaluations? Two reasons students feedback is inappropriate as a measure of teacher quality are that they may not measure all that the teachers are trying to achieve and often do not also provide useful information for teachers about what they need to know to teach more effectively [8,24]. Moreover, data from a randomised controlled trial suggests that trainee feedback can be manipulated by introducing seduction in teaching [25]. For these reasons we should evaluate trainers courses with specifically developed assessment tools. To maintain relevance of the course and to achieve its intended objectives, assessment tools need to be designed as a part of an ongoing evaluation cycle [26]. Effective teacher education is essential to improve EBM teaching quality [8] and constitutes the main reason for development of appropriate assessment tools. Conclusion Due of the lack of well developed assessment tools for courses for EBM teachers, the effect of teaching skills of tutors is currently not measurable. There is a need to develop educationally sound assessment tools to encourage EBM teachers to provide clinically integrated EBM teaching. Additional file 1: Search strategy for identification of articles for the review of assessment methods in EBM TTT courses. Additional file 2: Details of primary articles included in the review of assessment methods of EBM teaching the teachers courses 1. Shaneyfelt T , Baum KD , Bell D : Instruments for evaluating education in evidence-based practice: a systematic review . JAMA 2006 , 296 ( 9 ): 1116 - 1127 . 2. Coppus SF , Emparanza JI , Hadley J , Kulier R , Weinbrenner S , Arvanitis TN , Burls A , Cabello JB , Decsi T , Horvath AR , Kaczor M , Zanrei G , Pierer K , Stawiarz K , Kunz R , Mol BW , Khan KS : A clinically integrated curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU EBM project . BMC Medical Education 2007 , 7 : 46 . 3. Kljakovic M , Love T , Gilbert A : Attitudes of teachers to evidence based medicine . Australian Family Physician 2004 , 33 ( 5 ). 4. Coomarasamy A , Khan KS : What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review . BMJ 2004 , 329 : 1017 . 5. Kulier R , Hadley J , Weinbrenner S , Meyerrose B , Decsi T , Horvath AR , Nagy E , Emparanza JI , Coppus SF , Arvanitis TN , Burls A , Cabello JB , Kaczor M , Zanrei G , Pierer K , Stawiarz K , Kunz R , Mol BW , Khan KS : Harmonising evidence-based medicine teaching: a study of the outcomes of elearning in five European countries . BMC Med Edu 2008 , 8 : 27 . 6. European Union Evidence Based Medicine Unity . [http://www.ebm-unity. org]. 7. Thangaratinam S , Barnfield G , Weinbrenner S , Meyerrose B , Arvanitis TN , Horvath AR , Zanrei G , Kunz R , Suter K , Walczak J , Kaleta A , Oude Rengerink K , Gee H , Mol BW , Khan KS : Teaching trainers to incorporate evidence-based medicine (EBM) teaching in clinical practice: the EU-EBM project . BMC Med Edu 2009 , 9 : 59 . 8. Ingvarson L , Rowe K : Conceptualising and evaluating teacher quality: substantive and methodological issues . The Economics of Teacher Quality conference, Australian National University 2007 . 9. Bates I , Ansong D , Bedu-Addo G , Agbenyega T , Akoto AY , Nsiah-Asare A , Karikari P : Evaluation of a learner-designed course for teaching health research skills in Ghana . BMC Medical Education 2007 , 7 : 18 . 10. Khan KS , ter Riet G , Popay J , Glanville J , Sowden AJ , Kleijnen J: (2001b) Study quality assessment: conducting the review . Undertaking Systematic Reviews of Research on Effectiveness CRD Report No . 4 , Phase 5, Stage II 1- 20 . 11. Deeks JJ : Systematic reviews in health care: Systematic reviews of evaluations of diagnostic and screening tests . Br Med J 2001 , 323 : 157 - 162 . 12. Cartwright CA , Korsen N , Urbach LE : Teaching the teachers: helping faculty in a family practice residency improve their informatics skills . Acad Med 2002 , 77 : 385 - 91 . 13. Green ML : A train-the-trainer model for integrating evidence-based medicine training into podiatric medical education . Journal of the American Podiatric Medical Association 2005 , 95 : 497 - 504 . 14. Dorsch JL , Jacobson S , Scherrer CS : Teaching EBM teachers: a team approach . Medical Reference Services Quarterly 2003 , 22 . 15. Kouhpayehzadeh J , Baradaran H , Soltani Arabshahi K , Knill-Jones R : Clinical teachers' attitudes toward the efficacy of evidence-based medicine workshop and self-reported ability in evidence-based practice in Iran . The Journal of Continuing Education in the Health Professions 2006 , 26 : 210 - 214 . 16. Scherrer CS : An evaluation of a collaborative model for preparing evidence-based medicine teachers . J Med Libr Assoc l 2006 , 94 ( 2 ). 17. Fritsche L , Greenhalgh T , Falck-Ytter Y , Neumayer H-H , Kunz R : Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine . BMJ 2002 , 325 : 1338 - 1341 . 18. Yousefi-Nooraie R , Rashhidian A , Keating JL , Schonstein E : Teaching evidence-based practice: the teachers consider the content . J Eval Clin Pract 2007 , 13 : 569 - 75 . 19. Aiyer MK , Dorsch : The transformation of an EBM curriculum: a 10-year experience . Medical Teacher 2008 , 30 : 377 - 83 . 20. Ribeiro O : Developing criteria for assessing the impact of a trainer training program in a health care setting: from theory to practice . American Society for Training & Development 2006 . 21. Hutchinson L : Evaluating and researching the effectiveness of educational interventions . BMJ 1999 , 318 : 1267 - 1269 . 22. Khan KS , Awonuga AO , Dwarakanath LS , Taylor R : Assessments in evidence-based medicine workshops: Loose connection between perception of knowledge and its objective assessment . Medical Teacher 2001 , 23 : 92 - 94 . 23. Fenstermacher GD , Richardson V : On making determinations of quality in teaching . Teachers College Record 2005 , 107 : 186 - 213 . 24. Darling-Hammond L , Youngs P : Defining highly qualified teachers": What does scientifically-based research actually tell us? Educational Researcher 2002 , 9 : 13 - 25 . 25. Naftulin DH , Ware JE , Donnelly FA : The doctor Fox lecture: a paradigm of educational seduction . Journal of Medicine Education 1973 , 48 : 630 - 635 . 26. Wilkes M , Bligh J : Evaluating educational interventions . BMJ 1999 , 318 .


This is a preview of a remote PDF: http://www.biomedcentral.com/content/pdf/1472-6920-10-64.pdf

Jacek Walczak, Anna Kaleta, Elżbieta Gabryś, Krzysztof Kloc, Shakila Thangaratinam, Gemma Barnfield, Susanne Weinbrenner, Berit Meyerrose, Theodoros N Arvanitis, Andrea R Horvath, Gianni Zanrei, Regina Kunz, Katja Suter, Bernard Burnand, Chantal Arditi, Katrien Oude Rengerink, Gee Harry, Ben WJ Mol, Khalid S Khan. How are "teaching the teachers" courses in evidence based medicine evaluated? A systematic review, BMC Medical Education, 2010, 64, DOI: 10.1186/1472-6920-10-64