How are "teaching the teachers" courses in evidence based medicine evaluated? A systematic review
BMC Medical Education
How are teaching the teachers courses in evidence based medicine evaluated? A systematic review
Theodoros N Arvanitis
Andrea R Horvath
Katrien Oude Rengerink
Ben WJ Mol
Khalid S Khan
Background: Teaching of evidence-based medicine (EBM) has become widespread in medical education. Teaching the teachers (TTT) courses address the increased teaching demand and the need to improve effectiveness of EBM teaching. We conducted a systematic review of assessment tools for EBM TTT courses. To summarise and appraise existing assessment methods for teaching the teachers courses in EBM by a systematic review. Methods: We searched PubMed, BioMed, EmBase, Cochrane and Eric databases without language restrictions and included articles that assessed its participants. Study selection and data extraction were conducted independently by two reviewers. Results: Of 1230 potentially relevant studies, five papers met the selection criteria. There were no specific assessment tools for evaluating effectiveness of EBM TTT courses. Some of the material available might be useful in initiating the development of such an assessment tool. Conclusion: There is a need for the development of educationally sound assessment tools for teaching the teachers courses in EBM, without which it would be impossible to ascertain if such courses have the desired effect.
Evidence-based medicine (EBM), is defined as the
integration of the best research evidence with patients
values and clinical circumstances for clinical decision
making . However, for EBM to become a reality for
practice, studies have shown that it needs to be taught
by competent clinical trainers, who help trainees to
learn efficiently by exploiting on-the-job learning
opportunities . Trainers who have received formal EBM
training are more likely to teach application of EBM in
a clinical setting . Teaching EBM should not only
equip practitioners with knowledge and skills but also
foster their attitudes and encourage the practice of
EBM. Critical appraisal and EBM teaching that is
integrated into clinical practice appears to be effective in
1CASPolska, 32-400 Mylenice, ul. Mickiewicza 40, Poland
Full list of author information is available at the end of the article
improving substantial outcomes, including behavioural
changes . One of the objectives of TTT courses
should be to bring about a positive change in the
teachers knowledge, skills and attitude towards EBM
teaching in a clinical setting. For professional
development in this field, many trainers attend teaching the
teachers (TTT) courses for EBM, which are widespread.
The Leonardo da Vinci programme, part of the
European Commissions Lifelong Learning Programme, has
funded a EU project involving collaboration of 10
partners within Europe to design an e-learning curriculum
for continuing professional development (CPD) on how
to teach EBM in a clinical setting http://www.ebm-unity.
org[5,6]. Through this project, we have developed a
European qualification in Teaching Evidence-Based
Medicine which aims to improve transparency across the
European healthcare sector . Often teachers assess
their own learning through their students feedback.
However, this may be naive as it is well known that
there is no direct relationship between feedback and
learning achievement [8,9]. The use of assessment tools
if any to evaluate such courses is unknown. There are
currently many courses that train the trainer in teaching
EBM skills. A review of assessment tools used for such
courses can help us to gauge how TTT courses are
evaluated. We aimed to systematically review existing
assessment tools used to evaluate TTT courses in EBM.
The systematic review has been conducted by a well
designed protocol using robust methodology [10,11] to
allow us to identify educationally sound assessments of
EBM TTT courses.
We searched the following databases from database
inception to November 2008: PubMed, EmBase,
BioMed, Cochrane Library and Eric (Education
Resources Information Centre, a collection of
bibliographic records of education literature). Search
strategies for electronic databases were developed based on
the review of known relevant literature, and combined
MeSH and free text terms, as shown in Additional File
1. We handsearched relevant papers identified through
the reference list of retrieved articles, examined our
Consortiums personal files, explored for related articles
in Pubmed and searched citations in Google Scholar.
No language restrictions were applied. Target audiences
of the course (teachers) were defined as any health care
professional with teaching responsibilities. The
definition of a TTT course was based on three characteristics:
the aim of the course, the type of participants, and the
presence of an assessment strategy. We selected the
courses that were related to learning teaching skills in
EBM. We included papers that reported development,
validation or implementation of any assessment tool or
methods to evaluate an EBM TTT course. Due to the
specificity of the subject matter we did not use
restrictive inclusion criteria related to study quality. All titles
and abstracts were screened by two reviewers for studies
that met the inclusion criteria. Any disagreement
between reviewers was resolved by consensus. We
extracted data on the course objectives, duration, target
audience, teaching methods, assessment strategy and key
Our search identified 1230 primary articles, of which
five were included in the review (Figure 1). The
summary of all TTT courses, which have been included in
this systematic review is shown in Additional File 2,
providing details of course objectives, target audience,
teaching methods, duration of course, assessment
methods and key outcomes.
The target audience of the EBM TTT courses
identified in the review were faculty members in family
practice residences who help residents develop skills in the
use of informatics and evidence based medicine ,
podiatric medical educators , clinical teachers, and
librarians [14-16]. Diverse teaching methods such as
instructional workshops, facilitated discussions,
handson exercises and small group sessions were used in the
courses. We did not identify a clinically integrated
course teaching application of EBM in routine clinical
practice. The total duration of these courses varied from
1 day to 24 months.
The courses used various assessment tools
independent of the teaching method to evaluate the
effectiveness of the course. The most popular method of course
assessment was pre-and post-course appraisal as
observed in three of the five chosen articles [12,13,15].
One course surveyed the participants only once, after
the completetion of the course. The most detailed
assessment method to study the effectiveness of the
course was by Scherrer et al . It was a collaboration
between librarians and medical faculty to train librarians
and health professionals to teach principles of EBM.
EBM principles and skills were presented and
appropriate teaching methods and techniques were considered.
As a method of evaluation, a questionnaire with twenty
questions was created and designed to be completed by
a twenty-minute telephone interview. After a pilot
project, the final survey contained twenty-three questions
that took about fifteen minutes to complete.
Respondents were asked to state if the course resulted in
increased teaching of EBM concepts by the participants
and methods used when they returned to their
institutions. It explained the influence of the course on the
participants in teaching EBM principles, their
understanding of EBM principles and confidence in teaching
EBM concepts. One of the questions aimed to identify
the barriers faced by the participants in teaching EBM.
The components of this questionnaire may constitute a
basis for the development of an assessment tool to
evaluate teaching the teachers courses for EBM .
This review has revealed variations in the content of
EBM courses, their duration, methods of teaching and
course assessment. The reviewers have identified a
substantial gap in assessment methods for the evaluation of
EBM TTT courses. Very few studies in this field had
any detail of course assessments. Existing assessment
methods are very few and of poor quality. Moreover the
identified assessment tools were intended for teacher
courses focused on courses conducted traditionally
(workshops, one-to-one consultation, lectures) rather
than clinically integrated courses.
Total number of citations identified by search in
Articles retrieved for detailed evaluation
Articles excluded after review of
citations n= 1196
Reasons for exclusion
Inappropriate target population n= 15
No assessment method described n=3
Not specific to EBM teaching n=8
Full text not available n=3
Articles included in the review n= 5
Figure 1 Flow chart of study selection articles included in the review.
Our review has been the first one to systematically
search for assessment tools for teaching the teachers
courses in EBM. It was systematically conducted with
rigorous methodology and has comprehensively
searched, identified and described existing tools for
assessing EBM TTT courses. The findings of the review
are limited by the small number of published reports on
the assessment methods used in this area. EBM can
only improve care if it is integrated in clinical practice
. This can be achieved by effective application of
EBM skills in the clinical environment.To evaluate
attainment of this objective teaching the teachers
courses need to employ appropriate assessment tools.
The TTT EBM curricula is widely based on the model
developed in McMaster University, Sacketts books on
how to teach EBM and Users guide to medical
literature . But little seems to be reported or known
about EBM teaching evaluation tools, with much of this
knowledge observational .
Until now several tools for assessment are known to
be used in different kinds of courses: Research
SelfEfficacy Scale (RSES), the stages of change (SOC) model
and Kirkpatricks model [9,20,21]. RSES scale has good
internal consistency but it has been designed to assess
research self-confidence. Thus it lacks face validity for
teaching the teachers courses which focus on helping
with EBM teaching not on conducting or teaching how
to research. The SOC model suffers the same deficiency
as it examines learners attitudes, intentions and actions
towards research, not EBM teaching. Moreover, self
assessments of this type suffer in the area of criterion
validity as there is a loose link between self-perception
and objective assessment . The most commonly
known tool is the Kirkpatricks model, which is used
frequently as a tool for assessing individual trainee
performance but also for measuring the effectiveness of
teaching the teacher courses [20,21]. Evaluation of
educational interventions must concern at least four
dimensions embedded in Kirkpatricks hierarchy, such as
participants satisfaction, learning (both knowledge and
skills), behavioral change (understood as a transfer of
knowledge and skills to workplace), and outcomes
which are mainly considered as impact on patients .
However none of included articles described and made
use of this method. Furthermore, change in knowledge
may not equate to change in behaviour.
Successful teaching depends not only on good
teaching, but also on willingness and effort by the learner, a
supportive social environment with teaching and
learning opportunities [8,23]. For evaluation of a teachers
performance, can we rely on assessments which utilise
trainees evaluations? Two reasons students feedback
is inappropriate as a measure of teacher quality are
that they may not measure all that the teachers are
trying to achieve and often do not also provide useful
information for teachers about what they need to
know to teach more effectively [8,24]. Moreover, data
from a randomised controlled trial suggests that
trainee feedback can be manipulated by introducing
seduction in teaching . For these reasons we
should evaluate trainers courses with specifically
developed assessment tools. To maintain relevance of
the course and to achieve its intended objectives,
assessment tools need to be designed as a part of an
ongoing evaluation cycle .
Effective teacher education is essential to improve
EBM teaching quality  and constitutes the main
reason for development of appropriate assessment tools.
Due of the lack of well developed assessment tools for
courses for EBM teachers, the effect of teaching skills of
tutors is currently not measurable. There is a need to
develop educationally sound assessment tools to
encourage EBM teachers to provide clinically integrated EBM
Additional file 1: Search strategy for identification of articles for the
review of assessment methods in EBM TTT courses.
Additional file 2: Details of primary articles included in the review
of assessment methods of EBM teaching the teachers courses
1. Shaneyfelt T , Baum KD , Bell D : Instruments for evaluating education in evidence-based practice: a systematic review . JAMA 2006 , 296 ( 9 ): 1116 - 1127 .
2. Coppus SF , Emparanza JI , Hadley J , Kulier R , Weinbrenner S , Arvanitis TN , Burls A , Cabello JB , Decsi T , Horvath AR , Kaczor M , Zanrei G , Pierer K , Stawiarz K , Kunz R , Mol BW , Khan KS : A clinically integrated curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU EBM project . BMC Medical Education 2007 , 7 : 46 .
3. Kljakovic M , Love T , Gilbert A : Attitudes of teachers to evidence based medicine . Australian Family Physician 2004 , 33 ( 5 ).
4. Coomarasamy A , Khan KS : What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review . BMJ 2004 , 329 : 1017 .
5. Kulier R , Hadley J , Weinbrenner S , Meyerrose B , Decsi T , Horvath AR , Nagy E , Emparanza JI , Coppus SF , Arvanitis TN , Burls A , Cabello JB , Kaczor M , Zanrei G , Pierer K , Stawiarz K , Kunz R , Mol BW , Khan KS : Harmonising evidence-based medicine teaching: a study of the outcomes of elearning in five European countries . BMC Med Edu 2008 , 8 : 27 .
6. European Union Evidence Based Medicine Unity . [http://www.ebm-unity. org].
7. Thangaratinam S , Barnfield G , Weinbrenner S , Meyerrose B , Arvanitis TN , Horvath AR , Zanrei G , Kunz R , Suter K , Walczak J , Kaleta A , Oude Rengerink K , Gee H , Mol BW , Khan KS : Teaching trainers to incorporate evidence-based medicine (EBM) teaching in clinical practice: the EU-EBM project . BMC Med Edu 2009 , 9 : 59 .
8. Ingvarson L , Rowe K : Conceptualising and evaluating teacher quality: substantive and methodological issues . The Economics of Teacher Quality conference, Australian National University 2007 .
9. Bates I , Ansong D , Bedu-Addo G , Agbenyega T , Akoto AY , Nsiah-Asare A , Karikari P : Evaluation of a learner-designed course for teaching health research skills in Ghana . BMC Medical Education 2007 , 7 : 18 .
10. Khan KS , ter Riet G , Popay J , Glanville J , Sowden AJ , Kleijnen J: (2001b) Study quality assessment: conducting the review . Undertaking Systematic Reviews of Research on Effectiveness CRD Report No . 4 , Phase 5, Stage II 1- 20 .
11. Deeks JJ : Systematic reviews in health care: Systematic reviews of evaluations of diagnostic and screening tests . Br Med J 2001 , 323 : 157 - 162 .
12. Cartwright CA , Korsen N , Urbach LE : Teaching the teachers: helping faculty in a family practice residency improve their informatics skills . Acad Med 2002 , 77 : 385 - 91 .
13. Green ML : A train-the-trainer model for integrating evidence-based medicine training into podiatric medical education . Journal of the American Podiatric Medical Association 2005 , 95 : 497 - 504 .
14. Dorsch JL , Jacobson S , Scherrer CS : Teaching EBM teachers: a team approach . Medical Reference Services Quarterly 2003 , 22 .
15. Kouhpayehzadeh J , Baradaran H , Soltani Arabshahi K , Knill-Jones R : Clinical teachers' attitudes toward the efficacy of evidence-based medicine workshop and self-reported ability in evidence-based practice in Iran . The Journal of Continuing Education in the Health Professions 2006 , 26 : 210 - 214 .
16. Scherrer CS : An evaluation of a collaborative model for preparing evidence-based medicine teachers . J Med Libr Assoc l 2006 , 94 ( 2 ).
17. Fritsche L , Greenhalgh T , Falck-Ytter Y , Neumayer H-H , Kunz R : Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine . BMJ 2002 , 325 : 1338 - 1341 .
18. Yousefi-Nooraie R , Rashhidian A , Keating JL , Schonstein E : Teaching evidence-based practice: the teachers consider the content . J Eval Clin Pract 2007 , 13 : 569 - 75 .
19. Aiyer MK , Dorsch : The transformation of an EBM curriculum: a 10-year experience . Medical Teacher 2008 , 30 : 377 - 83 .
20. Ribeiro O : Developing criteria for assessing the impact of a trainer training program in a health care setting: from theory to practice . American Society for Training & Development 2006 .
21. Hutchinson L : Evaluating and researching the effectiveness of educational interventions . BMJ 1999 , 318 : 1267 - 1269 .
22. Khan KS , Awonuga AO , Dwarakanath LS , Taylor R : Assessments in evidence-based medicine workshops: Loose connection between perception of knowledge and its objective assessment . Medical Teacher 2001 , 23 : 92 - 94 .
23. Fenstermacher GD , Richardson V : On making determinations of quality in teaching . Teachers College Record 2005 , 107 : 186 - 213 .
24. Darling-Hammond L , Youngs P : Defining highly qualified teachers": What does scientifically-based research actually tell us? Educational Researcher 2002 , 9 : 13 - 25 .
25. Naftulin DH , Ware JE , Donnelly FA : The doctor Fox lecture: a paradigm of educational seduction . Journal of Medicine Education 1973 , 48 : 630 - 635 .
26. Wilkes M , Bligh J : Evaluating educational interventions . BMJ 1999 , 318 .