Establishing a library of resources to help people understand key concepts in assessing treatment claims—The “Critical thinking and Appraisal Resource Library” (CARL)
Establishing a library of resources to help people understand key concepts in assessing treatment claimsÐThe ªCritical thinking and Appraisal Resource Libraryº (CARL)
0 1 James Lind Initiative , Oxford , United Kingdom , 2 Minervation Ltd., Oxford, United Kingdom, 3 Global Health Unit, Institute of Public Health , Oslo, Norway, 4 Høgskulen på Vestlandet , Centre for Evidence-Based practice , Bergen , Norway , 5 Global Development Program, Bill & Melinda Gates Foundation, Seattle, Washington, United States of America, 6 Medicine in the Media Program, The Dartmouth Institute , Hanover , New Hampshire, United States of America, 7 School of Health Sciences, City University London , London , United Kingdom , 8 Laboratorio di ricerca sul coinvolgimento dei cittadini in sanità, Istituto di Ricerche Farmacologiche Mario Negri , Milano , Italy , 9 Centre for Research in Evidence-Based Practice, Faculty of Health Sciences and Medicine Bond University , Queensland , Australia
1 National Institute for Health Research , UK
2 Editor: Iratxe Puebla, Public Library of Science , FRANCE
People are frequently confronted with untrustworthy claims about the effects of treatments.
Uncritical acceptance of these claims can lead to poor, and sometimes dangerous,
treatment decisions, and wasted time and money. Resources to help people learn to think
critically about treatment claims are scarce, and they are widely scattered. Furthermore,
very few learning-resources have been assessed to see if they improve knowledge and
Our objectives were to develop the Critical thinking and Appraisal Resource Library (CARL).
This library was to be in the form of a database containing learning resources for those
who are responsible for encouraging critical thinking about treatment claims, and was to
be made available online. We wished to include resources for groups we identified as
`intermediaries' of knowledge, i.e. teachers of schoolchildren, undergraduates and
graduates, for example those teaching evidence-based medicine, or those communicating
treatment claims to the public. In selecting resources, we wished to draw particular attention
to those resources that had been formally evaluated, for example, by the creators of the
resource or independent research groups.
(https://www.nihr.ac.uk/), by IC. The funders had
no role in the study design, data collection or
analysis, decision to publish or preparation of the
manuscript. Funding was also received from
Minervation Ltd. In the form of author's salary for
Douglas Badenoch. Minervation Ltd is a consulting
contractor providing editorial, project management,
web design and technical development services to
the Testing Treatments project. There were no
competing or commercial interests.
Competing interests: The author Douglas
Badenoch is affiliated with Minervation Ltd.
Minervation Ltd is a consulting contractor
providing editorial, project management, design
and technical development services to the Testing
Treatments project. This does not alter our
adherence to PLOS ONE policies on sharing data
CARL was populated with learning-resources identified from a variety of sourcesÐtwo
previously developed but unmaintained inventories; systematic reviews of learning-interventions;
online and database searches; and recommendations by members of the project group and
its advisors. The learning-resources in CARL were organised by `Key Concepts' needed to
judge the trustworthiness of treatment claims, and were made available online by the James
Lind Initiative in Testing Treatments interactive (TTi) English (www.testingtreatments.org/
category/learning-resources).TTi English also incorporated the database of Key Concepts
and the Claim Evaluation Tools developed through the Informed Healthcare Choices (IHC)
We have created a database of resources called CARL, which currently contains over 500
open-access learning-resources in a variety of formats: text, audio, video, webpages,
cartoons, and lesson materials. These are aimed primarily at `Intermediaries', that is,
`teachers', `communicators', `advisors', `researchers', as well as for independent `learners'.
The resources included in CARL are currently accessible at www.testingtreatments.org/
We hope that ready access to CARL will help to promote the critical thinking about treatment
claims, needed to help improve healthcare choices.
People are confronted every day by claims about the effects of treatments. Many, if not most,
of these claims are unsupported by evidence , meaning they can put patients at risk of
harm. Where claims seek to sell a treatment, patients may spend their own money on
treatments of no known benefit, or may seek inappropriate treatments, which can waste public
resources on a large scale . Additionally, misleading claims can exacerbate peoples' natural
tendency to overestimate the benefits of treatments and to underestimate their potential risks
. This might lead patients to seek inadequately evaluated treatments with unrecognised
adverse effectsor avoid treatments likely to help them .
Similarly, health professionals may treat patients using methods that have been
insufficiently evaluated  or act based on comparisons which have not assessed treatments using
patient-valued outcomes. Professionals may also exploit treatment claims to fulfil their own,
conflicted, interests . Finally, as treatment claims often contradict each other and cause
confusion, patients may simply become disillusioned with, and lose respect for, the relevance
of research evidence .
To address these problems, people need to be able to think critically about the treatment
claims they come across . They should be able to assess the strength of the supporting
evidence underlying a treatment claim, the conflicts of interest among those making the claims,
and the relevance of the research evidence to their healthcare needs . These processes
2 / 12
Fig 1. Timeline demonstrating the stages that lead to the development of CARL. (A) Inventory by L Kendall Krause
created. (B) The James Lind Initiative begins adding resources to Testing Treatments interactive. (C) European Communication
on Research Awareness Needs (ECRAN) inventory created. (D) The Informed Healthcare Choices Project (IHC) formed,
Members begin to identify useful resources. (E) Discussions between IHC and JLI lead to expressions of support for a new
library, JLI agrees to coordinate development until 2019.
are part of a wider range of skills people should use to make well-reasoned decisions on a
Increased general knowledge about how to evaluate treatment claims would mean people
could more confidently use information about health claims to promote their own health 
]. It would mean that health professionals, together with their patients,
could make more balanced decisions [
]. It would also enable greater patient involvement in
shared-decision making [
], with fewer resources wasted and fewer avoidable harms.
Despite its importance, the promotion of critical thinkingÐthe use of logic and evidence to
assess the strength of the arguments that underlie claimsÐis not widespread. Furthermore, the
effects of learning-resources that aim to do this have very rarely been formally evaluated. This
lack of assessed learning-resources disadvantages people who wish to think for themselves
about treatment decisions. To address this, it would be valuable to collate and make accessible
an inventory of relevant, and where possible formally evaluated, existing learning-resources to
help people who are responsible for teaching others how to assess treatment claims.
There have been efforts since 2011 to create an inventory of learning-resources for this
] (Fig 1). In October 2015, in Vienna, the Informed Healthcare Choices (IHC)
project group and the James Lind Initiative (JLI) co-convened an international,
multidisciplinary workshop for people interested in helping others to make sense of treatment claims.
One of the presentations at the workshop was used to gauge support for a further attempt to
develop and maintain an inventory of learning-resources, as well as to inform the development
and evaluation of new resources. After written expressions of interest were submitted, it was
agreed that the JLI, which is funded by the National Institute for Health Research (NIHR),
would oversee the formation of such an inventory. Thus, we have created the Critical thinking
and Appraisal Resource Library (CARL).
In summary, our objectives were to develop the Critical thinking and Appraisal Resource
Library for those who are responsible for encouraging critical thinking about treatment
claims. These include: teachers of schoolchildren, undergraduates and graduates, for
example those teaching evidence-based medicine, or those communicating treatment claims
to the public. In selecting resources, we draw attention to those resources that have been
3 / 12
formally evaluated with pre-defined criteria, for instance by the resource creator or by
independent research groups.
Candidate learning-resources for possible inclusion in CARL have been identified in
previously compiled inventories [
] and through:
1. the Informed Health Choices project [www.informedhealthchoices.org]
2. Testing Treatments  and `Testing Treatments interactive' [www.testingtreatments.org]
3. the James Lind Library [
4. systematic reviews of educational interventions by Nordheim [
]; and Albarqouni (in preparation).
5. the Educational Endowment Foundation [https://educationendowmentfoundation.org.uk]
6. the Times Education Service. [https://www.tes.com/teaching-resources]
7. Online searches of databases such as ERIC, PubMed, and in search engines for resources
related to keywords.
When resources were identified, we made attempts to discover if the resource had been
formally evaluated to see if their intended learning objectives had been achieved. Where
formal evaluations were found, these were assessed for eligibility by pre-defined criteria [see
Development of CARL
Key concepts for organising resources in CARL. CARL is a database of learning
resources organized around a list of Key Concepts that people need to understand to assess
the trustworthiness of treatment claims, as developed by the IHC project[
testingtreatments.org/key-concepts-for-assessing-claims-about-treatment-effects) [see S1
Appendix]. Our definition of `treatments' includes any action intended to improve health or
relieve suffering. These include: changes in behaviour, screening programmes, drugs, surgery,
physical and psychological treatments; and public health and healthcare system changes.
The IHC Key Concepts have been largely derived from the content of Testing Treatments
, a book written for the public (currently in over a dozen languages; www.testingtreatments.
org), as well as from a variety of other resources with the same goal. Researchers and learners
in Norway, the UK, Uganda and Australia identified a list of over 30 Key Concepts, the process
of which has been described by Austvoll-Dahlgren et al. in 2015 [
We organised these concepts under three headings (see S1 Appendix):
Claims: are they justified? currently 12 Key Concepts
Comparisons: are they fair and reliable? currently 17 Key Concepts
Choices: are the findings relevant to you? currently 5 Key Concepts
As additional key concepts are identified and agreed, they will be added with explanations
Including and excluding resources. The principal eligibility assessor (JCC) judged whether
candidate resources should be included based on their relevance to the IHC Key Concepts. For
4 / 12
learning-resources of questionable eligibility, IC and other members of the CARL editorial
group also assessed the resources for inclusion. Resources to teach content specific to clinical
information, such as decision aids, have not been included. Given the varied formats of
resources, resource format-specific inclusion criteria were also agreed (see S2 Appendix).
Learning-resources that have been formally evaluated, for instance in randomised trials, are
of particular importance. As previously mentioned, separate inclusion criteria were agreed for
assessing the suitability of formal evaluations (S3 Appendix).
Coding resources. To facilitate navigation of CARL, each resource has been tagged with
categories deemed important by the editorial group:
1. Unique resource identification code.
2. Name/Title (as stated by the resource host)
3. Format: Text; Audio; Video; Cartoons; Websites/pages; and Lessons (including
presentations, e-Learning modules and specific materials for teaching students, such as learning
exercises or worksheets)
6. IHC Key Concepts to which the resource is relevant
7. Effects of a resource on knowledge/understanding, with links to reports of the evaluations.
Where reports are publications with restricted access, the JLI has summarised the findings
on the website www.testingtreatments.org.
8. Target user groups commonly mentioned by resource developers, categorized further for
· Teachers, including teachers of primary school, secondary school (coded as `School
Teachers'), and teachers of undergraduate and postgraduate health profession students (coded as
`Higher Level Teachers').
· Communicators, such as journalists and science writers.
· Advisors, for example, those who wish to help improve decision-making by policy makers,
or by members of research ethics committees.
· Researchers who wish to assess the effects of learning-resources designed to teach Key
Learners, anyone who wishes to teach themselves how to assess treatment claims, for
example, interested members of the public (coded as General Learners), and undergraduate,
postgraduate or professional students (coded as Higher Level learners).
Providing access to the resources in CARL
TTi (www.testingtreatments.org) is the first website providing access to the learning-resources
in the CARL database and we hope that the library may be used by other websites in the future.
The website has been redesigned to use the Key Concepts as the framework for organising its
content, specifically into three groups: `Claims: are they justified?'; `Comparisons: are they fair
and reliable?'; and `Choices: are the findings relevant?'.
Resources are accessed from the navigation menu, and content can be filtered according to
specific interests, such as Target Users, and Resource Format. Short descriptions of each
5 / 12
resource have also been added. Formally evaluated resources are listed higher in search results
and are clearly demarcated with a bold, green tick. In addition, an informal review process will
be implemented: users will be able to `like' or `recommend' resources, and those with the
highest number of likes will be ranked higher in the search results accordingly.
We have instituted an ongoing process to evaluate the website, including experimental
testing of functionality, presentation and content. A software platform has also been installed to
support randomised comparisons of alternative ways of promoting learning about the IHC
Key Concepts. The process and results of these evaluations will be described and submitted for
The website `Testing Treatments interactive' currently provides access to over 300
openaccess resources in CARL and more are being added following the same search and
screening processes described earlier. We expect this number to grow with periodic further
searches and through CARL users proposing additional resources using an online
The IHC Key Concepts that currently have the highest average number of resources are
those grouped under `Comparisons: are they fair and reliable?’. This contains IHC Key
Concepts relevant to randomised trials and systematic reviews, amongst other topics. The Key
Concept group with the lowest average number of resources per Key Concept is ‘Choices: are
the findings relevant?’, which contains IHC Key Concepts that address how users may apply
evidence and treatment claims in their own decisions. The average number of resources per
Key Concept may change as further searches are conducted, but it may also highlight which
areas are lacking, and therefore inform development of additional resources by interested
Of the resources that have been formally evaluated [
] the Key Concept group to which most are coded is `Comparisons: are
the results fair and reliable?' and this reflects that many of the resources currently available are
used for teaching evidence-based medicine. The group that has the fewest formally evaluated
resources is `Choices: are the findings relevant?’. This is likely to be because there are fewer Key
Concepts in this group. Work is ongoing to expand the number of Key Concepts in this
section. The most common formats of resources are Text and Audio, mostly derived from Testing
Treatments . The least common format of resources is currently Lessons. (Fig 2).
Specific IHC Key Concepts with relatively few resources include:
· Increasing the amount of a treatment does not necessarily increase the benefits of a treatment
and may cause harm
· Beliefs about how treatments work are not reliable predictors of the actual effects of treatments
· Treatment decisions should take account of both beneficial and harmful effects
· Average differences between treatments can be misleading
· The treatments evaluated in fair comparisons may not be relevant or applicable
· Results for a selected group of people within fair comparisons can be misleading
Additionally, relatively few resources for teachers of children and young people have been
identified, as well as few for `advisors', `communicators' and `researchers'.
The largest number of learning-resources is for `Learners', divided into: `General Learners'
i.e. those who wish to educate themselves about the Key Concepts, and `Higher Learners',
6 / 12
Fig 2. Graph to show the proportion of resources that are of each format in the CARL database. Vertical axis is percentage
of total resources of the specified target users. Horizontal represents the Key Concept group in question. (Blue) Text. (Red) Audio.
(Grey) Videos. (Yellow) Websites. (Purple) Cartoons. (Green) Lessons.
including undergraduate and postgraduate health profession students (see Fig 3). Many of the
learning-resources included in CARL had no specified Target User Group when originally
identified in searches. These have been coded to specific user groups after discussion. Mostly,
where resource target user groups were not specified, the resources were coded as for `General
Learners'. This is because they may be relevant to one or more of the `Intermediaries' listed
Fig 3. Graph to show the proportion of resources that are targeted at each Target User group in the CARL database by each
of the Key Concept groups. Vertical axis is percentage of resources in a group. Horizontal is the Key Concept group in question.
(Pink) Teachers of school age children. (Orange) Teachers of students in further education. (Light blue) Independent learners with
general interests. (Yellow) Higher level learners, such as those in further education or with higher level knowledge. (Red) Advisors,
e.g. policymakers. (Green) Communicators, e.g. journalists. (Purple) Researchers interested in developing educational resources.
7 / 12
above, or to anyone who wants to teach themselves how to use Key Concepts to assess the
validity of treatment claims.
The Critical thinking and Appraisal Resource LibraryÐCARLÐhas been created to help those
who teach people how to evaluate claims about the effects of treatments. This report describes
how CARL has been developed. We hope CARL will become an increasingly improved and
indispensable directory for teachers, communicators, advisors, researchers and learners.
Systematic and online searches for additional learning-resources relating to each Key
Concept will be conducted periodically. It is anticipated that additional resources will be
recommended to us by users of the site and by those who share our interests.
With the initial phase of resource identification and coding complete, future emphasis will
be on encouraging formal evaluation of the effects of resources on understanding and ability
to apply Key Concepts, by independent groups who express interest in the project.
Some examples of formally evaluated resources include the booklet `Know Your Chances'
, which was compared objectively with a decision aid in two randomised controlled trials,
in populations of different socioeconomic backgrounds [
]. Both trials found that people
who had read `Know Your Chances’ were better able to understand risk than those who had
used the decision aid.
Another formally evaluated resource is `Thinking, Doing, Talking Science' [
promotes an alternative method of teaching science to primary school children, with an emphasis
on greater cognitive challenge, interactivity and scientific reasoning [
]. This approach was
evaluated in a randomised controlled trial that demonstrated an average increase in pupil
progress in science, compared with control children, equivalent to two extra terms of teaching.
It also showed that progress was relatively greater in students from poorer socioeconomic
backgrounds. This study did not measure pupils' understanding or ability to apply Key
Concepts, but rather how well they did in national exams. It demonstrated that, in these terms,
scientific reasoning improved.
A challenge in formally evaluating the impact of learning-resources is how to define and
measure their effects. The IHC Project has created the Claim Evaluation Tools
(AustvollDahlgren et al. Submitted), a collection of multiple choice questions that test knowledge of the
Key Concepts (www.testingtreatments.org/create-test-claim-evaluation-tools-database). These
have been validated in Ugandan primary school children [
] for which information is
publicly available through www.testingtreatments.org, and will potentially help interested
researchers to conduct formal evaluations of resources.
In a randomised trial involving 120 schools, the Claim Evaluation Tools have been used
to assess the effects of IHC primary school learning-resources (including a comic book text)
on primary school children in Uganda [
]. In a linked trial, IHC resources were. used to
evaluate the effects on parents of primary school children who have listened to an IHC
Helping children to think critically about treatment claims is important, particularly at an
age when they may be more open to thinking critically, and have the time to learn the skills
needed. Acquiring these skills as young children could provide a foundation upon which to
strengthen their ability to make reasoned decisions. Indeed, the Key Concepts upon which the
content of CARL is organised apply not only to claims about the effects of treatments, but also
to other arguments and assertionsÐeconomic, social and political.
To summarise, the James Lind Initiative has coordinated the development of an
openaccess Critical thinking and Appraisal Resource Library (CARL), to help teachers and learners
8 / 12
to increase general knowledge of Key Concepts relevant to assessing claims about the effects of
treatment. These resources are currently available at www.testingtreatments.org.
The Fair Comparisons Network
Expressions of interest in CARL have been encouraging and have led to the development of a
`Fair Comparisons Network'. This an informal list of individuals who share a common interest
in promoting critical thinking and critical appraisal skills. The network is open for other
members with similar interests to join and to help identify resources for inclusion in CARL. We
hope that, as the Fair Comparisons Network expands, people who share an interest in a specific
target user group or resource format will work together to increase the proportion of that are
formally evaluated learning-resources. People who are not already members of the Fair
Comparisons Network can join it by emailing Patricia Atkinson at .
Emails should include Name, Contact Information, and areas of interest.
S1 Appendix. The key concepts.
S2 Appendix. Selection criteria for learning-resources.
S3 Appendix. Selection criteria for evaluations of learning-resources.
Conceptualization: John C. Castle, Iain Chalmers, Patricia Atkinson, Douglas Badenoch,
Andrew D. Oxman, Astrid Austvoll-Dahlgren, L. Kendall Krause, Amanda Burls, Paul
Data curation: John C. Castle, Iain Chalmers, Patricia Atkinson, Douglas Badenoch.
Formal analysis: John C. Castle.
Funding acquisition: Iain Chalmers.
Investigation: John C. Castle, Astrid Austvoll-Dahlgren, Lena Nordheim, L. Kendall Krause,
Paola Mosconi, Tammy Hoffmann, Leila Cusack, Loai Albarqouni.
Methodology: John C. Castle, Iain Chalmers, Douglas Badenoch, Andrew D. Oxman, Astrid
Austvoll-Dahlgren, Paul Glasziou.
Project administration: Iain Chalmers, Patricia Atkinson.
Resources: John C. Castle, Iain Chalmers, Astrid Austvoll-Dahlgren, L. Kendall Krause, Lisa
M. Schwartz, Steven Woloshin, Amanda Burls, Paola Mosconi, Tammy Hoffmann, Leila
Cusack, Loai Albarqouni.
Software: John C. Castle, Patricia Atkinson, Douglas Badenoch.
Supervision: Andrew D. Oxman, Astrid Austvoll-Dahlgren, Amanda Burls, Tammy
Hoffmann, Paul Glasziou.
Validation: Douglas Badenoch, Astrid Austvoll-Dahlgren.
Visualization: John C. Castle, Douglas Badenoch, Paul Glasziou.
9 / 12
Writing ± original draft: John C. Castle, Iain Chalmers.
Writing ± review & editing: John C. Castle, Iain Chalmers, Patricia Atkinson, Douglas
Badenoch, Andrew D. Oxman, Astrid Austvoll-Dahlgren, Lena Nordheim, L. Kendall Krause,
Lisa M. Schwartz, Steven Woloshin, Amanda Burls, Paola Mosconi, Tammy Hoffmann,
Leila Cusack, Loai Albarqouni, Paul Glasziou.
Haneef R, Lazarus C, Ravaud Phillippe, Yavchitz A, Boutron I. Interpretation of Results of Studies
Evaluating an Intervention Highlighted in Google Health News: A Cross-Sectional Study of News. PLoS
ONE. 2015; 10(10): e0140889. https://doi.org/10.1371/journal.pone.0140889 PMID: 26473725
Moynihan R, Bero L, Ross-Degnan D, Henry D, Lee K, Watkins J, et al. Coverage by the news media of
the benefits and risks of medications. New England Journal of Medicine. 2000; 342: 1645±50. https://
doi.org/10.1056/NEJM200006013422206 PMID: 10833211
Evans I, Thornton H, Chalmers I, Glasziou P. Testing Treatments: Better Research for Better
Healthcare. 2nd Edition. London: Pinter & Martin; 2011. http://www.testingtreatments.org/new-edition/
Eichler K, Wieser S, Brϋgger U. The costs of limited health literacy: a systematic review. International
Journal of Public Health. 2009; 54(5): 313±24. https://doi.org/10.1007/s00038-009-0058-2 PMID:
Hoffman T, Del Mar C. Patients' Expectations of the Benefits and Harms of Treatments, Screening, and
Tests. A Systematic Review. JAMA Internal Medicine. 2015; 175(2):274±286. https://doi.org/10.1001/
jamainternmed.2014.6016 PMID: 25531451
Findley S. Prescription drugs and mass media advertising (research brief). The National Institute for
Health Care Management Research Foundation; 2000. http://www.nihcm.org/pdf/DTCbrief.pdf
Fortuna RJ, Ross-Degnan D, Finkelstein J, Zhang F, Campion FX, Simon SR. Clinician attitudes
towards prescribing and implications for interventions in a multi-specialty group practice. Journal of
Evaluation in Clinical Practice. 2008; 14(6): 969±973. https://doi.org/10.1111/j.1365-2753.2007.00913.
x PMID: 18462286
Sporton RK, Francis SA. Choosing not to immunize: are parents making informed decisions? Family
Practice. 2000; 18(2): 181±188.
Allen B, Armstrong NE. Burden of proof: the evidence clinicians require before implementing an
intervention. Child and Adolescent Mental Health. 2014; 19(1): 52±56.
Moynihan R, Heath I, Henry D. Selling sickness: the pharmaceutical industry and disease mongering.
British Medical Journal. 2002; 324(7342): 886±91. PMID: 11950740
Glenton C, Paulsen EJ, Oxman AD. Portals to Wonderland: Health portals lead to confusing information
about the effects of health care. BMC Medical Informatics and Decision Making. 2005; 5:7. https://doi.
org/10.1186/1472-6947-5-7 PMID: 15769291
Nutbeam D. Health literacy as a public health goal: a challenge for contemporary health education and
communication strategies into the 21st century. Health Promotion International. 2006; 15(3): 259±267
Zarcadoolas C, Pleasant A, Greer D. Advancing Health Literacy: A framework for understanding and
action. 1st Edition. San Francisco, CA: Jossey-Bass; 2006
Nutbeam D. Health promotion glossary. Health Promotion International. 1998; 13(4):349±64. http://
Woloshin S, Schwartz LM, Black WC, Welch HG. Women's perceptions of breast cancer risk: how you
ask matters. Medical Decision Making. 1999; 19(3): 221±9. https://doi.org/10.1177/
0272989X9901900301 PMID: 10424829
Peterson G, Aslani P, Williams KA. How do consumers search for and appraise information on
medicines on the Internet? A qualitative study using focus groups. Journal of Medical Internet Research.
2003; 5(4): 1±15. Available from: https://opus.lib.uts.edu.au/handle/10453/15093.
Robinson E, Kerr C, Stevens A, Lilford R, Braunholtz D, Edwards S, et al. Lay public's understanding of
equipoise and randomisation in randomised controlled trials. Health Technology Assessment. 2005; 9
(8): 1±192. PMID: 15763039
Sillence E, Briggs P, Harris PR, Fishwick L. How do patients evaluate and make use of online health
information? Social Science & Medicine. 2007; 64(9): 1853±62
Akl EA, Oxman AD, Herrin J, Vist GE, Terrenato I, Sperati F, et al. Using alternative statistical formats
for presenting risks and risk reductions. Cochrane Database of Systematic Reviews. 2011. https://doi.
org/10.1002/14651858.CD006776.pub2 PMID: 21412897
10 / 12
11 / 12
20. Berkman ND , Sheridan SL , Donahue KE , Halpern DJ , Crottey K . Low Health Literacy and Health Outcomes: An Updated Systematic Review . Annals of Internal Medicine . 2011 ; 155 ( 2 ): 97 ± 107 . https://doi. org/10.7326/ 0003 -4819-155-2- 201107190 -00005 PMID: 21768583
21. Burgess AM , Okamura KH , Izmirian SC , Higa-McMillan CK , Shimabukoro S , Nakamura BJ . Therapist Attitudes Towards Evidence-Based Practice: A Joint Factor Analysis . Journal of Behavioural Health Services & Research . 2016 ; 1 ± 13 . https://doi.org/10.1007/s11414-016 -9517-8 PMID: 27230350
22. Coulter A . The Autonomous Patient: Ending Paternalism in Medical Care . London. Stationery Office Books: 2002 .
23. Hoffmann TC . The Connection Between Evidence-Based Medicine and Shared Decision Making . JAMA . 2014 ; 312 ( 13 ): 1295 ± 1296 . https://doi.org/10.1001/jama. 2014 .10186 PMID: 25268434
24. Mickan S , Burls A . Enhancing Public Understanding of Health Research. Report of a conference held at Kellogg College , Oxford University, 7± 8 April 2007 .
25. Mosconi P , Antes G , Barbareschi G , Burls A , Demotes-Mainard J , Chalmers I , et al. A European multilanguage initiative to make the general population aware of independent clinical research: the European Communication on Research Awareness Need (ECRAN) project . Trials . 2016 ; 17 : 19 . https://doi.org/ 10.1186/s13063-015 -1146-7 PMID: 26758509
26. Nordheim LV , Gundersen MW , Espehaug B , Guttersud Ø , Flottorp S. Effects of School-Based Educational Interventions for Enhancing Adolescents Abilities in Critical Appraisal of Health Claims: A Systematic Review . PLoS ONE . 2016 ; 11 ( 8 ): e0161485. https://doi.org/10.1371/journal.pone. 0161485 PMID: 27557129
27. Chalmers I , Milne I , TroÈhler U , Vandenbrouche J , Morabia A , Tait G , et al. James Lind Library Editorial Team. The James Lind Library: explaining and illustrating the evolution of fair tests of medical treatments . Journal of the Royal College of Physicians of Edinburgh . 2008 ; 38 ( 3 ): 259 ± 64 . PMID: 19227602
28. Austvoll-Dahlghren A , Semakula D , Nsangi A . Interventions and assessment tools addressing key concepts people need to know to appraise claims about treatment effects: a systematic mapping review . Systematic Reviews . 2017 ; 5 : 215 .
29. Cusack L , Del Mar CB , Chalmers I , Hoffmann TC . Educational interventions to improve people's understanding of key concepts in assessing the effects of health interventions: a systematic review protocol . Systematic Reviews . 2016 ; 5 ( 37 ): 1 ± 8 .
30. Austvoll-Dahlgren A , Oxman AD , Chalmers I , Nsangi A , Glenton C , Lewin S , et al. Key Concepts that people need to understand to assess claims about treatment effects . Journal of Evidence Based Medicine . 2015 ; 8 ( 3 ): 112 ± 25 . Available from: http://onlinelibrary.wiley.com/enhanced/doi/10.1111/jebm. 12160/ PMID: 26107552
31. Leshowitz B , Jenkens K , Heaton S , Bough TL . Fostering critical thinking skills in students with learning disabilities: an instructional program . Journal of Learning Disabilities . 1993 ; 26 ( 7 ): 483 ± 490 . https://doi. org/10.1177/002221949302600707 PMID: 8409746
32. Hendricks CC . Teaching causal reasoning through cognitive apprenticeship: what are results from situated learning? Journal of Educational Research . 2001 ; 94 ( 5 ): 302 ± 311 .
33. Krueger PM . Teaching critical appraisal: a pilot randomized controlled outcomes trial in undergraduate osteopathic medical education . Journal of the American Osteopathic Association . 2006 ; 106 ( 11 ): 658 ± 62 . PMID: 17192453
34. Woloshin S , Schwartz L , Welch G H. The effectiveness of a primer to help people understanding risk: two randomised trials in distinct populations . Annals of Internal Medicine . 2007 ; 146 ( 4 ): 256 ± 265 . PMID: 17310049
35. Kaelin MA , Huebner WW , Nicolich MJ , Kimbrough ML . Field test of an epidemiology curriculum for middle school students . American Journal of Health Education . 2007 ; 38 ( 1 ): 16 ± 31 . PMID: 18274623
36. Steckelberg A , Hulfenhaus C , Kasper J , Mϋlhauser I. EBM @ SchoolÐa curriculum of critical health literacy for secondary school students: results of a pilot study . International Journal of Public Health . 2009 ; 54 ( 3 ): 158 ± 165 . https://doi.org/10.1007/s00038-008 -7033-1 PMID: 19183847
37. Pearce-Smith N . CASP bibliography. 2012 . http://media.wix.com/ugd/dded87_ 699a6d739c663b77736b8e6262aef90d.pdf
38. Webershock T , Sorinola O , Thangaratinam S , Oude Rengerink K , Arvanitis TN , Khan KS . EBM Unity Group. How to confidently teach EBM on foot: development and evaluation of a web-based e-learning course . Evidence Based Medicine . 2013 ; 18 ( 5 ): 170 ±2. https://doi.org/10.1136/eb-2012 -100801 PMID: 22864372
39. Austvoll-Dahlgren A , Danielsen S , Opheim E , Bjørndal A , Reinar LM , Flottorp S , et al. Development of a complex intervention to improve health literacy skills . Health Information and Libraries . 2013 ; 30 ( 4 ): 278 ± 295 .
40. Mant J , Wilson H , Coates D. The effect of increasing conceptual challenge in primary science lessons on pupils' achievement and engagement . International Journal of Science Education . 2007 ; 29 ( 14 ): 1707 ± 1719 . Available from: http://dx.doi.org/10.1080/09500690701537973
41. Gorard S , Siddiqui N , Huat See B. Philosophy for Children: Evaluation report and Executive summary . Education Endowment Foundation . 2015 . https://v1.educationendowmentfoundation.org.uk/uploads/ pdf/Philosophy_for_Children.pdf
Woloshin S , Schwartz LM , Welch HG . Know Your Chances: Understanding Health Statistics. Berkeley (CA): University of California Press; 2008 .
43. Gorard S , Siddiqui N , Huat See B. Philosophy for Children: Evaluation report and Executive summary . Education Endowment Foundation . 2015 . https://v1.educationendowmentfoundation.org.uk/uploads/ pdf/Philosophy_for_Children.pdf
44. Austvoll-Dahlgren A , Semakula D , Nsangi A . The development of the ªclaim evaluation toolsº: assessing critical thinking about effects . BMJ Open forthcoming
45. Nsangi A , Semakula D , Oxman AO , Austvoll-Dahlgren A , Oxman M , Rosenbaum S , et al. Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. 2017; Lancet Online . http://dx.doi.org/10.1016/S0140- 6736 ( 17 ) 31226 - 6
46. Semakula D , Nsangi A , Oxman AO , Oxman M , Austvoll-Dahlgren A , Rosenbaum S , Morelli A , Glenton C , Lewin S , Kaseje M , Chalmers I , Fretheim A , Kristoffersen DT , Sewankambo NK . Effects of the Informed Health Choices podcast on the ability of parents of primary school children in Uganda to assess claims about treatment effects: a randomised controlled trial. 2017; Lancet Online . http://dx.doi. org/10.1016/S0140- 6736 ( 17 ) 31225 - 4