The Use of Bayesian Networks to Assess the Quality of Evidence from Research Synthesis: 1.
April
The Use of Bayesian Networks to Assess the Quality of Evidence from Research Synthesis: 1.
Gavin B. Stewart 0 1 2 3
Julian P. T. Higgins 0 1 2 3
Holger Schnemann 0 1 2 3
Nick Meader 0 1 2 3
0 1 Centre for Reviews and Dissemination, University of York, York, United Kingdom, 2 School of Social and Community Medicine, University of Bristol , Bristol , United Kingdom , 3 Department of Clinical Epidemiology & Biostatistics, McMaster University Health Sciences Centre , Hamilton, ON , Canada
1 Funding: Gavin Stewart and Nick Meader were supported by Centre for Reviews and Dissemination core contract funding from the National Institute For Health Research, UK. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript
2 Academic Editor: Neil R. Smalheiser, University of Illinois-Chicago , UNITED STATES
3 Current address: School of Agriculture, Food and Rural Development, Newcastle University , Newcastle upon Tyne , United Kingdom
-
Competing Interests: The authors have declared
that no competing interests exist.
The grades of recommendation, assessment, development and evaluation (GRADE)
approach is widely implemented in systematic reviews, health technology assessment and
guideline development organisations throughout the world. A key advantage to this
approach is that it aids transparency regarding judgments on the quality of evidence. However,
the intricacies of making judgments about research methodology and evidence make the
GRADE system complex and challenging to apply without training.
We have developed a semi-automated quality assessment tool (SAQAT) l based on
GRADE. This is informed by responses by reviewers to checklist questions regarding
characteristics that may lead to unreliability. These responses are then entered into the
Bayesian network to ascertain the probabilities of risk of bias, inconsistency, indirectness,
imprecision and publication bias conditional on review characteristics. The model then
combines these probabilities to provide a probability for each of the GRADE overall quality
categories. We tested the model using a range of plausible scenarios that guideline developers
or review authors could encounter.
Overall, the model reproduced GRADE judgements for a range of scenarios. Potential
advantages over standard assessment are use of explicit and consistent weightings for
different review characteristics, forcing consideration of important but sometimes neglected
characteristics and principled downgrading where small but important probabilities of
downgrading are accrued across domains.
Bayesian networks have considerable potential for use as tools to assess the validity of
research evidence. The key strength of such networks lies in the provision of a statistically
coherent method for combining probabilities across a complex framework based on both
belief and evidence. In addition to providing tools for less experienced users to implement
reliability assessment, the potential for sensitivity analyses and automation may be
beneficial for application and the methodological development of reliability tools.
The fundamental objective of evidence-based health care is to enable clinicians or policy
makers to make informed decisions regarding the development or delivery of effective health
interventions. The reliability of the evidence underpinning decisions is important particularly
where high and low quality evidence result in different conclusions regarding effectiveness.
Researchers have developed a range of tools for considering the validity of conclusions from
research synthesis. Easily applicable tools [e.g. 1] focus on the process and robustness of research
synthesis but their practical utility is limited as they do not provide direct information on the
quality of evidence underpinning decisions. This requires evaluation of biases across studies, as
well as biases detected during the process of evidence synthesis.
The Grades of Recommendation, Assessment, Development, and Evaluation (GRADE)
approach [216] is the most widespread method for rating the quality of evidence in healthcare.
GRADE has been adopted by organisations such as the World Health Organization, Cochrane
Collaboration, Agency for Healthcare Research and Quality, and National Institute for Health
and Clinical Excellence.
The GRADE approach is described in online support for software [http://ims.cochrane.org/
revman/gradepro] and in a BMJ series of papers in 2008, with updated guidance available in a
Journal of Clinical Epidemiology series beginning in 2011 [216]. According to the GRADE
approach, evidence based on randomised controlled trials is considered high quality (reliable),
but can be downgraded across five domains: risk of bias, inconsistency, indirectness,
imprecision, publication bias. Non-randomised studies begin at low quality evidence but their rating
can be upgraded (provided no other limitations have been identified in the five (...truncated)