The Use of Bayesian Networks to Assess the Quality of Evidence from Research Synthesis: 1.

PLOS ONE, Dec 2019

Background The grades of recommendation, assessment, development and evaluation (GRADE) approach is widely implemented in systematic reviews, health technology assessment and guideline development organisations throughout the world. A key advantage to this approach is that it aids transparency regarding judgments on the quality of evidence. However, the intricacies of making judgments about research methodology and evidence make the GRADE system complex and challenging to apply without training. Methods We have developed a semi-automated quality assessment tool (SAQAT) l based on GRADE. This is informed by responses by reviewers to checklist questions regarding characteristics that may lead to unreliability. These responses are then entered into the Bayesian network to ascertain the probabilities of risk of bias, inconsistency, indirectness, imprecision and publication bias conditional on review characteristics. The model then combines these probabilities to provide a probability for each of the GRADE overall quality categories. We tested the model using a range of plausible scenarios that guideline developers or review authors could encounter. Results Overall, the model reproduced GRADE judgements for a range of scenarios. Potential advantages over standard assessment are use of explicit and consistent weightings for different review characteristics, forcing consideration of important but sometimes neglected characteristics and principled downgrading where small but important probabilities of downgrading are accrued across domains. Conclusions Bayesian networks have considerable potential for use as tools to assess the validity of research evidence. The key strength of such networks lies in the provision of a statistically coherent method for combining probabilities across a complex framework based on both belief and evidence. In addition to providing tools for less experienced users to implement reliability assessment, the potential for sensitivity analyses and automation may be beneficial for application and the methodological development of reliability tools.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0114497&type=printable

The Use of Bayesian Networks to Assess the Quality of Evidence from Research Synthesis: 1.

April The Use of Bayesian Networks to Assess the Quality of Evidence from Research Synthesis: 1. Gavin B. Stewart 0 1 2 3 Julian P. T. Higgins 0 1 2 3 Holger Schnemann 0 1 2 3 Nick Meader 0 1 2 3 0 1 Centre for Reviews and Dissemination, University of York, York, United Kingdom, 2 School of Social and Community Medicine, University of Bristol , Bristol , United Kingdom , 3 Department of Clinical Epidemiology & Biostatistics, McMaster University Health Sciences Centre , Hamilton, ON , Canada 1 Funding: Gavin Stewart and Nick Meader were supported by Centre for Reviews and Dissemination core contract funding from the National Institute For Health Research, UK. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript 2 Academic Editor: Neil R. Smalheiser, University of Illinois-Chicago , UNITED STATES 3 Current address: School of Agriculture, Food and Rural Development, Newcastle University , Newcastle upon Tyne , United Kingdom - Competing Interests: The authors have declared that no competing interests exist. The grades of recommendation, assessment, development and evaluation (GRADE) approach is widely implemented in systematic reviews, health technology assessment and guideline development organisations throughout the world. A key advantage to this approach is that it aids transparency regarding judgments on the quality of evidence. However, the intricacies of making judgments about research methodology and evidence make the GRADE system complex and challenging to apply without training. We have developed a semi-automated quality assessment tool (SAQAT) l based on GRADE. This is informed by responses by reviewers to checklist questions regarding characteristics that may lead to unreliability. These responses are then entered into the Bayesian network to ascertain the probabilities of risk of bias, inconsistency, indirectness, imprecision and publication bias conditional on review characteristics. The model then combines these probabilities to provide a probability for each of the GRADE overall quality categories. We tested the model using a range of plausible scenarios that guideline developers or review authors could encounter. Overall, the model reproduced GRADE judgements for a range of scenarios. Potential advantages over standard assessment are use of explicit and consistent weightings for different review characteristics, forcing consideration of important but sometimes neglected characteristics and principled downgrading where small but important probabilities of downgrading are accrued across domains. Bayesian networks have considerable potential for use as tools to assess the validity of research evidence. The key strength of such networks lies in the provision of a statistically coherent method for combining probabilities across a complex framework based on both belief and evidence. In addition to providing tools for less experienced users to implement reliability assessment, the potential for sensitivity analyses and automation may be beneficial for application and the methodological development of reliability tools. The fundamental objective of evidence-based health care is to enable clinicians or policy makers to make informed decisions regarding the development or delivery of effective health interventions. The reliability of the evidence underpinning decisions is important particularly where high and low quality evidence result in different conclusions regarding effectiveness. Researchers have developed a range of tools for considering the validity of conclusions from research synthesis. Easily applicable tools [e.g. 1] focus on the process and robustness of research synthesis but their practical utility is limited as they do not provide direct information on the quality of evidence underpinning decisions. This requires evaluation of biases across studies, as well as biases detected during the process of evidence synthesis. The Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) approach [216] is the most widespread method for rating the quality of evidence in healthcare. GRADE has been adopted by organisations such as the World Health Organization, Cochrane Collaboration, Agency for Healthcare Research and Quality, and National Institute for Health and Clinical Excellence. The GRADE approach is described in online support for software [http://ims.cochrane.org/ revman/gradepro] and in a BMJ series of papers in 2008, with updated guidance available in a Journal of Clinical Epidemiology series beginning in 2011 [216]. According to the GRADE approach, evidence based on randomised controlled trials is considered high quality (reliable), but can be downgraded across five domains: risk of bias, inconsistency, indirectness, imprecision, publication bias. Non-randomised studies begin at low quality evidence but their rating can be upgraded (provided no other limitations have been identified in the five (...truncated)


This is a preview of a remote PDF: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0114497&type=printable

Gavin B. Stewart, Julian P. T. Higgins, Holger Schünemann, Nick Meader. The Use of Bayesian Networks to Assess the Quality of Evidence from Research Synthesis: 1., PLOS ONE, 2015, Volume 10, Issue 4, DOI: 10.1371/journal.pone.0114497