Impact of missing participant data for dichotomous outcomes on pooled effect estimates in systematic reviews: a protocol for a methodological study

Systematic Reviews Impact of missing participant data for dichotomous outcomes on pooled effect estimates in systematic reviews: a protocol for a methodological study Elie A Akl 0 1 Lara A Kahale 1 Arnav Agarwal 0 Nada Al-Matari 1 Shanil Ebrahim 0 Paul Elias Alexander 0 Matthias Briel 0 Romina Brignardello-Petersen 0 Jason W Busse 0 Batoul Diab 1 Alfonso Iorio 0 Joey Kwong Ling Li Luciane Cruz Lopes Reem Mustafa 0 Ignacio Neumann 0 Kari AO Tikkinen Per Olav Vandvik Yuqing Zhang 0 Pablo Alonso-Coello Gordon Guyatt 0 0 Department of Clinical Epidemiology and Biostatistics, McMaster University , Hamilton , Canada 1 Department of Internal Medicine, Clinical Epidemiology Unit, American University of Beirut Medical Center , PO Box: 11-0236, Riad-El-Solh, Beirut 1107, 2020, Beirut , Lebanon Background: There is no consensus on how authors conducting meta-analysis should deal with trial participants with missing outcome data. The objectives of this study are to assess in Cochrane and non-Cochrane systematic reviews: (1) which categories of trial participants the systematic review authors consider as having missing participant data (MPD), (2) how trialists reported on participants with missing outcome data in trials, (3) whether systematic reviewer authors actually dealt with MPD in their meta-analyses of dichotomous outcomes consistently with their reported methods, and (4) the impact of different methods of dealing with MPD on pooled effect estimates in meta-analyses of dichotomous outcomes. Methods/Design: We will conduct a methodological study of Cochrane and non-Cochrane systematic reviews. Eligible systematic reviews will include a group-level meta-analysis of a patient-important dichotomous efficacy outcome, with a statistically significant effect estimate. Teams of two reviewers will determine eligibility and subsequently extract information from each eligible systematic review in duplicate and independently, using standardized, pre-piloted forms. The teams will then use a similar process to extract information from the trials included in the meta-analyses of interest. We will assess first which categories of trial participants the systematic reviewers consider as having MPD. Second, we will assess how trialists reported on participants with missing outcome data in trials. Third, we will compare what systematic reviewers report having done, and what they actually did, in dealing with MPD in their meta-analysis. Fourth, we will conduct imputation studies to assess the effects of different methods of dealing with MPD on the pooled effect estimates of meta-analyses. We will specifically calculate for each method (1) the percentage of systematic reviews that lose statistical significance and (2) the mean change of effect estimates across systematic reviews. Discussion: The impact of different methods of dealing with MPD on pooled effect estimates will help judge the associated risk of bias in systematic reviews. Our findings will inform recommendations regarding what assumptions for MPD should be used to test the robustness of meta-analytical results. Missing participant data; Imputation; Risk of bias; Trials; Systematic reviews; Meta-analysis - Background Missing data for the outcomes of interest is a common problem in randomized trials [1]. In one study, almost one in every three trials with statistically significant results lost statistical significance when making plausible assumptions about the outcomes of participants with missing data [1]. This reduces our confidence in the effect estimates resulting not only from these trials but also from systematic reviews including them. One challenge with abstracting data from trial reports is understanding whether or not data from a specific category of participants is missing. For example, a trial report might indicate that a certain number of participants withdrew consent, without indicating whether or not they continued to be followed-up. In these cases, systematic review authors might need to make assumptions based on their best guess. Moreover, a recent methodological survey found that systematic reviews are deficient in terms of dealing with and assessing the risk of bias associated with missing participant data (MPD) (unpublished data by Akl et al.) [2]. The Cochrane Collaborations Risk of Bias (RoB) tool was designed to help in assessing bias associated with a number of factors, including MPD [3]. In a recent study evaluating stakeholders experiences with, and perceptions of, the Cochrane RoB tool participants cited incomplete outcome data as one of the most difficult domains to assess [4]. They also requested more guidance on how to incorporate RoB assessments into meta-analyses and conclusions [4]. One approach to assessing the extent of risk of bias associated with MPD is to conduct sensitivity analyses based on different assumptions regarding the outcomes of participants with missing outcome data [5-7]. Examples of these assumptions include none, and all or a specified proportion of participants in each group suffering the outcome of interest. No study has thus far assessed how systematic reviewers actually deal with MPD. Similarly, there is a lack of evidence about the impact of different approaches for dealing with MPD on pooled effect estimates. This leaves uncertain the extent to which results of systematic reviews are vulnerable to MPD. Objectives The objectives of this study are to assess in Cochrane and non-Cochrane systematic reviews: (1) which categories of trial participants the systematic review authors consider as having MPD, (2) how trialists reported on participants with missing outcome data in trials, (3) whether systematic reviewer authors actually dealt with MPD in their meta-analyses of dichotomous outcomes consistently with their reported methods, and (4) the impact of different methods of dealing with MPD on pooled effect estimates in meta-analyses of dichotomous outcomes. Methods/Design We did not register this protocol with PROSPERO given the register does not accept methodological reviews. Definitions MPD refers to outcome data for trial participants that are not available to the systematic reviewer authors (from the published trial reports or personal contact with trial authors) for inclusion in their meta-analyses. MPD do not relate to any of the following: Missing studies (e.g., unpublished studies); Entire unreported outcomes (e.g., outcomes planned in trial protocols but not included in trial reports). Cochrane systematic reviews are defined as systematic reviews published in the Cochrane Database of Systematic Reviews. All other systematic reviews will be considered non-Cochrane systematic reviews. A patient-important outcome is defined as an outcome for which one would answer with yes to the following question: if the patient knew that this outcome was the only thing to change with treatment, would the patient consider receiving this treatment if associated with burden, si (...truncated)