Profound effect of study design factors on ventilator-associated pneumonia incidence of prevention studies: benchmarking the literature experience

Journal of Antimicrobial Chemotherapy (2008) 61, 1154– 1161 doi:10.1093/jac/dkn086 Advance Access publication 8 March 2008 Profound effect of study design factors on ventilator-associated pneumonia incidence of prevention studies: benchmarking the literature experience James C. Hurley1 – 3* School of Rural Health, University of Melbourne, Australia; 2Infection Control Units, Ballarat Health Services and St John of God Hospital, Ballarat, Australia; 3Division of Internal Medicine, Ballarat Health Services, PO Box 577, Ballarat, Victoria 3353, Australia Received 30 December 2007; returned 23 January 2008; revised 6 February 2008; accepted 11 February 2008 Background: The ventilator-associated pneumonia incident proportion (VAP-IP) is highly variable among control groups of studies of methods for its prevention. The objective here is to develop and validate a literature-derived benchmark against which these groups can be profiled. Methods: A literature search yielded 95 cohort groups and control and intervention groups of 150 studies of either non-antimicrobial or antimicrobial methods of VAP prevention. The 95 cohort groups comprise a benchmark set (30 groups), from which the reference funnel plot (RFP) was derived, and a search set (65 groups), against which the benchmark was validated. The VAP-IP data of the benchmark set were found in five published systematic reviews, whereas the VAP-IP data of the search set were abstracted directly from the literature. Findings: Among the 95 cohort groups, the VAP-IP of groups with size >399 was significantly lower than the VAP-IP of smaller groups. Compared with the RFP, 15 of 51 (29%) control groups from studies of antimicrobial methods of VAP prevention with concurrent design were high outlier versus 2 of 110 (2%) control groups from other types of study design (P < 0.001). There were only 22 (14%) outlier groups, all low outlier, among the 162 intervention groups. Conclusions: Study design factors such as concurrency and study size have potentially greater influence on the VAP-IP than do the VAP prevention methods under study. The outlier status of control groups were inapparent in the individual studies and the meta-analyses and yet would have confounded the estimates of treatment effect. Keywords: antimicrobial prophylaxis, cross-infection, funnel plots Introduction There are several published range estimates for the ventilatorassociated pneumonia incident proportion (VAP-IP) among patients receiving prolonged (.48 h) mechanical ventilation (Table 1). These estimates represent expert opinion and are in turn derived from the results of over 30 observational studies.1 – 5 In addition, there are .140 intervention studies of various non-antimicrobial-based methods and antimicrobial-based methods of VAP prevention in this patient group. The results of these studies, which are summarized in over 20 narrative and systematic reviews,6 – 28 are marked by heterogeneity in treatment effect and VAP-IP data, which are highly variable, particularly so for studies of antimicrobial-based methods of VAP prevention. The effect of widespread antimicrobial use in the ICU environment is 2-fold. It may prevent infection, but it may also alter the ecology of the ICU and increase the colonization pressure and cross-infection risk.29,30 The impact on crossinfection on the results of these studies of VAP prevention methods is unknown.31 Outbreaks and cross-infection occur in the ICU setting but both are thought to be under-recognized by conventional surveillance methods.32,33 However, using molecular typing techniques to identify cross-colonization, this accounts for up to 23% of colonization and up to 37% of cases of infection with Staphylococcus aureus in the typical ICU setting.34,35 ..................................................................................................................................................................................................................................................................................................................................................................................................................................... *Correspondence address. Internal Medicine Service, Ballarat Health Services, PO Box 577, Ballarat, Victoria 3353, Australia. Tel: þ61-3-53-204322; Fax: þ61-3-53-204472; E-mail: ..................................................................................................................................................................................................................................................................................................................................................................................................................................... 1154 # The Author 2008. Published by Oxford University Press on behalf of the British Society for Antimicrobial Chemotherapy. All rights reserved. For Permissions, please e-mail: 1 Study designs and ventilator-associated pneumonia Table 1. Literature-based range estimates for ventilator-associated pneumonia incident proportion (VAP-IP) VAP-IP range (%)c Authorsa Year No. of abstracted studiesb George1 Cook and Kollef2 Chastre and Fagon3 Bergmans and Bonten4 Safdar et al.5 This study 1993 1998 2002 2004 11 8 10 15 9–30 13–38 8–28 8.6–65 2005 2007 28 31 7–12.5d 8–46e References1 – 5 are source documents. Number of studies abstracted in defining a VAP-IP range estimate. c The VAP-IP range estimates1 – 4 are maximum-minimum range intervals. d The range estimate of Safdar et al.5 is a weighted average and 95% confidence interval. e The range estimate from this study corresponding to the two sigma limits of the RFP at a group size of 100 patients. b Patterns of VAP isolates including an increase in S. aureus have been noted among control groups of studies of antimicrobial methods of VAP prevention suggesting that inapparent outbreaks had occurred.33 Partly because of these concerns, in studies of antimicrobial-based methods, different study designs had been used such as concurrent versus non-concurrent group design, and use or non-use of topical placebo to achieve study blinding where topical antimicrobial is one of the study interventions. In the interpretation of these prevention studies, both for any one study and also in the summary result of a meta-analysis, the presumption is that the study effect occurs only in the intervention groups (i.e. a reduction in VAP-IP) and not in the control groups (i.e. an increase in VAP-IP). This presumption has never been challenged and cannot be tested without a benchmark and a method for profiling the results. Moreover, an objective benchmark against which infection rates could be profiled would also facilitate the detection of outbreaks. Funnel plot methodology is commonly used to detect outlier results in systematic reviews.36 More recently, funnel plot methodology has been applied as a type of statistica (...truncated)