Systematic reviews of health economic evaluations: a protocol for a systematic review of characteristics and methods applied
Luhnen et al. Systematic Reviews
Systematic reviews of health economic evaluations: a protocol for a systematic review of characteristics and methods applied
Miriam Luhnen 1 2
Barbara Prediger 0
Edmund A. M. Neugebauer 3 4
Tim Mathes 0
0 Institute for Research in Operative Medicine, Witten/Herdecke University , Ostmerheimer Str. 200, Haus 38, 51109 Cologne , Germany
1 Faculty of Health, Department of Medicine, Witten/Herdecke University , Ostmerheimer Str. 200, Haus 38, 51109 Cologne , Germany
2 Department Health Care and Health Economics, Institute for Quality and Efficiency in Health Care (IQWiG) , Im Mediapark 8, 50670 Cologne , Germany
3 Interdisciplinary Centre for Health Services Research, Witten/Herdecke University , Alfred-Herrhausen-Straße 50, 58448 Witten , Germany
4 Faculty of Health, Brandenburg Medical School - Theodor Fontane , Campus Neuruppin, Fehrbelliner Str. 38, 16816 Neuruppin , Germany
Background: The number of systematic reviews of economic evaluations is steadily increasing. This is probably related to the continuing pressure on health budgets worldwide which makes an efficient resource allocation increasingly crucial. In particular in recent years, the introduction of several high-cost interventions presents enormous challenges regarding universal accessibility and sustainability of health care systems. An increasing number of health authorities, inter alia, feel the need for analyzing economic evidence. Economic evidence might effectively be generated by means of systematic reviews. Nevertheless, no standard methods seem to exist for their preparation so far. The objective of this study was to analyze the methods applied for systematic reviews of health economic evaluations (SR-HE) with a focus on the identification of common challenges. Methods/design: The planned study is a systematic review of the characteristics and methods actually applied in SR-HE. We will combine validated search filters developed for the retrieval of economic evaluations and systematic reviews to identify relevant studies in MEDLINE (via Ovid, 2015-present). To be eligible for inclusion, studies have to conduct a systematic review of full economic evaluations. Articles focusing exclusively on methodological aspects and secondary publications of health technology assessment (HTA) reports will be excluded. Two reviewers will independently assess titles and abstracts and then full-texts of studies for eligibility. Methodological features will be extracted in a standardized, beforehand piloted data extraction form. Data will be summarized with descriptive statistical measures and systematically analyzed focusing on differences/similarities and methodological weaknesses. Discussion: The systematic review will provide a detailed overview of characteristics of SR-HE and the applied methods. Differences and methodological shortcomings will be detected and their implications will be discussed. The findings of our study can improve the recommendations on the preparation of SR-HE. This can increase the acceptance and usefulness of systematic reviews in health economics for researchers and medical decision makers. Systematic review registration: The review will not be registered with PROSPERO as it does not meet the eligibility criterion of dealing with clinical outcomes.
Systematic review; Economic evaluation; Reimbursement; Medical decision making
Continuing pressure on health budgets worldwide makes
an efficient resource allocation increasingly crucial. In
recent years, the introduction of several high-cost
interventions presents enormous challenges regarding accessibility
and sustainability of health care systems [
]. This makes
economic considerations more important for health
authorities and their decision-making process regarding pricing
and reimbursement especially of new interventions.
Systematic reviews of health economic evaluations
(SR-HE) can provide evidence about the
costeffectiveness of an intervention within a limited time
frame. They are valuable (1) to inform the development
of an own economic model, (2) to identify the most
relevant studies for a particular decision, and (3) to
identify the implicated economic trade-offs [
Moreover, provided that high-quality economic evaluations
that exist are sufficiently transferable and demonstrate
similar results regarding cost-effectiveness, SR-HE might
indicate the most cost-effective intervention.
Jefferson et al. [
] found that SR-HE show fundamental
methodological flaws, especially regarding their search
strategy and the application of an appropriate quality
assessment tool. Nevertheless, little research has been
performed to further develop the methods for SR-HE in the
meantime. Standards for the preparation of SR-HE do not
seem to exist so far: More recent studies focusing on the
available methodological guidelines found that the
recommendations still vary widely and are partly
]. It is therefore to be expected that the conduct
of SR-HE still varies widely and still shows methodological
shortcomings. The aim of this paper is
To provide a detailed overview of the characteristics
and applied methods in recently published SR-HE
To identify similarities and differences between the
characteristics and methods of SR-HE
To identify common challenges
We used the PRISMA-P (Preferred Reporting Items for
Systematic review and Meta-Analysis Protocols) 2015
checklist to develop the methods for this systematic
review protocol [
] (please see Additional file 1).
Should protocol amendments be necessary, these will
be documented including details of the date, changes
made, and the rationale for changes.
A systematic search in Ovid MEDLINE(R) Epub Ahead
of Print, In-Process & Other Non-Indexed Citations,
Ovid MEDLINE(R) Daily, and Ovid MEDLINE(R) 1946
to Present will be performed. We will limit the
publication date of our search to the period 2015/01/01
to present. A validated search filter for economic
evaluations (Emory University [Grady] [
]) will be combined
with a validated filter for the retrieval of systematic
reviews (Lee [
]), as presented in Table 1. This strategy
was chosen as it provides an optimal balance between
sensitivity and precision. Search results will be
downloaded to EndNote version X7 where duplicates will be
identified and removed.
We will include articles available as full-text and written
in English, German, French, or Spanish if they fulfill all
of the following criteria:
1. Systematic literature search in at least one electronic
database and transparent description of study
selection. We will exclude articles applying
abbreviated review methods (e.g., scoping reviews
and short reviews) as judged by the authors of the
2. Inclusion of full economic evaluations (i.e.,
and/or the cost-effectiveness of an intervention was
reviewed. Articles reviewing solely partial economic
evaluations (like cost-of-illness studies or budget
impact analyses) will be excluded.
3. Objective to answer a cost-effectiveness research
question, i.e., we will exclude articles focusing
exclusively on methodological aspects (e.g.,
analysis of methods applied in health economic
4. Full-text journal article. Protocols, commentaries,
editorials, and conference proceedings will be
excluded. Likewise, secondary publications of HTA
reports will be excluded as the focus of our study
will be on the scientific literature instead of
documents stemming from regulatory processes
within a certain jurisdiction in a health care system.
Two reviewers will independently assess the titles and
abstracts retrieved in the electronic literature search
against the inclusion criteria. Possible eligible full-text
articles will be retrieved and screened by two reviewers
to reach a final decision about inclusion. Any
disagreements will be resolved through discussion or
involvement of a third reviewer.
We will prepare a PRISMA flowchart to illustrate the
Methodological features will be extracted in a
standardized, beforehand piloted data extraction form (Table 2).
We developed an electronical extraction form in
Microsoft Excel 2010 for a previous study (not published yet)
in which we analyzed HTA reports of international HTA
organizations for the methods applied for SR-HE and
adapted it for the purpose of the present study. This
approach for data abstraction and data presentation was
inspired by the publication of Page et al. [
provides an overview of epidemiology and reporting
characteristics of systematic reviews of biomedical research.
Data items presented in the included articles will be
classified according to the categories depicted in Table 3.
Data will be extracted each by a single reviewer. After
extraction of the first articles, a 10% random sample will
be verified for accuracy and correctness of data entries
by a second reviewer. Discrepancies will be resolved
through discussion or third party, if necessary. In case of
frequent and/or substantial disagreements, a verification
of 100% is intended.
Data analysis and presentation
We will analyze all data using Microsoft Excel 2010.
Results for each data item extracted will be presented in
tables. For nominal data, we will provide numbers and
percentages. We will provide median and ranges for
In order to allow an estimation of the number of
SRHE published per year and to analyze possible changes
over time, we will present the number of hits resulting
from our search strategy for the years 2015 to 2017.
Since no tool for the critical appraisal of SR-HE exists
(comparable e.g., to AMSTAR [A Measurement Tool to
Assess Systematic Reviews] [
]), we will not critically
appraise included articles by means of a certain tool
acceptance and usefulness of systematic reviews in health
economics for researchers and medical decision makers.
Additional file 1: Preferred Reporting Items for Systematic review and
Meta-Analysis Protocols (PRISMA-P) 2015 checklist: recommended items
to address in a systematic review protocol. (DOCX 36 kb)
AMSTAR: A Measurement Tool to Assess Systematic Reviews; CEA:
Costeffectiveness analysis; PICOS: Patient, intervention, comparison, outcome,
setting; PRISMA: Preferred Reporting Items for Systematic review and
MetaAnalysis; PROSPERO: Prospective Register of Systematic reviews;
SRHE: Systematic reviews of health economic evaluations
No funding will be received for the proposed study.
Availability of data and materials
The datasets generated and analyzed during the current study will be
available from the corresponding author on reasonable request.
ML and TM developed and piloted the data extraction form. ML developed
the search strategy for the proposed systematic review and drafted the
manuscript. TM and BP commented on the manuscript. EAMN supported
the conceptualization of the systematic review. All authors reviewed the
manuscript and approved the final manuscript.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.
Methods for valuing outcomes/benefits
Cost/resource items included
Data sources for costs
Data sources for clinical data
Data sources for utility data
Analysis of uncertainty
Analysis of uncertainty
but focus on similarities, differences, and
As far as possible, the results of our study will be
reported in accordance with the PRISMA guidelines [
The systematic review will provide a detailed overview
of characteristics of SR-HE and the applied methods.
Differences and methodological shortcomings will be
detected and their implications will be discussed. The
findings of our study can improve the recommendations
on the preparation of SR-HE. This can increase the
1. European Commission . Inception impact assessment-strengthening of the EU cooperation on. Health Technology Assessment (HTA) Available from : http://ec.europa.eu/smart-regulation/roadmaps/docs/2016_sante_144_ health_technology_assessments_en .pdf. Accessed April 19 , 2017
2. OECD. Fiscal sustainability of health systems: bridging health and finance perspectives . Paris: OECD Publishing; 2015 .
3. Anderson R . Systematic reviews of economic evaluations: utility or futility? Health Econ . 2010 Mar; 19 ( 3 ): 350 - 64 .
4. Jefferson T , Demicheli V , Vale L . Quality of systematic reviews of economic evaluations in health care . JAMA. 2002 Jun 5 ; 287 ( 21 ): 2809 - 12 .
5. Mathes T , Walgenbach M , Antoine SL , et al. Methods for systematic reviews of health economic evaluations: a systematic review, comparison, and synthesis of method literature . Med Decis Mak . 2014 Oct; 34 ( 7 ): 826 - 40 .
6. Thielen FW , Van Mastrigt G , Burgers LT , et al. How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3) . Expert Rev Pharmacoecon Outcomes Res . 2016 Dec; 16 ( 6 ): 705 - 21 .
7. van Mastrigt GA , Hiligsmann M , Arts JJ , et al. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3) . Expert Rev Pharmacoecon Outcomes Res . 2016 Dec; 16 ( 6 ): 689 - 704 .
8. Wijnen B , Van Mastrigt G , Redekop WK , et al. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3) . Expert Rev Pharmacoecon Outcomes Res . 2016 Dec; 16 ( 6 ): 723 - 32 .
9. Shamseer L , Moher D , Clarke M , et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation . BMJ. 2015 Jan 02 ; 349 : g7647 .
10. Glanville J , Fleetwood K , Yellowlees A , et al. Development and testing of search filters to identify economic evaluations in MEDLINE and EMBASE . Ottawa: Canadian Agency for Drugs and Technologies in Health; 2009 .
11. Lee E , Dobbins M , Decorby K , et al. An optimal search filter for retrieving systematic reviews and meta-analyses . BMC Med Res Methodol . 2012 Apr 18 ; 12 : 51 .
12. Drummond MF , Sculpher MJ , Claxton K , et al. Methods for the economic evaluation of health care programmes . Oxford: Oxford University Press; 2015 .
13. Page MJ , Shamseer L , Altman DG , et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a crosssectional study . PLoS Med . 2016 May; 13 ( 5 ): e1002028 .
14. Shea BJ , Grimshaw JM , Wells GA , et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews . BMC Med Res Methodol . 2007 Feb 15 ;7: 10 .
15. Moher D , Liberati A , Tetzlaff J , et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Ann Intern Med . 2009 ; 151 ( 4 ): 264 - 9 . w64