Capturing Budget Impact Considerations Within Economic Evaluations: A Systematic Review of Economic Evaluations of Rotavirus Vaccine in Low- and Middle-Income Countries and a Proposed Assessment Framework
Capturing Budget Impact Considerations Within Economic Evaluations: A Systematic Review of Economic Evaluations of Rotavirus Vaccine in Low- and Middle-Income Countries and a Proposed Assessment Framework
Natalie Carvalho 0 2 3 4 5 6 7 8
Mark Jit 0 1 2 3 4 5 6 7 8
Sarah Cox 0 2 3 4 5 6 7 8
Raymond C. W. Hutubessy 0 2 3 4 5 6 7 8
0 Department of Infectious Disease Epidemiology, London School of Hygiene and Tropical Medicine , London , UK
1 0 Avenue Appia , 1211 Geneva 27 , Switzerland
2 Centre for Health Policy and Global Burden of Disease Group, School of Population and Global Health, University of Melbourne , Melbourne, VIC , Australia
3 & Raymond C. W. Hutubessy
4 Initiative for Vaccine Research, World Health Organization
5 Saw Swee Hock School of Public Health, National University of Singapore and National University Hospital System, Singapore , Singapore
6 Center for Economic and Social Research, University of Southern California , Los Angeles, CA , USA
7 Initiative for Vaccine Research, World Health Organization, Program in Applied Vaccine Experiences Scholar , Geneva , Switzerland
8 Modelling and Economics Unit, Public Health England , London , UK
Background In low- and middle-income countries, budget impact is an important criterion for funding new interventions, particularly for large public health investments such as new vaccines. However, budget impact analyses remain less frequently conducted and less well researched than cost-effectiveness analyses. Objective The objective of this study was to fill the gap in research on budget impact analyses by assessing (1) the quality of stand-alone budget impact analyses, and (2) the feasibility of extending cost-effectiveness analyses to capture budget impact.
Methods We developed a budget impact analysis checklist
and scoring system for budget impact analyses, which we
then adapted for cost-effectiveness analyses, based on
current International Society for Pharmacoeconomics and
Outcomes Research Task Force recommendations. We
applied both budget impact analysis and cost-effectiveness
analysis checklists and scoring systems to examine the
extent to which existing economic evaluations provide
sufficient evidence about budget impact to enable decision
making. We used rotavirus vaccination as an illustrative
case in which low- and middle-income countries uptake
has been limited despite demonstrated cost effectiveness. A
systematic literature review was conducted to identify
economic evaluations of rotavirus vaccine in low- and
middle-income countries published between January 2000
and February 2017. We critically appraised the quality of
budget impact analyses, and assessed the extension of
costeffectiveness analyses to provide useful budget impact
Results Six budget impact analyses and 60
cost-effectiveness analyses were identified. Budget impact analyses
adhered to most International Society for
Pharmacoeconomics and Outcomes Research recommendations, with
key exceptions being provision of undiscounted financial
streams for each budget period and model validation. Most
cost-effectiveness analyses could not be extended to
provide useful budget impact information; cost-effectiveness
analyses also rarely presented undiscounted annual costs,
or estimated financial streams during the first years of
Conclusions Cost-effectiveness analyses vastly outnumber
budget impact analyses of rotavirus vaccination, despite
both being critical for policy decision making.
Straightforward changes to the presentation of cost-effectiveness
analyses results could facilitate their adaptation into budget
Despite their equal importance in aiding decision makers to allocate limited resources, fewer budget impact analyses are published in the literature compared with cost-effectiveness analyses.
Furthermore, published budget impact analyses do
not meet current best-practice recommendations
such as not discounting future costs, providing
annual or budget relevant financial streams of costs,
model validation, and sensitivity and scenario
analyses among others.
The proposed framework through which
costeffectiveness analyses could be used as a tool to
provide useful budget impact analysis information
could facilitate the uptake and improvement of
goodquality budget impact analyses that would be useful
for decision makers, in particular in low- and
Cost effectiveness is one of several key gateway criteria
that inform new technology adoption decisions. However,
interventions found to be cost effective or very cost
effective are often not implemented [
]. While current
uncertainties surrounding appropriate cost-effectiveness
thresholds may explain some of this [
healthcare budget implications and issues of sustainability
also play an important role [
]. Decision makers require
estimates of the real financial consequences of introducing
a new intervention within a defined budget and budgetary
period rather than relying on anticipated savings in
economic costs alone [
1, 5, 6
]. The importance of high-quality
budget impact analyses (BIAs) for accurate budgeting and
resource allocation is increasingly recognised, especially
for severely resource-constrained environments [
Many high-income countries require budget impact
information alongside cost-effectiveness estimates when
making decisions to adopt new interventions [
7, 9, 10
lowand middle-income countries (LMICs), understanding the
short- and long-term impact of intervention adoption on
national budgets is critical for ensuring programme
], especially when international
funding is uncertain or of a temporary nature [
Unfortunately, the majority of published economic
evaluations are cost-effectiveness analyses (CEAs), with
very few BIAs, especially focused on LMICs [
impact analyses may be conducted as often but less
commonly published compared with CEAs for a variety of
reasons, including secrecy on behalf of industry not
wanting to disclose tender prices, or journal editors
believing BIAs to be more of an administrative or financial
task rather than research. Another possible explanation is
that appropriate methodology for a BIA remains less well
researched and less widely understood than for a CEA [
For instance, a previous review appears to classify nearly
all (57/68) economic evaluations of rotavirus vaccination
as BIAs if they provide an estimate of the net costs of
vaccination as part of a CEA, despite these estimated
economic (and often discounted) costs being very different
from the true financial stream of costs that would be
estimated within a BIA [
The first BIA analytic framework was published in 1998
]. Since then, additional publications have offered
methodological guidance for BIAs [
7, 10, 17–19
recently, the International Society for Pharmacoeconomics
and Outcomes Research (ISPOR) Task Force’s
best-practice recommendations on conducting BIAs [
]. The recent
ISPOR guidelines, however, do not include a checklist that
summarises the methodological recommendations and
supports a critical appraisal of the quality of BIAs. Such
checklists are useful in ensuring the quality of research
outputs. For example, CEAs are often appraised using
Drummond’s checklist for assessing economic evaluations
] or the Consolidated Health Economic Evaluation
Reporting Standards checklist [
In this study, we develop a framework that incorporates
the latest ISPOR recommendations into a BIA quality
assessment checklist. This framework can be turned into a
scoring system to critically assess the quality of a BIA.
While there have been previous quality appraisal checklists
in this area [
], none have been explicit about how to
use them to grade individual studies in particular
As an illustrative example, we develop a scoring system
for vaccines. Newer vaccines are more costly than
traditional childhood vaccines, representing increasingly large
]. Delivery costs can be as great as vaccine
costs, creating sustainability challenges over the long term
]. Although development assistance for vaccination has
increased over the last two decades, this is still a small
proportion of total health expenditures for vaccination in
]. In particular, for countries ineligible for
assistance through Gavi, the Vaccine Alliance or those
graduating out of Gavi support, the long-term budget
implications from new vaccine adoptions are very
important to consider. In fact, government expenditure on
vaccines in countries that have adopted new vaccines was
recently found to be on average double that of countries
that have not [
As CEAs are more common than BIAs, particularly in
], it would be helpful if they could be adapted to
provide budget impact information without requiring much
additional data or analysis. Such pragmatic adaptations
could be immediately useful to policy makers seeking to
fill the gap in stand-alone BIAs. This idea has been
proposed by others in relation to vaccine CEAs, based on how
population estimates are taken into account [
to the best of our knowledge, the feasibility of such
adaptation has never been systematically assessed, and
there is no existing guidance on preparing CEAs with such
an objective in mind. We therefore also modify the BIA
checklist and scoring system for CEAs, to evaluate
whether, and if so, the extent to which, a CEA can be used to
provide sufficient information for a BIA.
We first systematically review the literature to identify
existing economic evaluations in LMICs. To keep the
review manageable, we focus on rotavirus vaccination,
which is recommended by the World Health Organization
for all countries, particularly those with high rotavirus
gastroenteritis case-fatality risks [
]. Most of the
estimated 215,000 deaths in children under 5 years of age from
rotavirus gastroenteritis occur in LMICs [
], and while
there exist many studies demonstrating the cost
effectiveness of rotavirus vaccination [
], it has seen only
partial uptake in LMICs. We then apply the checklists and
scoring systems to examine the extent to which existing
economic evaluations provide sufficient evidence about
budget impact to enable decision making, and identify key
areas where BIA and CEA reporting could be improved for
2.1 Search Strategy
We conducted a systematic literature review to identify
economic evaluations of rotavirus vaccine in LMICs. Our
search was conducted in November 2015 and updated in
February 2017. We included articles published between
January 2000 and February 2017. Search terms were
broken down into four components, which included terms
relating to (1) costs/BIA/CEA/economic evaluation, (2)
vaccines, (3) rotavirus and (4) LMICs [see Online
Resource 1 of the Electronic Supplementary Material
(ESM) for full details on search terms used].
Relevant Medical Subject Headings terms were included
where appropriate. We developed and used an LMIC filter
based on the most recent World Bank country income
], expanded to include 25 countries that
transitioned from low or middle income to high income
from 2000 to 2017. We supplemented our LMIC filter with
the Cochrane 2012 LMIC Filters [
] (Online Resource 2
of the ESM).
We searched the following electronic databases:
EconLit, EMBASE (Ovid), MEDLINE (Ovid) and the National
Health Service Economic Evaluation Database (through
the Cochrane Library). Additionally, we ran basic searches
in Research Papers in Economics and the Tufts CEA
]. We also carried out searches of the first 150
hits in Google and Google Scholar using BIA-specific
search terms to identify any unpublished BIAs (Online
Resource 1 of the ESM). We supplemented the database
searches by conducting manual bibliographic searches
from recent and relevant rotavirus vaccine review papers
14, 15, 29, 33–35
Duplicate citations were removed and all remaining
papers were screened based on title and abstract. Two
reviewers (NC and SC) screened titles and abstracts during
the original search conducted in November 2015. One
reviewer (NC) screened titles and abstracts identified in the
updated search conducted in February 2017. We verified
inclusion as a LMIC based on the World Bank’s fiscal year
at the time of publication. Two reviewers (NC and SC)
read the full text of papers identified as relevant.
NonEnglish language papers were translated to English. Studies
included consisted of those self-defined as a BIA or other
economic evaluation including a CEA and cost-utility
analysis (we refer to both types as CEA from now on).
Studies classified as a cost-benefit, fiscal or revenue
analysis, or costing studies that did not capture both the costs of
rotavirus vaccination and the cost savings owing to reduced
disease, were excluded, as were papers tagged as Review,
Editorial, Perspective or Discussion pieces. Full inclusion
and exclusion criteria are summarised in Table 1. We
followed the Preferred Reporting Items for Systematic
Reviews and Meta-analysis guidelines and a checklist for
the review [
2.2 Development of a Budget Impact Analysis (BIA)
Based on the ISPOR best-practice recommendations [
we produced a framework to provide guidance on assessing
the quality of BIAs. The BIA Checklist consisted of 15
items divided into four categories: Background,
Interventions, Analytic Framework and Results (Table 2). From this
framework, we developed a vaccine-specific scoring
system to critically appraise papers. Each item was assigned a
Publisheda before 2000
Focused on high-income country(ies)
Target population children over 5 years of age or unspecified
Abstracts, posters or presentations
1. Fiscal, revenue or cost-benefit analyses, or costing studies that do not capture
(a) Costs of rotavirus vaccination; and
(b) Cost savings
2. Studies that do not report either:
(a) Novel analysis from primary data collection; or
(b) Secondary analysis or modelling of primary data
3. Review, editorial, perspective or discussion pieces
BIA budget impact analysis, CEA cost-effectiveness analysis, CUA cost-utility analysis, LMIC low- and middle-income country, N/A not
a If unpublished, refers to date reported
b Based on World Bank Analytical Classification using gross national income per capita (Atlas methodology), based on the World Bank’s fiscal
year at the time of publication
full (1), partial (0.5) or null (0) score, based on how closely
the article met the relevant recommendation. Strict scoring
rules were followed for each item, as specified in Online
Resource 3 of the ESM, and described in the following
For this category, relevant features of the healthcare system
that may influence the budget must be considered. We
identified five such features: financing available, budget for
vaccines, the country’s decision to introduce the new
vaccine, rotavirus disease burden and other relevant
healthcare system factors such as availability of
infrastructure. The recommended perspective is that of the
decision maker or budget holder. Finally, the size of the
eligible population must be described and data sources or
approaches used to estimate population size explained.
Articles should describe the current mix of interventions
and the expected mix after the introduction of the new
vaccine. This includes identifying all cost categories
relevant to the current mix of interventions, including
outpatient cases and hospitalisation. To receive a full score, we
specified that costs must be estimated using microcosting.
For pragmatic reasons, a study that uses local
reimbursement rates (or reference costs) from the country (for
example, based on a basic benefits package) was
considered equivalent to microcosting. Second, the anticipated
uptake and coverage of the new vaccine must be
considered. For a full score, the article must discuss where
coverage estimates come from, why they are reasonable, and
their reason for modelling or not modelling scale-up. Third,
all cost categories included in estimating the cost of
vaccine introduction should be identified. This includes
microcosting of operational delivery and administration
costs of the vaccination programme in addition to
specifying the vaccine procurement cost. Finally, the impact on
healthcare costs should be modelled, including a
description of how this was done.
2.2.3 Analytic Framework
Within this category, we assessed aspects related to
modelling choices and data inputs, including stating and
justifying a time horizon appropriate to the budget holder, not
discounting costs, and providing full details of the model
used and input parameters. If an article makes no mention
of discounting costs, we assumed no discounting was used.
We generated a list of the six most relevant data inputs:
demography; estimated vaccine coverage; burden of
disease; vaccine efficacy; vaccine-related costs; and other
health systems-related costs. Articles were scored based on
how much local level data were used to inform these
Impact on healthcare
Discounting and time
Uncertainty and scenario
Classification for modified BIA checklist
Desirable: describe characteristics/justify
population size estimates
Essential: health systems costs included
Desirable: microcosting of healthcare
Essential: report coverage level of new
Desirable: discussion of where coverage
estimates come from
Desirable: microcosting of intervention
Essential: model impact of intervention
Desirable: describe how intervention
impact was modelled
Essential: time horizon stated
Desirable: time horizon justified
Essential: report model used
Desirable: describe model used
Estimate size of eligible population, and distribution of any
characteristics that may influence budget impact
CEA cost-effectiveness analysis
a Each item gets a maximum score of 1 and a minimum score of 0. Items subdivided into ‘Essential’ and ‘Desirable’ components are scored 0/0.5
for each sub-component such that the maximum still sums to 1 for each item. Maximum score for the full checklist is 15 points. Maximum score
for Essential vs. Desirable items in the modified BIA checklist for CEA are six and nine points, respectively
b Adapted from Sullivan et al. (2014) (International Society for Pharmacoeconomics and Outcomes Research Task Force on Good Research
Practices—Budget Impact Analysis)
A description of how the intervention’s impact on healthcare
costs was modelled should be included, including estimation
of indirect effects where relevant
State and justify the time horizon(s) over which costs and
consequences are being evaluated. Time horizon should be
appropriate to the budget holder
Financial streams at each budget period should be
undiscounted. Other aspects that vary over time (inflation/
deflation, changes in price) should be included
Describe and justify the specific type of model used
Specify data sources and, if possible, obtain estimates directly
from budget holders
Present results (both resource use and costs) for each budget
period after the new intervention is adopted
Determine face validity through: (1) agreement with relevant
decision makers on the computing framework, aspects
included, and how they are addressed; and (2) verification of
cost calculator or model implementation, including all
Present alternative scenarios (e.g. allow users to view results
with and without condition-related costs, to include or
exclude different categories of costs)
State main conclusions on the basis of the results of the BIA.
Report the main limitations regarding key issues including
assumptions and completeness and quality of data inputs and
Results must be presented as estimates of financial costs at
each budget period after the new vaccine was introduced.
This final category also assessed the determination of face
validity, the presence of uncertainty and scenario analyses,
and the article’s main conclusions from a budget impact
perspective, including discussion of the main limitations.
For these last three sub-sections within the results (validity,
uncertainty and scenario analyses, and conclusions), scores
do not take into account how well each of these items was
done, but rather provides guidance on whether they were
done at all.
2.3 Development of a Modified BIA Checklist for Cost-Effectiveness Analyses (CEAs)
We modified the BIA Checklist for CEAs by classifying
each item as ‘Essential’ or ‘Desirable’ for estimating
budget impact. Some items were subdivided into both an
‘Essential’ and ‘Desirable’ component. As a result, the
Modified BIA Checklist for CEAs consisted of ten Essential
components, and a further 12 Desirable components
(Table 2). The scoring system was similarly modified to
provide a ‘feasibility score’ to reflect a CEA’s suitability
for adaptation for BIA purposes (Online Resource 4 of the
ESM). Cost-Effectiveness Analyses were first scored on the
Essential criteria, and only articles receiving a full score
(6.5 points total) were assessed on the Desirable
components (8.5 additional points).
All articles included in the final review were scored
based on the BIA Checklist or the Modified BIA Checklist
for CEA scoring systems. The maximum score for both
checklists was 15 points. Two reviewers (NC and SC)
scored the articles independently. Scoring disagreements
across any of the items within the checklists were identified
and discussed by the two reviewers prior to jointly agreeing
on a score. If an agreement was not reached, the conflicting
scores were discussed with the broader team (all authors)
until a consensus was reached.
3.1 Articles Included in the Review
Our search yielded 834 articles, of which 305 were
duplicates. The 529 original articles were screened for inclusion
and exclusion criteria based on title and abstract. Two
reviewers read the full-text version of 103 studies, of which
an additional 37 articles were excluded. Sixty-six articles
were included in the final review, of which six were BIAs
and 60 were CEAs (Fig. 1).
The six BIAs were published between 2010 and 2013,
and focused on countries across different regions of the
world (Indonesia, Thailand, Armenia, Brazil), with one that
evaluated rotavirus vaccine across all Gavi-eligible
countries (Online Resource 5 of the ESM). Five BIAs were
conducted alongside a CEA.
The CEAs (Online Resource 5 of the ESM) were
published between 2005 and 2016, with the largest numbers of
studies published in 2009 and 2015 (12 in each year). The
CEAs focused on individual LMICs around the world as
well as groups of countries.
3.2 Assessment of Papers Reviewed
3.2.1 Application of the BIA Checklist
184.108.40.206 Background All BIAs described the burden of
rotavirus disease and whether a decision had been made to
introduce the vaccine in the country. Most (4/6) [
adequately described with justification the estimated target
population. All analyses were conducted from the decision
maker or budget holder’s perspective. Most BIAs fully
considered the relevant features of the healthcare system
that could influence the budget, and reported on the current
budget for existing vaccines as well as financing available
for the introduction of the new vaccine, such as whether the
country is eligible for Gavi support and, if so, when this
support would end.
220.127.116.11 Interventions All but one  BIA laid out the
current mix of interventions and expected mix after the
introduction of the new vaccine. Half [
] took into
account the anticipated uptake of the new vaccine,
including describing the estimated scale-up of the vaccine
over time with justification based on prior vaccine scale-up
trends or other sources. Most (4/6) BIAs [
all relevant cost categories and described the approaches
used to estimate the costs of introducing the new vaccine.
Some articles were penalised for not microcosting to
estimate costs associated with the current interventions (1/6)
 or the new vaccine (2/6) [
], or not justifying the
coverage level at which the vaccine would be rolled out or
discussing scaling up (3/6) [
37, 41, 42
]. All BIAs modelled
the impact of the vaccine on health care costs and
described how this was done.
18.104.22.168 Analytic Framework All BIAs specified either
the programmatic time horizon or stated that they were
modelling one birth cohort; however, only two justified this
time horizon from the perspective of the budget holder
]. All BIAs accounted for time dependencies of
costs; however, three discounted future costs at 3%
37, 41, 42
]. All articles described the model and input
parameters, and used budget holder or local
countryspecific data sources to inform model input parameters.
22.214.171.124 Results Only two BIAs presented financial cost
estimates and resource use at each budget period after the
new vaccine is introduced [
]. One BIA presented
financial cost estimates only ; the remaining appeared
to present annual total costs based on a mature
programme operating at full scale [
37, 41, 42
Determination of face validity was present in some capacity in four
of six articles: one conducted a partial verification of
model projections by comparing two different models
]; three compared the financial cost estimates for one
budget period to the current immunisation budget in the
37, 41, 42
]. With one exception [
], all BIAs
carried out uncertainty and scenario analyses. All but one
stated the main conclusions from a budget impact
], including discussion of the main
126.96.36.199 Overall BIA Quality Assessment Scores Among
all BIAs, validity, time dependencies and discounting, and
programmatic time horizon received the lowest average
scores across all checklist items. Average scores for the
BIA papers across each of the 15 checklist items are shown
in Fig. 2. Overall, the total score received by BIAs ranged
from 10.5 to 13.5 (out of a maximum of 15 points).
Articlespecific scores for each checklist item are shown in Online
Resource 5 of the ESM.
3.2.2 Application of the Modified BIA Checklist for CEA
188.8.131.52 Background The majority of CEAs reviewed (53/
60) were conducted from the perspective of the decision
maker or budget holder. Many (38/54) also included some
form of a societal perspective. Of the seven CEAs not
conducted from the healthcare payer or provider
perspective, most included household out-of-pocket costs that
could not be separated from the healthcare systems costs.
The size of the eligible population was specified or data
sources provided in most (53/60) CEAs.
184.108.40.206 Interventions All CEAs performed well in this
category. All articles received a full score for laying out the
current mix of interventions and expected mix after the
introduction of the new vaccine, taking into account the
anticipated uptake of the new vaccine, and modelling the
impact on healthcare costs. Most (56/60) CEAs adequately
identified all major cost categories relevant to the new
220.127.116.11 Analytic Framework The programmatic time
horizon was clear in all but one CEA. All reported on the
model used for the analysis. In many cases (14/60), it was
unclear whether future costs were discounted or not. Either
nothing was stated or discounting was only mentioned in
the context of disease burden. The majority of articles that
discounted costs used a 3% discount rate. Less than half of
all studies (18/60) presented costs undiscounted or
presented both discounted and undiscounted costs. While
some studies also specified varying the discount rate in
sensitivity analyses (usually assuming no discounting as a
lower bound), most did not adequately show their results
(total costs, or programme costs plus cost savings)
18.104.22.168 Results The cost estimates/budget impact
category scored lowest of all items included in the modified
checklist. Only four of 60 CEAs received a full score,
while five articles received a partial score. Most articles
presented the total costs and/or resource use for the entire
period modelled, rather than disaggregating it year by year.
Furthermore, most articles did not present the true
timevarying financial costs of rolling out the vaccine and
subsequently scaling up incrementally, instead choosing to
report annualised costs based on a mature programme
operating at full scale.
22.214.171.124 Overall BIA Feasibility Scores for CEAs Average
BIA feasibility scores across all articles for the nine
essential items are shown in Fig. 3. The two worst
performing categories were ‘Discounting and time
dependencies’, and ‘Cost estimates/budget impact’. Overall
feasibility scores across all essential items ranged from 3 to
6.5, with an average of 4.7 out of 6.5 possible points. Three
papers scored full points (6.5/6.5) on the essential items
and were subsequently scored on the desirable items. Final
scores for the complete checklist ranged from 12 to 13.5
out of 15 possible points for these three CEAs (Online
Resource 5 of the ESM).
The majority of economic evaluations of rotavirus vaccine
in LMICs identified in this review were CEAs. Only six of
66 articles were classified as a BIA, and all but one were
conducted alongside a CEA. The BIAs adhered well to the
majority of ISPOR’s best-practice recommendations.
However, our review identified three areas where BIAs fell
short from current best-practice recommendations: not
discounting future costs, providing annual or budget
relevant financial streams of costs and model validation. Most
BIAs were conducted alongside a CEA, and intervention
costs were often handled in a manner more appropriate to
CEA rather than BIA in terms of discounting and reporting.
Previous reviews of BIAs have determined that good
methods are not used consistently, with discounting,
reporting and sensitivity analyses also reported as key
issues in need of improvement, among others
14, 19, 43, 44
The majority of existing rotavirus CEAs did not provide
adequate detail to be useful for budget impact assessment.
Only three of 60 CEAs passed the ‘minimum requirements’
to be assessed using the full checklist. Similar to the BIAs,
CEAs rarely present undiscounted annual costs, or estimate
financial streams during the first few years of programme
scale-up. One pragmatic recommendation for CEAs (and
those who commission them) could therefore be to
explicitly require the reporting of yearly undiscounted
costs in addition to the standard reporting of costs within a
CEA (possibly in an appendix if not directly relevant for
the cost-effectiveness results), and estimates of real
financial streams during the first few years of programme
There are important limitations of this study that merit
mentioning. First, in constructing a checklist and scoring
criteria, we have simplified the recommendations laid out
by the ISPOR Task Force. While we attempted to capture
the most important considerations addressed in the
bestpractice recommendations, the categories included in the
checklist are not an exhaustive list. For example, our
checklist does not capture whether the model allows users
to input or vary parameter values, an important
consideration for BIAs according to the guidelines. Additionally,
some categories included in the checklist can be subjective,
for example, stating appropriate conclusions and
limitations from a budgetary perspective. As such, in developing
the scoring system, we do not consider the quality and
completeness of each of the items in the checklist. Instead,
we have developed the scoring system to be as explicit as
possible for scoring vaccine-specific articles.
We focused only on full economic evaluations in this
review. Thus, we do not include vaccine costing studies
that may have tried to provide an estimate of budget impact
but did not take into account cost savings from reduced
burden on the healthcare system. Only a small number of
impact analysis papers reviewed: 11.9 (10.5, 13.5) out of 15 possible
points. Each category is worth 0, 0.5 or 1 point (maximum = 1)
BIAs was included in our review, limiting the
discriminatory power of our checklist and the generalisability of
our findings to BIAs conducted in areas outside of rotavirus
vaccine. Finally, it is important to reiterate that the
majority of papers reviewed were not intended to be formal
BIAs. As such, the ‘feasibility’ scores for CEAs do not
reflect an assessment of their quality as an economic
evaluation per se, but rather the extent to which they
provide useful budget impact information. This is especially
the case for articles scored only on Essential items. For
these items in particular, scores do not reflect how well, or
how appropriate for the budget holder this item was
handled in the analysis. Rather, they reflect whether or not the
item was taken into account or specified at all. For
example, CEAs specifying the size of the eligible population
score full marks for the essential component of this item,
regardless of whether the population size was appropriate
Despite these limitations, our study provides a simple
and easily implementable quality assessment checklist and
illustrates how it can be converted into a scoring system, in
this case for vaccine-specific BIAs. It is to the best of our
Fig. 3 Average modified checklist scoring for cost-effectiveness
analysis papers (n = 60). Average score (range in brackets) shown for
each ‘Essential’ category. Average overall feasibility score for all
cost-effectiveness analysis papers reviewed: 4.7 (3, 6.5) out of 6.5
knowledge, the first framework for systematically
evaluating the feasibility of adapting a CEA to a BIA. We
identified several areas where existing BIAs fell short from
best-practice recommendations. We also identified
relatively straightforward changes to the presentation of CEA
results that could facilitate the adaptation of most
costeffectiveness studies into BIAs, potentially increasing their
utility and application. Although we have used produced a
vaccine-specific scoring system, the checklist could
similarly be adapted to other types of interventions.
To assist countries in making informed decisions about
adding a vaccine to a national immunisation programme,
the World Health Organization emphasises that
programmatic, economic and financial feasibility are important
criteria among others to consider [
]. This is especially
true in the case of LMICs not eligible for financial support
or price negotiations through Gavi and those graduating out
of Gavi support [
]. For these countries, the real fiscal costs
of introducing and sustaining new vaccines within
immunisation schedules will have enormous implications on a
country’s immunisation budget and overall healthcare
budget. The World Health Organization’s global vaccine
and immunisation policy recommendations also
acknowledge budget impact considerations as being highly relevant
alongside cost effectiveness [
There is potential for improved decision making in the
area of vaccines through programmes such as the ProVac
possible points. Each category with a solid line is worth 0, 0.5 or 1
point (maximum = 1). Each starred category with a dashed line is
worth 0 or 0.5 points (maximum = 0.5)
Initiative, which provides economic evaluation tools
including a CEA model (TRIVAC) and BIA model
(COSTVAC) along with country-level support. [
Beyond vaccines, the ‘Gates Reference Case, recently
developed through funding by the Bill and Melinda Gates
Foundation with the aim to improve the quality of
economic evaluations conducted in LMICs, recommends
reporting on budget impact as one of 11 key principles
The limited number of published rotavirus vaccine BIAs
in LMIC settings compared with CEAs may reflect the
need for more guidance on best-practice methodologies for
BIAs of preventive interventions in these settings. In the
context of the current difficulties in determining
appropriate threshold values for cost-effectiveness ratios in LMICs,
there is potentially greater value for budget impact
estimates for reflecting opportunity costs for these countries.
Additional use and validation of the BIA quality
assessment framework proposed in this review is necessary and
would be useful for further work. For example, these
checklists and a vaccine-specific scoring system could be
applied to economic evaluations of other vaccines, and
results compared with the findings in this review. Further
research is needed to investigate how this framework could
be adapted to other public health or medical interventions.
Both budget impact and cost effectiveness are key criteria,
among others, for policy makers deciding how to allocate
limited resources. In our study, we propose and
demonstrate the use of a generalised BIA checklist and
vaccinespecific scoring system for guiding and assessing the
quality of BIA as a stand-alone analysis or alongside CEA,
and highlight current gaps in standard practices. The
proposed framework could facilitate the uptake and
improvement of the quality of BIAs useful for decision making, in
particular in LMICs.
Acknowledgements We thank Mabel Fong and Sucitro Sidharta
(National University of Singapore) for their valuable contributions as
research assistants. We also thank Tanji Hwang and Arun Bharatula
(University of Melbourne) for their research assistance. Raymond
C.W. Hutubessy is a staff member of the World Health Organization.
The views expressed here are his and not necessarily those of the
World Health Organization.
Author Contributions MJ and RH conceived the study. NC
developed the search strategy, inclusion and exclusion criteria, as well as
conducted the searches with input from SC, RH, MJ and JY. NC and
SC performed the initial screening of titles and abstracts. NC
developed the quality assessment checklist and scoring with input from SC,
RH, MJ and JY. NC and SC assessed full-text articles for inclusion in
review, with any discrepancies discussed by NC, SC, RH, MJ and JY.
NC and SC scored and abstracted data from all articles reviewed, with
any discrepancies discussed by NC, SC, RH, MJ and JY. NC and SC
wrote the report with input from all authors. All authors approved the
final version of the report.
Compliance with Ethical Standards
Funding Natalie Carvalho was funded by the University of
Melbourne McKenzie Post Doctoral Fellowship Scheme. Mark Jit and
Joanne Yoong were funded by the World Health Organization and
Gavi, the Vaccine Alliance. Sarah Cox was funded by the Bill and
Melinda Gates Foundation, the Johns Hopkins Vaccine Initiative and
the Bloomberg School of Public Health through the Program in
Applied Vaccine Experiences. No other funding was received for this
Conflict of interest Natalie Carvalho, Mark Jit, Sarah Cox, Joanne
Yoong and Raymond C.W. Hutubessy have no conflicts of interest
directly relevant to the content of this article.
Data availability statement This article is a systematic review of the
literature, and thus there are no underlying data used for this research
apart from the data extracted from the articles included in this review.
A list of all articles included in the review is available in the
Electronic Supplementary Material (Online Resource 5). Data extracted
from all budget impact analyses used to inform the budget impact
analysis checklist scores are available in the Electronic
Supplementary Material (Online Resource 5). Data extracted from
cost-effectiveness analyses used to inform the budget impact analysis checklist
for cost-effectiveness analysis scores are available from the
corresponding author upon reasonable request.
Open Access This article is distributed under the terms of the
Creative Commons Attribution-NonCommercial 4.0 International
License (http://creativecommons.org/licenses/by-nc/4.0/), which
permits any noncommercial use, distribution, and reproduction in any
medium, provided you give appropriate credit to the original
author(s) and the source, provide a link to the Creative Commons
license, and indicate if changes were made.
1. Newall AT , Jit M , Hutubessy R . Are current cost-effectiveness thresholds for low- and middle-income countries useful? Examples from the world of vaccines . Pharmacoeconomics . 2014 ; 32 ( 6 ): 525 - 31 .
2. Shillcutt SD , Walker DG , Goodman CA , Mills AJ . Cost effectiveness in low- and middle-income countries: a review of the debates surrounding decision rules . Pharmacoeconomics . 2009 ; 27 ( 11 ): 903 - 17 .
3. Glassman A , Canon O , Silverman R . How to get cost-effectiveness analysis right? The case of vaccine economics in Latin America . Value Health . 2016 ; 19 ( 8 ): 913 - 20 .
4. Makinen M , Kaddar M , Molldrem V , Wilson L. New vaccine adoption in lower-middle-income countries . Health Policy Plan . 2012 ; 27 ( Suppl . 2): ii39 - 49 .
5. Hoffmann C , Graf von der Schulenburg JM . The influence of economic evaluation studies on decision making: a European survey . The EUROMET Group. Health Policy . 2000 ; 52 ( 3 ): 179 - 92 .
6. Jit M , Hutubessy R . Methodological challenges to economic evaluations of vaccines: is a common approach still possible? Appl Health Econ Health Policy . 2016 ; 14 ( 3 ): 245 - 52 .
7. Trueman P , Drummond M , Hutton J . Developing guidance for budget impact analysis . Pharmacoeconomics . 2001 ; 19 ( 6 ): 609 - 21 .
8. Carroll S , Garc´ıa Rojas AJ , Glennga˚rd AH, Marin C . Vaccination: short- to long-term benefits from investment . J Mark Access Health Policy . 2015 ; 12 : 3 .
9. Garattini L , van de Vooren K . Budget impact analysis in economic evaluation: a proposal for a clearer definition . Eur J Health Econ . 2011 ; 12 ( 6 ): 499 - 502 .
10. Nuijten MJ , Mittendorf T , Persson U . Practical issues in handling data input and uncertainty in a budget impact analysis . Eur J Health Econ . 2011 ; 12 ( 3 ): 231 - 41 .
11. Ngcobo NJ , Cameron NA . The decision making process on new vaccines introduction in South Africa . Vaccine . 2012 ; 30 ( Suppl . 3): C9 - 13 .
12. Mori AT , Norheim OF , Robberstad B . Budget impact analysis of using dihydroartemisinin-piperaquine to treat uncomplicated malaria in children in Tanzania . Pharmacoeconomics. 2016 ; 34 ( 3 ): 303 - 14 .
13. Marseille E , Larson B , Kazi DS , Kahn JG , Rosen S . Thresholds for the cost-effectiveness of interventions: alternative approaches . Bull World Health Organ . 2015 ; 93 ( 2 ): 118 - 24 .
14. Orlewska E , Gulacsi L . Budget-impact analyses: a critical review of published studies . Pharmacoeconomics . 2009 ; 27 ( 10 ): 807 - 27 .
15. Aballea S , Millier A , Quilici S , et al. A critical literature review of health economic evaluations of rotavirus vaccination . Hum Vaccin Immunother . 2013 ; 9 ( 6 ): 1272 - 88 .
16. Mauskopf J . Prevalence-based economic evaluation . Value Health . 1998 ; 1 ( 4 ): 251 - 9 .
17. Mauskopf JA , Sullivan SD , Annemans L , et al. Principles of good practice for budget impact analysis: report of the ISPOR Task Force on good research practices: budget impact analysis . Value Health . 2007 ; 10 ( 5 ): 336 - 47 .
18. Sullivan SD , Mauskopf JA , Augustovski F , et al. Budget impact analysis-principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force . Value Health . 2014 ; 17 ( 1 ): 5 - 14 .
19. Mauskopf J , Earnshaw S. A methodological review of US budget-impact models for new drugs . Pharmacoeconomics . 2016 ; 34 ( 11 ): 1111 - 31 .
20. Drummond M , Sculpher MJ , Claxton K , Stoddart GL , Torrance GW . Methods for the economic evaluation of health care programmes . 4th ed. Oxford: Oxford University Press; 2015 . p. 42 - 44
21. Husereau D , Drummond M , Petrou S , et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement . Value Health . 2013 ; 16 : e1 - 5 .
22. Patented Medicine Prices Review Board . Budget impact analysis guidelines: guidelines for conducting pharmaceutical budget impact analyses for submission to public drug plans in Canada . 2007 . http://www.pmprb-cepmb.gc.ca/cmfiles/bia-may0738lvv5282007 -5906.pdf. Accessed 9 Sept 2017 .
23. Me ´decins Sans Frontie`res. The right shot: bringing down barriers to affordable and adapted vaccines . 2nd ed. 2015 . http://cdn. doctorswithoutborders.org/sites/usa/files/attachments/the_right_ shot_2nd_edition.pdf. Accessed 9 Sept 2017 .
24. Brenzel L. What have we learned on costs and financing of routine immunization from the comprehensive multi-year plans in Gavi eligible countries? Vaccine . 2015 ; 33 ( Suppl . 1): A93 - 8 .
25. Haakenstad A , Birger M , Singh L , et al. Vaccine assistance to low- and middle-income countries increased to $3.6 billion In 2014 . Health Aff (Millwood). 2016 ; 35 ( 2 ): 242 - 9 .
26. Mauskopf J , Talbird S , Standaert B . Categorization of methods used in cost-effectiveness analyses of vaccination programs based on outcomes from dynamic transmission models . Expert Rev Pharmacoecon Outcomes Res . 2012 ; 12 ( 3 ): 357 - 71 .
27. World Health Organization. WHO position paper, January 2013 . Wkly Epidemiol Rec. 2013 ; 5 : 49 - 64 .
28. Tate JE , Burton AH , Boschi-Pinto C , Parashar UD , World Health Organization-Coordinated Global Rotavirus Surveillance Network. Global , regional, and national estimates of rotavirus mortality in children \5 years of age, 2000 - 2013 . Clin Infect Dis . 2016 ; 62 ( Suppl . 2): S96 - 105 .
29. Tu HA , Woerdenbag HJ , Kane S , et al. Economic evaluations of rotavirus immunization for developing countries: a review of the literature . Expert Rev Vaccines . 2011 ; 10 ( 7 ): 1037 - 51 .
30. The World Bank. World Bank analytical classifications, country analytical history (Atlas methodology) . http://siteresources. worldbank.org/DATASTATISTICS/Resources/OGHIST.xls. Accessed 9 Sept 2017 .
31. The Cochrane Collaboration . Cochrane 2012 LMIC filters. 6 November 2015 . Available from: http://epoc.cochrane. org/lmicfilters. Accessed 5 Sept 2017 .
32. Cost-Effectiveness Analysis Registry . Available from: http:// healtheconomics.tuftsmedicalcenter. org/cear4/Home.aspx. Accessed 5 Sept 2017 .
33. Rheingans R , Amaya M , Anderson JD , Chakraborty P , Atem J . Systematic review of the economic value of diarrheal vaccines . Hum Vaccin Immunother . 2014 ; 10 ( 6 ): 1582 - 94 .
34. Ozawa S , Mirelman A , Stack ML , Walker DG , Levine OS . Costeffectiveness and economic benefits of vaccines in low- and middle-income countries: a systematic review . Vaccine . 2012 ; 31 ( 1 ): 96 - 108 .
35. Thiboonboon K , Santatiwongchai B , Chantarastapornchit V , Rattanavipapong W , Teerawattananon Y. A systematic review of economic evaluation methodologies between resource-limited and resource-rich countries: a case of rotavirus vaccines . Appl Health Econ Health Policy . 2016 ; 14 ( 6 ): 659 - 72 .
36. Moher D , Liberati A , Tetzlaff J , Altman DG , PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . BMJ . 2009 ; 339 : b2535 .
37. Muangchana C , Riewpaiboon A , Jiamsiri S , Thamapornpilas P , Warinsatian P . Economic analysis for evidence-based policymaking on a national immunization program: a case of rotavirus vaccine in Thailand . Vaccine. 2012 ; 30 ( 18 ): 2839 - 47 .
38. Jit M , Yuzbashyan R , Sahakyan G , Avagyan T , Mosina L . The cost-effectiveness of rotavirus vaccination in Armenia . Vaccine. 2011 ; 29 ( 48 ): 9104 - 11 .
39. Centenari C , Gurgel RQ , Bohland AK , Oliveira DM , Faragher B , Cuevas LE . Rotavirus vaccination in northeast Brazil: a laudable intervention, but can it lead to cost-savings? Vaccine . 2010 ; 28 : 4162 - 8 .
40. Kim SY , Sweet S , Slichter D , Goldie SJ . Health and economic impact of rotavirus vaccination in Gavi-eligible countries . BMC Public Health . 2010 ; 10 : 253 .
41. Suwantika AA , Postma MJ . Effect of breastfeeding promotion interventions on cost-effectiveness of rotavirus immunization in Indonesia . BMC Public Health . 2013 ; 13 : 1106 .
42. Suwantika AA , Tu HA , Postma MJ . Cost-effectiveness of rotavirus immunization in Indonesia: taking breastfeeding patterns into account . Vaccine . 2013 ; 31 ( 32 ): 3300 - 7 .
43. Mauskopf JA , Earnshaw S , Mullins CD . Budget impact analysis: review of the state of the art . Expert Rev Pharmacoecon Outcomes Res . 2005 ; 5 ( 1 ): 65 - 79 .
44. van de Vooren K , Duranti S , Curto A , Garattini L . A critical systematic review of budget impact analyses on drugs in the EU countries . Appl Health Econ Health Policy . 2014 ; 12 ( 1 ): 33 - 40 .
45. World Health Organization. Principles and considerations for adding a vaccine to a national immunization programme: from decision to implementation and monitoring . Geneva: World Health Organization; 2014 .
46. World Health Organization. Guidance for the development of evidence-based vaccination-related recommendations . Strategic Advisory Group of Experts on Immunization. Geneva: World Health Organization; 2017 .
47. Jauregui B , Janusz CB , Clark AD , et al. ProVac Global Initiative: a vision shaped by ten years of supporting evidence-based policy decisions . Vaccine . 2015 ; 33 ( Suppl . 1): A21 - 7 .
48. NICE International. The Gates reference case: what it is, why it's important, and how to use it . Bill and Melinda Gates Foundation Methods for Economic Evaluation Project . 2014 . Bill and Melinda Gates Foundation, NICE International, the Health Intervention and Technology Assessment Program (Thailand), and The University of York, Centre for Health Economics. https://www.nice.org.uk/Media/Default/About/what-we-do/ NICE-International/projects/Gates-Referencecase- what-it-ishow-to-use-it . pdf. Accessed 9 Sept 2017 .