Reflections on the Future of Pharmaceutical Public-Private Partnerships: From Input to Impact
Reflections on the Future of Pharmaceutical Public-Private Partnerships: From Input to Impact
Remco L. A. de Vrueh 0 1 2
Daan J. A. Crommelin 0 1 2
D SGC SME TI Pharma WHO 0 1 2
0 Research & development Structural genomics consortium Small and medium-sized enterprise Top institute pharma World health organization
1 Department of Pharmaceutics, Utrecht Institute for Pharmaceutical Sciences, UIPS Utrecht University , Utrecht , The Netherlands
2 Lygature , Utrecht , The Netherlands
3 Daan J. A. Crommelin
Public Private Partnerships (PPPs) are multiple stakeholder partnerships designed to improve research efficacy. We focus on PPPs in the biomedical/ pharmaceutical field, which emerged as a logical result of the open innovation model. Originally, a typical PPP was based on an academic and an industrial pillar, with governmental or other third party funding as an incentive. Over time, other players joined in, often health foundations, patient organizations, and regulatory scientists. This review discusses reasons for initiating a PPP, focusing on precompetitive research. It looks at typical expectations and challenges when starting such an endeavor, the characteristics of PPPs, and approaches to assessing the success of the concept. Finally, four case studies are presented, of PPPs differing in size, geographical spread, and research focus.
key drivers; performance evaluation; public-private-partnerships; R&D business models
AD Alzheimer’s disease
ADNI Alzheimer’s disease neuroimaging initiative
EMA European medicines agency
FDA Food and drug administration
IMI European innovative medicines initiative
PPP Public private partnerships
Bilateral R&D interactions between academic and
industrial scientists have a long tradition. In the last decade,
alongside these bilateral (‘vertical’) interactions, the
multiple stakeholder partnerships called Public-Private
Partnerships (PPPs) have emerged. These are R&D
networks that do not work on a one-to-one basis but rather
involve a range of stakeholders. As well as traditional
academia and industry stakeholders, such a PPP may
include charities, patient organizations, and even national
competent authorities (‘regulators’).
The scope of this review is a discussion of the role and
added-value of PPPs in facilitating precompetitive
multistakeholder collaborative research. First, we will focus on the
key drivers for different stakeholders as they embrace
the multi-stakeholder collaborative research model.
Second, we will provide insight into the challenges of
managing multi-stakeholder collaborative research
projects and how different PPPs have dealt with these
challenges. Third, we will elaborate more on current models
for monitoring and evaluating the performance of
precompetitive PPPs. Finally, given the growing number
of PPPs, we conclude by discussing their value for society
and the need to justify the public and private investments
being made (
SHIFT IN THE PHARMACEUTICAL BUSINESS
MODEL: FROM FULLY INTEGRATED TO OPEN
The process of bringing new treatments from Bbench to
bedside^ has been described as a translation continuum,
where various types of resources and areas of knowledge are
involved when moving from basic fundamental research to
proven clinical application and, eventually, medical practice
(and back) (
). Major advances in a.o. molecular biology,
functional genomics, and genetics have resulted in enhancing our
scientific understanding of disease pathophysiology (
Furthermore, the different ‘omics’ fields have enabled
largescale measurements of biological processes (
). Around the
turn of the century, the general expectation by the global
biomedical community, both public and private, was that
this increased knowledge base would fuel the
development of a new generation of innovative therapies,
impacting public health (
). Despite the promise, these
major advances in science have not yet increased the
success rate of moving a compound from lab bench to
first-in- man to approval and into medical practice (
In fact, the rate of development by pharmaceutical
companies of innovative medicinal products which were
approved by regulatory authorities slowed down (
indicating that a number of chasms continue to exist
in the translational process. One explanation for the high
rate of attrition in drug development mentioned in literature
is that industry is increasingly targeting more complex
diseases. Another is that regulatory authorities have become
increasingly demanding (
The pharmaceutical industry is already confronted with
the patent expiry of many blockbuster drugs, the so-called
‘patent cliff’ (
). The disappointing rate of development
attrition has been a strong motivator, especially for large
companies, to thoroughly review drug development strategies and
accompanying processes (
). Pharma companies know that if
the trend remains unaltered, costs of developing a drug will
continue to increase and ultimately render the drug
development process untenable (
). This would jeopardize the
longterm profitability that allows them to reinvest in future R&D.
As a first response, a wave of mergers and acquisitions has
taken place to save costs and to counter the weakening
pipeline. However, several studies have shown that these mergers
and acquisitions have not resulted in improvement of R&D
). What seems to have been a more
effective Bproductivity improvement^ strategy has been the
increased focus of the pharmaceutical sector on developing
medicinal products for rare diseases, so-called orphan drugs (
Recent 10-year approval/positive opinion overviews
presented by the FDA reveal an upward trend in the number of drug
approvals/positive opinions since 2010 (
). In 2015, almost
half of the approvals/positive opinions in the US were for
orphan drugs (
), compared with one third in 2005 (
similar trend is observed for Europe (
Alongside an increased focus on rare diseases, the role of
large pharmaceutical companies within the field of
pharmaceutical innovation has been changing over the past 2–3
). Companies continue to move away from the
traditional Bfully integrated discovery and development company^ model,
towards the adoption and implementation of various open
innovation strategies such as strategic alliances, purchase of
scientific services and in-licensing (
In the various models, companies tend to focus internally
more on late (phase II/III) clinical development and
distribution of products. At the same time, the open innovation
models allow them to access efficiently the best science
). Consequently, the discovery of potentially new
therapies and subsequent pre-clinical and early (phase I/IIa)
clinical evaluation have increasingly become the domain of
academic parties and small and medium-sized enterprises
(SMEs). Ultimately, the translation of basic scientific research
into a potential treatment modality is then driven by Bfully
integrated discovery and development networks^, in which expertise
and knowledge from academic parties, large pharmaceutical
companies, SMEs, and other stakeholders converge in
). As exemplified in Table 1, there is
increasing focus on better understanding open innovation
models in the bio-pharmaceutical sector and their impact on
R&D productivity. This paper specifically focuses on the
emergence of PPPs and their socio-economic impact.
TO SPUR BIOMEDICAL R&D
Collaboration in R&D between academia and the
pharmaceutical industry is not itself a novel concept. It has been
around for a while, with the aim of sustaining pharmaceutical
innovation and advancing new product development (
The US is a prime example, where the Bayh-Dole Act of 1980
is considered an important spur for numerous collaborations
between academic research groups and small biotech
companies or large pharmaceutical companies (
development is part of a shift that universities have made, especially
in the US and the EU, through which they consider
technology transfer and commercialization as an integral part of their
mission, a trend dubbed by D’Este and Perkman as the
BEntrepreneurial University^ (36).
University-industry collaboration within biomedical R&D
over the last decades has moved beyond vertical, bilateral
) such as contract research, consulting, and
research agreements with an option to license drug candidates
and technologies. It has evolved into more horizontal,
multistakeholder public-private partnerships (PPPs) (
important reason is that the multi-stakeholder PPP model
Organizational modes for Open
Innovation in the
An exploratory analysis
Measuring Open Innovation in the Michelino et al. (
Models for open innovation
in the pharmaceutical industry
Schuhmacher et al. (
Based on two rounds of interviews with 20 industry experts, a model describing the adoption
of inbound and outbound open Innovation by bio-pharmaceutical companies was presented
Bio-pharmaceutical firms employ a mix of open innovation modes (i.e. licensing agreements,
alliances and supply/provision of technical and scientific services) to engage with large
pharmaceutical companies, biotech firms and universities with the aim to acquire or
commercially exploit technologies and knowledge.
Focus on degree of openness of 126 global top R&D spending companies (Period 2008–2012)
Model built from three different perspectives: inbound versus outbound processes,
economic versus financial transactions and the nature of the traded entities: research
and development, intellectual property and know-how
Negative correlation of openness degree with firm age, dimension and efficiency, with biotech
companies being more open than pharmaceutical ones.
Based on an analysis of their R&D models, 13 multinational pharmaceutical companies were
categorized with respect to their preference in innovation management (introverted or
extroverted) and proportion of externally acquired R&D projects (low or high).
Based on the analysis and categorization, four types of open innovators were proposed:
‘knowledge creator’, ‘knowledge integrator’, ‘knowledge translator’ and ‘knowledge leverager’
Examples of Research Focusing on Better Understanding the Open Innovation Concept in The Bio-pharmaceutical Sector
generates the necessary common ground where governmental
bodies, universities, patient organizations, health foundations,
and the private sector can combine resources and expertise.
PPPs allow these stakeholders to address jointly the grand
challenges within the field of pharmaceutical innovation that
are of mutual interest. In simple terms, PPPs have the
potential to make things happen that would not be possible in their
Van Ham and Koppenjan defined multi-stakeholder PPPs
as Bthe cooperation of some sort of durability between public and private
actors in which they jointly develop products and services and share
risks, costs and resources which are connected with these products^
). Within the area of pharmaceutical R&D, there are
essentially two main types of multi-stakeholder collaborative
research initiatives or PPPs (
Product Development PPPs, which have positioned
themselves to address the global prosperity gap by developing
pharmaceutical solutions (vaccine, diagnostic, drug) for
low and middle income countries with an alternative
(not-for-profit) business model (
Precompetitive PPPs, which aim to generate novel
scientific concepts (e.g. disease targets and research
models) and infrastructures (e.g. databases) through
effective collaboration between multiple public and
private entities based on mutual trust, pooling of
complementary expertise and knowledge, and sharing of
rewards. Potential disputes over issues such as intellectual
property are avoided by limiting activities to the
precompetitive space (
Also important, and closely related to these two types of
PPPs, is the group of Access PPPs. These aim to improve
accessibility to a specific treatment modality through
massdrug administration programs, in order to alleviate disease
burden in low and middle income countries and to overcome
obstacles in the distribution system of treatments (
interesting initiative to increase patient access to expensive
biologics in low and middle income countries is the joint
development of affordable, high quality biosimilars by a consortium
of companies from low and middle income countries. This
development is undertaken in collaboration with academic
partners and coordinated by the Utrecht Centre for
Table 2 shows examples of precompetitive and product
According to Lim, the average number of
multistakeholder PPPs launched per year has grown from 8 in
2001–2003 to 54 in 2011–2013 (
). A major contributor to
this growth has been the launch of various high-profile
precompetitive public-private initiatives, involving multiple
consortia, such as the European Innovative Medicines
Initiative (IMI), the Dutch Top Institute Pharma, and the
US Foundation for National Institutes of Health. Faster
Cures has included in its consortiapedia catalogue more than
350 consortia profiles, both public-public as well as
The literature clearly indicates that the concept of PPP has
matured. A non-exhausting bibliometric analysis performed
through PubMed (searching by BPublic-Private Sector
Partnerships^[Mesh] OR Bpublic-private partnership^[TIAB]
OR Bpublic-private partnerships^[TIAB]) revealed over 2000
citations, of which more than 300 could be directly linked to
the concept of multi-stakeholder collaborative research. Closer
evaluation reveals that as successfully launched PPPs grew over
the last decade, the number of citations on the concept of
Bbiomedical R&D^ PPP also increased, from around 35
between 2001–2005 to around 225 in 2011–2015.
Critical Path Institute
Innovative Medicines Initiative
Top Institute Pharma
Structural Genomics Initiative
Medicines for Malaria Venture
Drugs for Neglected Diseases initiative
International AIDS Vaccine Initiative
Pediatric Praziquantel Consortium
To foster development of new evaluation tools and standards for drug
therapy trials, which accelerates regulatory qualification and medical
product approval and adoption.
To improve health by speeding up the development of, and patient
access to, innovative medicines, particularly in areas where there is an
unmet medical or social need.
To establish, support and manage public-private collaborations between
academia and the (inter-) national pharmaceutical industry to create
‘health & wealth’.
To catalyze research in new areas of human biology and drug discovery
research by focusing on less-well-studied domains of the human
To reduce the burden of malaria in disease-endemic countries by
discovering, developing and facilitating delivery of new, effective
and affordable antimalarial drugs.
To develop new drugs or new formulations of existing drugs for
people living with neglected diseases.
To ensure the development of safe, effective, accessible, preventive
HIV vaccines for use throughout the world.
To develop, register and provide access to a suitable pediatric praziquantel
formulation for treating schistosomiasis in preschool-age children.
Such a high number of citations demonstrates that the
concept of PPP has been adopted fully by funders, academia,
and the private sector as an important tool that unites two
intrinsically different systems of knowledge creation. Within
academia, the emphasis is on curiosity-driven, conceptual,
publicly accessible research with a long horizon. Within
industry, competitive advantage is pursued by shielding in-house
research findings and disclosure through patents (
intrinsic difference was once considered as being the core of
the obstacles to successful public-private collaboration in
biomedical R&D (
). Organizational aspects of
boundaryspanning activities between universities and industry (
as conflicts over IP and university administration (
also been mentioned by industry as an important barrier
towards successful collaboration with universities (
appears that PPPs in the precompetitive stage, where scientific
concepts are developed, offer a perfect level playing field for
all stakeholders to facilitate translational medicine, and
ultimately to improve pharmaceutical R&D productivity.
climate that would allow for such a strategy, public-private partnerships have
been proposed.^ (67). We fully agree with this statement, but full
exploitation of basic science into clinical utility also requires
more. It needs the lifting of existing boundaries between
scientific disciplines and stakeholders, resulting from traditions,
policies, and bureaucracies (
). Apart from an increased level of
Btransdisciplinarity^, exploitation depends on integrating the
contributions of the major stakeholders involved in translational
medicine: academia, governmental bodies, small and large
biopharmaceutical industry, health foundations, and patient
organizations. The successful integration of resources and expertise
from multiple stakeholders, especially between universities and
industry, is built on a clear understanding and alignment of
their missions and needs within biomedical R &D.
Furthermore, public and private stakeholders play different
roles in the biomedical research translation continuum (
an awareness of the intrinsic motivation of the various
stakeholders seeking a multi-stakeholder PPP collaborative model
will help to value properly public-private collaboration.
WHAT ARE THE MAIN DRIVERS FOR KEY
STAKEHOLDERS TO EMBRACE
THE MULTI-STAKEHOLDER PPP MODEL?
As former NIH director Elias Zerhouni put it, "The way forward
in multidisciplinary research is to engage in predictive, personalized,
preemptive and participatory medicine. For the creation of the optimal innovation
Public organizations, mainly universities, focus primarily on
technology push. Their footprint can be seen at all stages of the
research and development continuum. They are the major
providers of basic science concepts, and universities, especially
in the US and the EU, now consider commercialization of
their research activities an integral part of their mission, as
mentioned above. Although commercialization has been
identified as a reason for universities to engage with industry
), academics in particular are driven by a number of
motivators: 1) the opportunity to learn about industry
challenges and research activities, with feedback on applicability of
research and the chance to become part of a network; 2) access
to in-kind resources (materials, research expertise and
equipment); and 3) access to alternative sources of public and
The term Btechnology^ encompasses, among other things,
biochemical findings and new molecular entities, implying an
innovation which is pushed through R&D to the market.
Here, academic research groups act based on curiosity and
on their passion for performing hypothesis-driven research.
Crowley mentions the intrinsic level of unpredictability as an
important and also attractive characteristic of curiosity-driven
). There are various examples of important
solutions to clinical problems that were derived from basic science
that could never have been predicted at their outset, such as
the successful treatment of leukemia with Bcr-Abl tyrosine
kinase inhibitors and the clinical use of bisphosphonates (
Private entities, mainly from the industrial side, focus on
market pull. They react to the needs of patients through market
analysis, and have the appropriate resources, knowledge and
capabilities at their disposal to bring new, promising and often
innovative therapies to the market. As discussed above, the
pharmaceutical industry has increasingly embraced different
models of open innovation, including PPPs (
), because of
the need to target diseases of greater complexity, and because
they are faced with a high attrition rate in drug development.
Various scholars (
) have listed key incentives for
industry seeking collaboration with universities. These include
gaining access to basic knowledge; improving the ability to
solve problems; gaining access to new tools and techniques
for the development of new technologies; improving a firm’s
reputation in the labor market and among potential partners;
entering into the academic network; and exploiting
opportunities for public funding.
Several studies and comments have pointed out that
funding of clinical research by the pharmaceutical industry is
strongly associated with pro-industry results (
). We do
not take this concern about the influence of the
pharmaceutical industry on the development and use of medicines lightly.
However, we also believe that influence in the basic science
stage, which is by its nature precompetitive and
concept-driven, will be limited for several reasons. First, using data from
the Carnegie Mellon Survey on industrial R&D, and
investigating over 30 manufacturing industries, Cohen et al. showed
that the pharmaceutical industry is unique and highly
dependent on basic science (77). The latter has been confirmed by
other studies (
). Second, as summarized by Perkmann (
there are several studies showing that university-industry
collaboration doesn’t negatively impact university’s research
productivity, nor does it result in shifting away from basic science
and more towards applied research. Finally and perhaps most
importantly, as mentioned above, the intrinsic motivation of
academics to engage with industry goes beyond mere financial
). This is especially true in a PPP setting, in which
funding of public partners and SMEs is in general provided
by governmental bodies and/or health foundations (
Taking the various positions of both public and private entities
into consideration reveals that they are really interdependent
(80). As stated by Crowley (
), BAcademia and industry need each
other to effect substantive improvements in health. Without substantive
collaborations between the two, maximum advances in healthcare for the
US public are unlikely^.
Apart from universities and industry, other stakeholders have
also started to embrace this type of partnership to reach their
objectives. Health foundations or charities increasingly
demonstrate that PPPs can be instrumental in obtaining maximum
benefit for the patients they serve (
). In reviewing their
research portfolio, these organizations could see that there was
considerable progress towards advancing basic disease
understanding. However, it was also observed that translation of this
knowledge into clinical proof of concept was lagging
considerably, with a need for alternative strategies. Of course, health
foundations do not possess the budgets of larger
pharmaceutical companies or governmental funders (
), but, based
on a survey amongst 11 major US health foundations, Chang
identified a number of non-monetary value drivers that health
foundations consider of added value in a partnership (83).
First, health foundations can provide the necessary clinical
network and access to patients. Second, because of their
central position, foundations have relationships with and
unfettered access to the world’s leading academic experts who can
facilitate understanding of the basic science, but who can also
play a role in clinical trial design and evaluation. Third, as
already mentioned, foundations can help to attract additional
funding to a partnership, and place great emphasis on
collaboration and information sharing. Finally, although knowledge
sharing amongst partners in a PPP can be cumbersome due to
concerns about IP/trade secrets and/or other adverse
competitive consequences, health foundations have demonstrated
that they can act as a trusted intermediary or broker. They can
ensure proper storage, handling, analysis and/or mediated
revealing of information (e.g. aggregation and anonymization
of information before it is distributed within the PPP).
Numerous health foundations, such as the Cystic Fibrosis
) and the Polycystic Kidney Foundation (
were originally founded by patients (or their carers) with a key
objective: Bto find treatments and a cure^ for a specific disease or
group of diseases (
). A review of 75 Innovative Medicines
Initiative (IMI) projects reveals that European or international
patient associations (such as EURORDIS, the European
Patient Forum and the International Alliance of Patients’
Organizations) are participating in a total of 16 IMI PPPs
(88). Although not all 16 projects provide detail on
organizations’ roles in the work program, closer examination reveals
that patient participation occurs at three levels (
lowest level of participation is to support partnerships with the
dissemination of project results to patients and public.
Examples in this respect are the IMI projects EPAD (
and PharmaCog (
) projects. The next level of patient
participation, best exemplified by GetReal (
), Protect (
), is to ensure that the perspective of patients
and patients’ organizations is included from the start of the
work program via focus groups, patient input platforms, and
memberships to other project governance bodies. Finally,
EUPATI is a prime example of the highest level of patient
participation, a patient-led project. It focused on training close
to 100 patients or their representatives in all aspects of
medicines development, and on developing an extensive,
multilanguage training toolbox to be rolled out across Europe (
Finally, regulatory and health technology assessment agencies,
such as the EMA and the Dutch Medicines Evaluation Board,
also see the PPP model as a platform that allows them to share
data, expertise, and resources, and to enhance their
understanding of other stakeholders’ perspectives (
Nowadays, these public bodies participate in a variety of
PPPs, including those focusing on the development of new
evaluation tools and standards (Adapt-SMART, C-Path,
)); training programs in drug regulatory
sciences e.g. PharmaTrain (101); and new approaches for
incorporating real life data into drug development (e.g. GetReal
HOW TO EVALUATE THE PERFORMANCE
& IMPACT OF PRECOMPETITIVE RESEARCH
With so many PPPs launched in the last two decades, we agree
with Freire that it is time to Bmove beyond the buzzword^ of PPPs
as engines that simply facilitate the translation of basic
fundamental research, looking also at their impact on proven
clinical application and eventually, medical practice (
Like biomedical research in general, there is a growing
demand to better understand the socio-economic value of
biomedical R&D PPPs (
). Apart from understanding
socio-economic value to ensure continued funding
support, other identified reasons for this move towards
evaluating the (research) impact of PPPs include the
requirement for accountability. Research organizations
need to judge and manage their performance and to
improve understanding of which research (model)
ultimately leads to the desired impact (103).
In the last decade, various scholars have independently
investigated indicators to evaluate the performance of
research PPPs (
) and of university-industry alliances in
). As depicted in Table 3, the proposed
frameworks for a performance measurement system are
comparable, and follow the basics of the logical framework method.
This involves defining objectives and securing the necessary
inputs through to producing a concrete output and eventually
generating the desired impact (107). The adoption of
unambiguous definitions for each stage, generally accepted by all
stakeholders, will be essential, because this ensures a clear
distinction between input, process, output, short term (or
intermediate) outcomes and long term outcomes/impact.
Building on existing literature (
), we propose
the definitions that follow below as a starting point for further
The input stage revolves around Bthe PPP’s ability to bring
together human, financial and physical resources, public and private
researchers’ capabilities and motivation, and the partners’ knowledge and
experience^. These inputs provide the basis for the following
process stage, which reflects Bthe PPP’s ability to pursue a
highquality research program, as well as educational and/or other services that
are relevant to all stakeholders, and to create an environment of genuine trust
that allows ample sharing of knowledge and resources throughout the
PPP^. The process stage will result in the generation of various
concrete outputs or Bthe PPP’s ability to generate immediate tangible
scientific knowledge, products and/or services^. Depending on Bthe
PPP’s ability to encourage their use in the short term^, these outputs
will translate into short term (or intermediate) outcomes.
Finally, long term outcome/impact can be defined as Bthe longer
term economical and health impact of the PPP’s generated output, for
society, patient as well as the stakeholders involved^.
As depicted in Table 4, a number of performance
indicators have been suggested for each stage of the proposed logical
framework, to allow the implementation of PPP performance
measurement in practice (
). A number of indicators
are also included for biomedical research output and outcome
that are appropriate for research PPPs (
). Finally, building
on top of work by Denee et al. and Thonon et al. (
suggest classifying the indicators into five categories or
domains: Networks and collaboration; research activity and
Logical framework key stages
Perkmann et al. (62)
Denee et al. (
Innovative Medicines Initiative (
# Resources, processes and facilities that can link inputs to outcomes
knowledge; knowledge sharing and dissemination; human
capital; and financials and operations.
Establishing a framework, a clear definition of the key
stages, and a set of performance indicators provides a good
starting point for implementation of a meaningful PPP
performance measurement system. However, several challenges,
especially around outcomes and long-term impact, have been
reported that will need to be taken into consideration to
ensure that a PPP performance measurement system is
implemented that is endorsed by all the stakeholders involved.
To start with, the lengthy timeline associated with
pharmaceutical R&D, combined with research becoming more and
more multidisciplinary in nature (and consequently involving
many different processes, individuals, and organizations), makes
it challenging to attribute directly a specific contribution from a
group or PPP to a specific health or socio-economic impact
). On top of this, ‘knowledge creep’ is considered a
challenge when assessing the impact of research, because policy
makers have a tendency to respond slowly to translating
accumulating evidence into policy and/or guidelines, and often do
so without recognition of the contributing research (112).
In line with Penfield et al. (
), given that the type of
impact stakeholders anticipate varies according to the type
and scope of a PPP, impact-specific challenges may prohibit
a fair comparison of impact between PPPs. Linking the results
of a clinical trial (output) to clinical benefit (outcome) will
obviously be more straightforward than discovery of a new
drug candidate or a specific training program to clinical
). A recent systematic review by Thonon et al. revealed a
total of 57 indicators to measure output and outcome of
medical research. Most identified indicators were in fact the many
different indexes (e.g. h-index) currently available to measure
research production and impact. With research being
complex, non-linear, and unpredictable in nature (
), there is a
propensity to ‘count what can be easily measured’ (
rather than ‘measuring what counts’ in terms of significant,
enduring changes (
). We concur with this, however
the underlying problem is not the availability of possible
indicators, but rather the measurability of indicators, especially
outcome ones. A series of expert workshops, using IMI and
TI Pharma as examples, revealed that, with regard to output,
9 out of a total of 10 suggested indicators were considered
measurable; however, with regard to outcome/impact, only
2 out of a total of 12 suggested indicators were considered
measurable (109). This difficulty of measuring outcome,
especially in disciplines such as social and policy science, was also
reported in a recent analysis of 162 impact case studies from
the 2014 UK Research Excellence (
). As highlighted by
Perkmann et al., the intangibility of many outcome indicators
will require the identification of suitable and measurable
). This is especially true when it comes to
multistakeholder public-private collaborations, for which various
soft elements of critical success factors have been identified,
such as trust (
), value recognition (
), and stakeholder involvement (
Research impact assessment studies are all part of a
relatively new scientific endeavor. With PPPs becoming
increasingly multidisciplinary, involving multiple stakeholders with
competing interests and generating a multidirectional
exchange of knowledge, a recent review of the normative
literature on infrastructural PPPs suggests that the logic model of
impact assessment (cf. Table 3) doesn’t fully capture the
interaction between researchers, funders, innovation
intermediaries and end-users (
). Currently, other models have been
developed and are under evaluation, which take into
consideration the multiple-stakeholder complexity mentioned
). Several scholars have started to argue that in the
end, research impact assessment is most likely to be assessed
through a multi-indicator, multi-method approach (
involving a mix of quantitative analysis (e.g. publications,
citations, patents); interviews with key stakeholders; peer
assessment; case studies; and ultimately routine engagement with
the end-users of research.
PRECOMPETITIVE BIOMEDICAL PPPS
STARTING TO GENERATE TANGIBLE
A quick glance of the scientific literature reveals a number of
reviews highlighting that considerable outputs, such as
publications, patent applications, tools, in silico and animal models,
training modules, and open-source databases have been
generated by the various biomedical R&D PPPs initiated in the
last two decades (
). However, as
mentioned above, the true success of the precompetitive
multistakeholder public-private collaboration model ultimately
goes beyond mere outputs. Success is about its ability to
generate tangible outcomes, and in the long run a sustainable
socioeconomic impact. Questions that must be addressed include:
Which research has translated into novel disease targets, or in
policy and/or guideline changes? To what extent have
developed tools and models been adopted by relevant stakeholders?
What is the number of end-users of the various infrastructures
and databases that have been generated? How many PPPs
continue beyond their original term? In contrast to the
increasing number of publications on the concept of biomedical
R&D PPPs mentioned above, publications focusing
specifically on (short-term) outcomes and impact of biomedical R&D
PPPs supported by tangible evidence are still limited.
In the following sections four case studies differing in size,
geographical spread and research focus will be discussed.
The Structural Genomics Consortium (SGC)
In 2014, RAND Europe and the Institute on Governance
evaluated the SGC model of operation (
). SGC is a
precompetitive public-private partnership that was founded in
2003, and operates from six academic institutions - the
University of Toronto (Canada); the University of Oxford
(UK); Karolinska Institute (Sweden); University of North
Carolina (US); the State University of Campinas (Brazil);
and the Goethe University Frankfurt (Germany). The
partnership comprises an open collaborative network of scientists
in hundreds of universities around the world and in nine
global pharmaceutical companies. Its main goal is to accelerate
Bresearch in human biology and drug discovery by making all of its research
output freely available to the scientific community^ (
). The focus is
on providing a boost to drug discovery by determining 3D
protein structures in biomedically relevant areas in a
costeffective manner. Apart from perspectives on the approach,
strengths and weaknesses of the model and lessons learnt, part
of the evaluation was to extract the value of knowledge in line
with the framework presented above: outputs, outcomes and
impacts of the SGC. In 2013, the SGC had produced a total
of 1195 protein structures and 83 sequences; 17 chemical
probes and compound tools; and 98 antibodies. It made these
publicly accessible through established databases (Protein
Data Bank, Uniprot). More than 1400 clones were provided
to both the private and academic sectors. This considerable
output was accompanied by 452 peer-reviewed publications,
including Nature, PNAS and PLoS One, and by attendance
and presentations at well over 250 conferences. With regard
to outcomes, defined as Bthe utility of those outputs in delivering
additional impacts^, the report focused on three aspects: the
reach of the SGC research, the influence of SGC researchers
and the economic impact of SGC developments (
the list of indicators and categories depicted in Table 4 as a
point of reference, the reach of the SGC was identified in the
area of human capital (collaboration with over 500 scientists and
training of over 300 scholars). Its influence was acknowledged
in the areas of human capital (over 100 staff moved to academia,
industry and business schools) and knowledge sharing &
dissemination (around 100 meetings with industry, policy maker
discussion and workshops). Assessing the economic impact of
the SGC, such as the costs of new products or valuation of new
companies, was difficult. Apart from a $15 m financing of
sp in -out Tensha Th erap eu tics, a second spin-ou t
(1DegreeBio) estimated a potential $1bn cost-savings in
R&D expenditure relating to the identification of effective
). Finally, based on several assumptions, the
revenue of products related to the SGC was appraised at a value
of over $60 m CAD. Important to note is that apart from these
tangible deliverables, more intangible ones were also
mentioned. As mentioned by private sector representatives in
interviews, the SGC also provided a platform that allowed
pooling of resources and sharing of expertise, saving costs on
bureaucracy and generating a more efficient research
The Innovative Medicines Initiative (IMI)
The difficulty to attribute scientific outcomes to input
mentioned earlier was confirmed by the SGC evaluators as a key
limitation for a robust analysis. Research is not performed in
splendid isolation, but is part of a complex network of
interactions that contribute jointly to a specific scientific
). This complexity, and also the existing
lengthy timelines for pharmaceutical R&D (
), was also
mentioned in a recent report on the socio-economic impact of the
). With the aim of improving the competitive
situation of the European Union in the field of pharmaceutical
research, the European Commission (EC) and the European
Federation of Pharmaceutical Industries and Associations
(EFPIA) implemented the IMI in 2008 (
). With a total
budget of over EUR 5 billion (IMI1: 2 billion in 2008–2013;
IMI2: 3.3 billion in 2014–2024), IMI is generally considered
the largest biomedical PPP in the world. It aims to support the
development of next generation vaccines, medicines and
treatments, focusing on, among others, the improvement of the
current drug development process and reduction of the time
to reach clinical proof of concept. Currently, IMI has funded
over 80 public-private consortia spanning the complete drug
development cycle, including training, post-marketing and
In 2016, the IMI appointed an expert group to assess the
socio-economic impact of the first nine IMI projects (total
value >EUR 200 million; completed early 2016) of which
eight were focused on precompetitive research and one on
training. Although in the report the complexity of interactions
between various measures of inputs, outputs, outcomes and
impact was acknowledged, the expert group chose to adhere
to the logical framework (See Table 3) to assess the
performance of the project. In terms of output, the nine projects
combined generated over 550 scientific publications
(proxy for research activity and knowledge sharing);
cell-based or animal models in five different (disease)
areas; three biobanks (diabetes, medicine-induced
injuries, pain); novel imaging techniques in the area of
diabetes; new or improved biomarkers; a patient cohort
database; and novel tools to test biomarkers. In
addition, a plethora of novel targets, processes, approaches,
and methods were reported that could incite ongoing
and new collaborative research. Tangible outcomes that
were mentioned in the report were four
start-ups/spinoffs; patents; several assay products; tools and animal
models being commercialized; and a training program.
Taking into consideration the complexity of
pharmaceutical R&D and its lengthy timelines, the expert group
considered tangible socio-economic impacts, such as delivery of
novel treatments; cost and time savings; reductions in risk and
attrition rate; and the reduced need for animal testing beyond
the scope of the IMI. The impact IMI projects can generate is
primarily related to improvements and amendments within
the medicines development process itself, also defined by the
expert group as the Bpathway to socio-economic impact^ (
However, generating true tangible socio-economic impact will
rely heavily on recognition of the benefits arising from the
IMI, and subsequent actions by pharmaceutical companies,
regulators, payers and policy makers through adoption of
novel processes, the implementation of novel or updated
guidelines and policies, and so on.
The Dutch top Institute Pharma (TI Pharma)
This PPP (total budget: EUR 274 million) ran between 2006
and 2014 (motto: ‘health and wealth’) and built pharmaceutical
research and development networks in five disease areas:
(auto)immune diseases, cardio-vascular diseases, cancer,
infectious diseases, and brain diseases. Enabling technologies
included therapeutic target finding; validation & animal models; lead
selection & in-silico modelling; predictive drug disposition &
toxicology; biomarkers & bio-sensoring; and drug formulation,
delivery & targeting. In addition to contributions by Dutch
universities, Dutch SMEs and international big pharma
companies, the Dutch government supported this PPP with a total
contribution of EUR 135 million. TI Pharma operated as an
independent body, and the only condition for the Dutch
government’s contribution was that projects were in line with the
recommendations from the ‘WHO Priority Medicines report’
published in 2004 and (updated in 2013 (
)). A total of 26
academic institutions, 43 SMEs, and 20 big pharma companies
were partnering via 74 TI Pharma projects. At the end of the
2013, TI Pharma’s output consisted of 470 trained PhD and
post-doctoral fellows; almost 750 publications; 41 lead
compounds, lead series and libraries; 18 novel formulations; 11
biomarkers; 33 preclinical models; 28 clinical models; 11
research databases; and 87 research tools (assays, discovery
). Several tangible early outcomes were reported.
Regarding human capital, of the 257 TI Pharma fellows who had
finished their project by the end of 2013, an impressive 98%
continued their career in industry, academia and other pharma
(e.g., regulatory) and non-pharma organizations. In the area of
research activity and knowledge, clear benefit for patients is
anticipated through the delivery of a safe morphine dosage regimen
for newborns; new clinical protocols for COPD; a vaccine
candidate against acute myeloid leukemia; and a disease registry
for rare metabolic diseases, which will facilitate development of
diagnosis and treatment for this specific group of diseases. A
total of 74 follow-up projects could also be identified, which
built on the outputs of various TI Pharma projects (Networks &
collaboration). Finally, in 2015 TI Pharma merged with another
Dutch PPP, resulting in Lygature and continues to successfully
manage international PPP projects (current portfolio: >15
The Alzheimer’s Disease Neuroimaging Initiative (ADNI)
One of the most well-described precompetitive biomedical
R&D PPPs is the Alzheimer’s Disease Neuroimaging
Initiative or ADNI (
). ADNI was initiated in 2004,
and with a total budget of more than $150 million is
considered the biggest PPP in Alzheimer disease (AD)
research. Alzheimer disease is a complex disorder affecting tens
of millions of people around the world (
), and translation
of disease understanding into viable treatment options
remains disappointing (
). Apart from its intrinsic complexity,
one major limitation faced by both academic and
pharmaceutical partners during clinical trials was that outcome measures
were limited to clinical and cognitive ones. Through ADNI,
US leading Alzheimer research centers, the National Institute
on Aging, 13 pharmaceutical companies and two
not-forprofit foundations all combined resources and expertise to
facilitate development of effective AD treatments, by
developing, validating, and qualifying a novel set of imaging and other
biomarkers for clinical trials (
). Originally planned as a
five-year project (ADNI-1), with an important coordinating
effort by the Foundation for the National Institutes of
Health, the partnership has been successful in: 1) attracting
additional funding to extend (ADNI-GO) and renew (ADNI-2
& ADNI-3) the overall project until mid-2021; 2) increasing
the number of partners to more than 30; and 3) attracting the
Canadian Institutes for Health Research to join (133). To
date, there have been no successful clinical trials with
ADmodifying drugs. However, Weiner et al. and Jones-Davis
and Buckholtz recently summarized the key outputs and
outcomes of ADNI, which in essence started out as a multi-site,
longitudinal study of normal cognitive aging, mild cognitive
impairment, and early AD (
All data (PET, MRI, clinical, biospecimens, genetics)
acquired during the ADNI project to date is housed in a
database hosted by the Laboratory of Neuroimaging, and is open
to the entire scientific research community. This has resulted
in over 14 million downloads, and more than 750 publications
citing the use of ADNI data (
), of which a number are
in high-impact journals, such as Nature, PNAS, and PLOS
One (141). Populating the database continues, with the
upload of data derived from the use of ADNI samples. Apart
from ensuring its own sustainability, ADNI has also looked
beyond the US border (
). It has been a key driver in
promoting the ADNI concept worldwide, resulting in ADNI
projects based in North America, Europe, Japan, Australia,
Korea, and Argentina, w hich are united through
). The ADNI concept has also inspired
other initiatives focusing on closely related aspects of AD (e.g.,
traumatic brain injury as an AD risk factor) and other PPPs
(e.g., developing biomarkers for Parkinson’s disease).
By making data and samples available, the infrastructures
and databases described above have also made an important
contribution to enhancing the basic understanding of various
complex diseases. As well as providing open-access to data and
sharing of samples, ADNI has also made available a set of
protocols and methods related to the study itself, to MRI
and PET analysis and acquisition, and to biomarker and
proteomic analysis. This allows comparison of data gathered
across multiple sites. According to Weiner et al. Bpharmaceutical
companies developing disease-modifying treatments for AD and studies
funded by the National Institutes of Health and private foundations have
used ADNI methods in virtually all their clinical trials^ (
knowledge of the progression of AD pathology, of biomarker
interrelationship, and of genetic risk factors for the disease that
ADNI has generated, is also being acknowledged in recently
issued draft regulatory guidelines on clinical investigation of
medicines for the treatment of AD and other dementias
). In addition, to helping enrich recruitment in
regulatory clinical trials for Mild and Moderate AD, the first two
biomarkers (CSF Aβ and tau protein; and low hippocampal
volume) have been or are in the process of being qualified by
the EMA and FDA (
Despite the intrinsic difficulties involved in measuring the
impact of biomedical PPPs, recent evaluations of the SGC, IMI,
TI Pharma and ADNI highlighted in this review provide clear
evidence that precompetitive biomedical PPPs have started to
generate tangible outcomes. These outcomes are in areas
where not only the pharmaceutical industry and academia,
but also health foundations, patient organizations and
regulatory agencies share a mutual interest. These include: 1)
providing the necessary infrastructure and databases to facilitate
the discovery of novel disease targets and leads; 2) increasing
basic understanding of complex diseases, including novel
models and standards; 3) capacity building through jointly
designed training programs; and 4) providing a platform to
allow exchange of different perspectives on, for example, the
regulatory system and regulatory science (
Considering the long timelines for uptake of validated new
developments in the biomedical/pharmaceutical field,
continuation of PPPs beyond their original time span (often 4–
6 years) will certainly contribute to PPPs reaching their full
potential. This can be achieved either through a continuation
of funding (ADNI, IMI) or by changing the business model (TI
As highlighted above, pharmaceutical R&D is complex
and characterized by lengthy timelines, and one could easily
argue that the challenges mentioned in this review make it
very difficult to properly assess the long-term impact of the
multi-stakeholder PPP concept on medical practice. However,
we should continue to improve the metrics. Given the growing
number of PPPs, the availability of funding and resources will
become a limiting factor, and will Bforce^ stakeholders to
become even more prudent in selecting which PPP
initiatives to back or join, and which ones to avoid (
simple terms, partners, funders, and civil society will
increasingly seek confirmation of the incremental value
achieved through partnerships.
This paper highlights a link between the emergence of the
PPP concept and the growing ‘popularity’ of open innovation
in the biomedical/pharmaceutical R&D world. One may
wonder how this open innovation model will evolve, and both
whether and how the format of PPPs will change with it. We
have already seen changes since the time when a typical PPP
was based on an academic and industrial pillar, with
governmental or other third party funding as ‘carrot’. Over time,
health foundations, patient organizations and regulatory
scientists have regularly joined in (
). So, what is next?
More funding and ‘research priority setting’ from
philanthropists, also coined as philanthrocapitalism (148)? The future
ACKNOWLEDGEMENTS AND DISCLOSURES
Nothing to declare
Open Access This article is distributed under the terms of the
Creative Commons Attribution 4.0 International License
permits unrestricted use, distribution, and reproduction in any
medium, provided you give appropriate credit to the original
author(s) and the source, provide a link to the Creative
Commons license, and indicate if changes were made.
1. Pardoe D , Hunter J , Cooke R , Barrett P . Assessing the value of R & D partnerships . Drug Discov World. 2010 /11;Winter: 9 - 17 .
2. Denee TR , Sneekes A , Stolk P , Juliens A , Raaijmakers JA , Goldman M , et al. Measuring the value of public-private partnerships in the pharmaceutical sciences . Nat Rev Drug Discov . 2012 ; 11 ( 5 ): 419 .
3. Drolet BC , Lorenzi NM . Translational research: understanding the continuum from bench to bedside . Transl Res . 2011 ; 157 ( 1 ): 1 - 5 .
4. Price AL , Spencer CC , Donnelly P . Progress and promise in understanding the genetic basis of common diseases . In Proc. R. Soc. B 2015 Dec 22 (Vol. 282 , no. 1821 , p. 20151684 ). The Royal Society.
5. Ioannidis JP . Expectations, validity, and reality in omics . J Clin Epidemiol . 2010 ; 63 ( 9 ): 945 - 9 .
6. Kola I , Landis J . Can the pharmaceutical industry reduce attrition rates? Nat Rev Drug Discov . 2004 ; 3 ( 8 ): 711 - 6 .
7. Mervis J. Productivity counts - but the definition is key . Science . 2005 ; 309 ( 5735 ): 726 - 7 .
8. Scannell JW , Blanckley A , Boldon H , Warrington B. Diagnosing the decline in pharmaceutical R&D efficiency . Nat Rev Drug Discov . 2012 ; 11 ( 3 ): 191 - 200 .
9. Munos B . Lessons from 60 years of pharmaceutical innovation . Nat Rev Drug Discov . 2009 ; 8 ( 12 ): 959 - 68 .
10. Chabner BA . Defining a new role for the National Cancer Institute cooperative groups: more science, fewer trials . Oncologist . 2014 ; 19 : 1113 - 4 .
11. Moors EHM , Cohen AF , Schellekens H. Towards a sustainable system of drug development . Drug Discov Today . 2014 ; 19 ( 11 ): 1711 - 20 .
12. Schuhmacher A , Gassmann O , Hinder M. Changing R & D models in research-based pharmaceutical companies . J Transl Med . 2016 ; 14 ( 1 ): 105 .
13. Denoon A , Vollebregt E. Can regenerative medicine save big Pharma's business model from the patent cliff? Regen Med . 2010 ; 5 : 687 - 90 .
14. Gilbert J , Henske P , Singh A . Rebuilding big pharma's business model . In Vivo-New York Then Norwalk . 2003 ; 21 ( 10 ): 73 - 80 .
15. Kaitin KI . Deconstructing the drug development process: the new face of innovation . Clin Pharmacol Ther . 2010 ; 87 ( 3 ): 356 - 61 .
16. LaMattina JL . The impact of mergers on pharmaceutical R&D. Nat Rev Drug Discov . 2011 ; 10 ( 8 ): 559 - 60 .
17. Comanor WS , Scherer FM . Mergers and innovation in the pharmaceutical industry . J Health Econ . 2013 ; 32 ( 1 ): 106 - 13 .
18. Dolgin E. Big pharma moves from Bblockbusters^ to Bniche busters^ . Nat Med . 2010 ; 16 ( 8 ): 837 .
19. FDA . Novel drugs 2015 summary . January 2016 . US Food and Drug Administration .
20. Regulatory Affairs Professionals Society . EMA carries 2014 momentum, recommends 39 new drugs and sets orphan record . http://www.raps.org/Regulatory-Focus/News/2016/01/12/ 23890/ EMA-Carries- 2014 -Momentum-Recommends- 39 -NewDrugs- and - Sets- Orphan-Record/#sthash. cgU5Cbie.dpuf. Accessed 17 Mar 2017 .
21. Owens J. 2006 drug approvals: finding the niche . Nat Rev Drug Discov . 2007 ; 6 ( 2 ): 99 - 101 .
22. EMA . Human medicines highlights - November 2015. Information on Medicines 2015 .
23. Cockburn IM . The changing structure of the pharmaceutical industry . Health Aff . 2004 ; 23 ( 1 ): 10 - 22 .
24. Bianchi M , Cavaliere A , Chiaroni D , Frattini F , Chiesa V . Organisational modes for open innovation in the biopharmaceutical industry: an exploratory analysis . Technovation . 2011 ; 31 ( 1 ): 22 - 33 .
25. Dixon J. RIPCO , FIPCO, NRDO, FIPNET, VIPC. Nature Biotechnology . 2011 . http://blogs.nature.com/tradesecrets/ 2011/05/31/ripco-fipco -nrdo-fipnet-vipco . Accessed 17 Mar 2017 .
26. Hunter J . Challenges for pharmaceutical industry: new partnerships for sustainable human health . Philos Trans a Math Phys Eng Sci . 2011 ; 369 ( 1942 ): 1817 - 25 .
27. Rafols I , Hopkins MM , Hoekman J , Siepel J , O'Hare A , PerianesRodriguez A , et al. Big Pharma, little science? A bibliometric perspective on big Pharma's R&D decline . Technol Forecast Soc Change . 2014 ; 81 ( 1 ): 22 - 38 .
28. Wang L , Plump A , Ringel M. Racing to define pharmaceutical R&D external innovation models . Drug Discov Today . 2015 ; 20 ( 3 ): 361 - 70 .
29. Gehr S , Garner CC . Rescuing the lost in translation . Cell . 2016 ; 165 ( 4 ): 765 - 70 .
30. Michelino F , Lamberti E , Cammarano A , Caputo M. Measuring open innovation in the bio-pharmaceutical industry . Creativity and Innovation Management . 2015 ; 24 ( 1 ): 4 - 28 .
31. Schuhmacher A , Germann PG , Trill H , Gassmann O . Models for open innovation in the pharmaceutical industry . Drug Discov Today . 2013 ; 18 ( 23 ): 1133 - 7 .
32. Blumenthal D , Causino N , Campbell E , Louis KS . Relationships between academic institutions and industry in the life sciences-an industry survey . N Engl J Med . 1996 ; 334 ( 6 ): 368 - 73 .
33. Blumenthal D , Campbell E , Causino N , Louis KS . Participation of life-science faculty in research relationships with industry . N Engl J Med . 1996 ; 335 ( 23 ): 1734 - 9 .
34. Bagley CE , Tvarno CD . Pharmaceutical public-private partnerships in the United States and Europe: moving from the bench to the bedside . Harvard Bus Law Rev . 2013 ; 4 : 373 - 400 .
35. Milne CP , Malins A . Academic-industry partnerships for biopharmaceutical research & development: advancing medical science in the US . Tuft Center for the Study of Drug Development . 2012 Apr.;
36. D'Este P , Perkmann M. Why do academics engage with industry? The entrepreneurial university and individual motivations . J Technol Transf . 2011 ; 36 ( 3 ): 316 - 39 .
37. Stuart TE , Ozdemir SZ , Ding WW . Vertical alliance networks: the case of university-biotechnology-pharmaceutical alliance chains . Res Policy . 2007 ; 36 ( 4 ): 477 - 98 .
38. Lim MD . Consortium sandbox: building and sharing resources . Sci Transl Med . 2014 ; 6 ( 242 ): 242cm6 .
39. Yildirim O , Gottwald M , Schüler P , Michel MC . Opportunities and challenges for drug development: public-private partnerships . Adaptive Designs and big Data Front Pharmacol . 2016 ; 7 : 461 .
40. Stolk P. Priority Medicines for Europe and the World B A Public Health Approach to Innovation ^ Update on 2004 Background Paper Background Paper 8.1 Public Private Partnerships . 2013 ; 1 - 25 .
41. Reypens C , Lievens A , Blazevic V . Leveraging value in multistakeholder innovation networks: a process framework for value co-creation and capture . Ind Mark Manag . 2014 ; 56 : 40 - 50 .
42. van Tulder R , Seitanidi MM , Crane A , Brammer S. Enhancing the impact of cross-sector partnerships: four impact loops for channeling partnership studies . J Bus Ethics . 2015 ; 135 ( 1 ): 1 - 17 .
43. Hodge GA , Greve C. PPPs: the passage of time permits a sober reflection . Econ Aff . 2009 ; 29 ( 1 ): 33 - 9 .
44. Stevens H , Van Overwalle G , Van Looy B , Huys I . Perspectives and opportunities for precompetitive public-private partnerships in the biomedical sector . Biotechnol Law Rep . 2013 ; 32 ( 3 ): 131 - 9 .
45. Croft SL . Public-private partnership: from there to here . Trans R Soc Trop Med Hyg . 2005 ; 99 ( Suppl 1 ): S9 - 14 .
46. De Pinho CK , Norman CD , Jadad AR . Product development public-private partnerships for public health: a systematic review using qualitative data . Soc Sci Med . 2011 ; 73 ( 7 ): 986 - 94 .
47. Wheeler C , Berkley S . Initial lessons from public-private partnerships in drug and vaccine development . Bull World Health Organ . 2001 ; 79 ( 8 ): 728 - 34 .
48. Goldman M. New frontiers for collaborative research . Sci Transl Med . 2013 ; 5 ( 216 ): 216ed22 .
49. Widdus R . Public-private partnerships: an overview . Trans R Soc Trop Med Hyg . 2005 Oct; 99 ( Suppl 1 ): S1 - 8 .
50. Woodcock J , Woosley R . The FDA critical path initiative and its influence on new drug development . Annu Rev Med . 2008 ; 59 : 1 - 12 .
51. Woosley RL , Cossman J . Drug development and the FDA's critical path initiative . Clin Pharmacol Ther . 2007 ; 81 ( 1 ): 129 - 33 .
52. Goldman M. The innovative medicines initiative: a European response to the innovation challenge . Clin Pharmacol Ther . 2012 ; 91 ( 3 ): 418 - 25 .
53. Stevens H , Van Overwalle G , Van Looy B , Huys I. Innovative medicines initiative (IMI) case study analysis reveals the true added value of early-phase public-private partnerships (PPPs) . Biotechnol Law Rep . 2015 ; 34 ( 4 ): 153 - 65 .
54. TI Pharma . TI Pharma new tracks to medicines . 2014 . http:// www.lygature.org/files/atoms/fil es/TI%20Pharma% 20Report % 20 -% 20New%20Tracks%20to%20Medicines.pdf. Accessed 17 March 2017 .
55. Edwards A . Team up with industry . Nature . 2016 ; 531 ( 7594 ): 299 - 301 .
56. Morgan Jones M , Castle-Clarke S , Brooker S , Nason E , Huzair F , Chataway J. Genomics Consortium A knowledge platform for drug discovery . 2014 ; RAND Europe with The Institute on Governance:RR-512-SGC . http://www.rand.org/content/ dam/rand/pubs/research_reports/RR500/RR512/RAND_ RR512.pdf. Accessed 17 Mar 2017 .
57. Nwaka S , Riopel L , Ubben D , Craft JC . Medicines for Malaria Venture new developments in antimalarials . Travel Med Infect Dis . 2004 ; 2 ( 3 -4): 161 - 70 .
58. Maxmen A. Big Pharma's cost cutting challenger . Nature . 2016 ; 536 : 388 - 90 .
59. Holmes D. Profile: IAVI-searching for the Achilles' heel of HIV . Lancet. 2013 ; 381 ( 9883 ): 2072 .
60. Kourany-Lefoll E , Keiser J , De Vrueh R . Case study: schistosomiasis . EBR . 2015 , January. P10 - 14 .
61. Murray F , O'Mahony S. Exploring the foundations of cumulative innovation: implications for Organization science . Organ Sci . 2007 ; 18 : 1006 - 21 .
62. Perkmann M , Neely A , Walsh K. How should firms evaluate success in university-industry alliances? A performance measurement system . R D Manag . 2011 ; 41 ( 2 ): 202 - 16 .
63. Piva E , Rossi-Lamastra C . Systems of indicators to evaluate the performance of university-industry alliances: a review of the literature and directions for future research . Meas Bus Excell . 2013 ; 17 ( 3 ): 40 - 54 .
64. Bruneel J , Este PD , Salter A . Investigating the factors that diminish the barriers to university - industry collaboration . Res Policy . 2010 ; 39 ( 7 ): 858 - 68 .
65. Grimaldi R , Kenney M , Siegel DS , Wright M. 30 years after Bayh-dole: reassessing academic entrepreneurship . Res Policy . 2011 ; 40 ( 8 ): 1045 - 57 .
66. Jacobsson S , Lindholm-Dahlstrand Å , Elg L. Is the commercialization of European academic R&D weak? - a critical assessment of a dominant belief and associated policy responses . Res Policy . 2013 ; 42 ( 4 ): 874 - 85 .
67. Luijten PR , van Dongen GAMS , Moonen CT , Storm G , Crommelin DJA . Public-private partnerships in translational medicine: concepts and practical examples . J Control Release . 2012 ; 161 ( 2 ): 416 - 21 .
68. Yamamoto K. Time to play ball . Science . 2013 ; 340 ( 6139 ): 1375 .
69. Ramos-Vielba I , Sanchez-Barrioluengo M , Woolley R . Scientific research groups' cooperation with firms and government agencies: motivations and barriers . J Technol Transf . 2016 ; 41 ( 3 ): 558 - 85 .
70. Crowley WF . Translation of basic research into useful treatments: how often does it occur? Am J Med . 2003 ; 114 ( 6 ): 503 - 5 .
71. Perkmann M. University - industry relations. In: Audretsch D , Hayter C , Link A , editors. Concise guide to entrepreneurship, technology and innovation . Cheltenham: Edward Elgar; 2015 . p. 227 - 33 .
72. Monaghan BJ , Malek A -M, Simson H . Public-private partnerships in healthcare: criteria for success . Healthc Manag Forum . 2001 ; 14 ( 4 ): 44 - 9 .
73. Angell M , Relman AS . Patents, profits & American medicine: conflicts of interest in the testing & marketing of new drugs . Daedalus . 2002 ; 131 ( 2 ): 102 - 11 .
74. Lexchin J , Bero LA , Djulbegovic B , Clark O . Pharmaceutical industry sponsorship and research outcome and quality: systematic review . BMJ . 2003 ; 326 ( 7400 ): 1167 - 70 .
75. Bekelman JE , Li Y , Gross CP . Scope and impact of financial conflicts of interest in biomedical research: a systematic review . JAMA . 2003 ; 289 ( 4 ): 454 - 65 .
76. Sismondo S. How pharmaceutical industry funding affects trial outcomes: causal structures and responses . Soc Sci Med . 2008 ; 66 ( 9 ): 1909 - 14 .
77. Cohen WM , Nelson RR , Walsh JP . Links and impacts: the influence of public research on Industrial R & D. Manag Sci . 2002 ; 48 ( 1 ): 1 - 23 .
78. Toole AA . The impact of public basic research on industrial innovation: evidence from the pharmaceutical industry . Res Policy . 2012 ; 41 ( 1 ): 1 - 12 .
79. De Vrueh RL , Awad W , Stolk A , Dijcks FA , Rijnders TW , Janssen JW . Deal watch: roles and strategies for health foundations in public-private partnerships . Nat Rev Drug Discov . 2014 ; 13 ( 6 ): 406 .
80. Chin-Dusting J , Mirzahi J , Jennings G , Fitzgerald D . Finding improved medicines: the role of academic-industrial collaboration . Nat Rev Drug Discov . 2005 ; 4 ( 11 ): 891 - 7 .
81. Technopolis group. Effects of NGI investments . Study commissioned by the Netherlands Genomics Initiative . Jun 26 2012. http://www.technopolis-group.com/wp-content/uploads/2014/ 05/1481_NGI-Multiplier -Report.pdf. Accessed 17 Mar 2017 .
82. Research America. U.S. Investments in Medical and Health Research and Development 2013 -2015. Fall 2016 . https:// www.researchamerica.org/sites/default/files/2016US_Invest_ R%26D _report.pdf. Accessed 17 Mar 2017 .
83. Chang J. Best Practices for Venture Philanthropy Collaborations between Disease- Focused Foundations and For-Profit Life Science Companies . June 2010. Master Thesis; Massachusetts Institute of Technology http://hdl.handle. net/1721.1/58400. Accessed 17 Mar 2017 .
84. Marshall BC , Penland CM , Hazle L , Ashlock M , Wetmore D , Campbell PW , et al. Cystic fibrosis foundation: achieving the mission . Respir Care . 2009 ; 54 ( 6 ): 788 - 95 .
85. Lim MD , Brumfield MA , Goldman M . Philanthropies as partners for drug development in public-private partnerships . J Med Dev Sci . 2016 ; 1 ( 1 ): 10 - 3 .
86. Anderson M , Kimberly MCK . On the path to a science of patient input . Sci Transl Med . 2016 ; 8 ( 336 ): 336ps11 .
87. Anderson M , McCleary KK . From passengers to co-pilots: Patient roles expand . Sci Transl Med . 2015 ; 7 ( 291 ): 291fs25 .
88. Innovative Medicines Initiative. IMI Project portfolio . http:// www.imi.europa.eu/content/ongoing-projects. Accessed 1 Feb 2017 .
89. De Wit M , Bloemkolk D , Teunissen T , van Rensen A. Voorwaarden voor succesvolle betrokkenheid van patiënten/ cliënten bij medisch wetenschappelijk onderzoek . Tijdschrift Voor Gezondheidswetenschappen . 2016 ; 94 ( 3 ): 91 - 100 .
90. EPAD website. European Prevention of Alzheimer's Dementia (EPAD) Consortium http ://ep-ad. org/project-structure/. Accessed 1 Feb 2017 .
91. Alzheimer Europe. PharmaCog. http://www.alzheimer-europe. org/Research/PharmaCog. Accessed 1 Feb 2017 .
92. Panayidou K , Gsteiger S , Egger M , Kilcher G , Carreras M , Efthimiou O , et al. GetReal in mathematical modelling: a review of studies predicting drug effectiveness in the real world . Res Synth Methods . 2016 ; 7 ( 3 ): 264 - 77 .
93. Reynolds RF , Kurz X , de Groot MCH , Schlienger RG , Grimaldi-Bensouda L , Tcherny-Lessenot S , et al. Theories of attempted suicide: should they differ from theories of completed suicide? Pharmacoepidemiol Drug Saf 2016 ; 25 ( Suppl . 1): 5 - 10 .
94. Supple D , Roberts A , Hudson V , Masefield S , Fitch N , Rahmen M , et al. From tokenism to meaningful engagement: best practices in patient involvement in an EU project . Res Involv Engagem . 2015 ; 1 ( 1 ): 5 .
95. Pushparajah D , Geissler J , Westergaard N. EUPATI: collaborating between patients, academia and industry to champion the informed patient in medicines research and development . J Med Dev Sci . 2015 ; 1 ( 1 ): 74 - 80 .
96. Goldman M , Seigneuret N , Eichler H-G. The innovative medicines initiative: an engine for regulatory science . Nat Rev Drug Discov . 2015 Jan; 14 ( 1 ): 1 - 2 .
97. Parekh A , Buckman-Garner S , McCune S , ONeill R , Geanacopoulos M , Amur S , et al. Catalyzing the critical path initiative: FDA's progress in drug development activities . Clin Pharmacol Ther . 2015 Mar; 97 ( 3 ): 221 - 33 .
98. Goldman M , Compton C , Mittleman BB . Public-private partnerships as driving forces in the quest for innovative medicines . Clin Transl Med . 2013 ; 2 ( 1 ): 2 .
99. Ermisch M , Bucsics A , Bonanno PV , Arickx F , Bybau A , Bochenek T , et al. Payers' views of the changes arising through the possible adoption of adaptive pathways . Front Pharmacol . 2016 ; 7 : 305 .
100. Spindler P , Bach KF , Schmiegelow M , Bedlington N , Eichler HG . Innovation of medical products: the evolution of regulatory science, research, and education . Ther Innov Regul Sci . 2015 ; 50 ( 1 ): 44 - 8 .
101. Hardman M. European initiatives for better training in medicines development . J med Dev Sci . 2015 ; 1 : 1 - 9 .
102. Freire MC . Partnerships: beyond the buzzword . Sci Transl Med . 2014 , 6 : 242ed15 .
103. Penfield T , Baker MJ , Scoble R , Wykes MC . Assessment evaluations and definition of research impact . Res Eval . 2013 ; 23 ( 1 ): 1 - 12 .
104. Molas-Gallart J , D'Este P , Llopis O , Rafols I . Towards an alternative framework for the evaluation of translational research initiatives . Res Eval . 2016 ; 25 ( 3 ): 235 - 43 .
105. Innovative Medicines Initiative. IMI Socio-economic Impact Assessment Expert Group Final Report . 2016 , May. Document Reference: IMI2/OUT/2016-01541 http://www.imi.europa. eu/sites/default/files/uploads/documents/Publications/ SocioeconomicImpactAssessment_FINALMay2016.pdf. Accessed 17 Mar 2017 .
106. Seppo M , Lilles A . Indicators measuring university cooperation. Discuss Est Econ Policy . 2010 ; 20 ( 1 ): 204 - 25 .
107. Baccarini D. The logical framework method for defining project success . Int J Confl Manag . 1999 ; 10 ( 2 ): 130 - 53 .
108. UK Evaluation Forum . Report: Medical research: assessing the benefits to society . 2016, May. ISBN No. I-903401-12-7.
109. Dittrich K. How valuable is your partnership? A framework for valuing public-private research partnerships in biomedical sciences . DRUID Society Meetings 2012 . http://druid8.sit.aau.dk/ acc_papers/mnsc0qm81x1fx27vt2leqq9fji0i.pdf. Accessed 17 Mar 2017 .
110. Thonon F , Boulkedid R , Delory T , Rousseau S , Saghatchian M , Van Harten W , et al. Measuring the outcome of biomedical research: a systematic literature review . PLoS One . 2015 ; 10 ( 4 ): e0122239 .
111. Hughes A , Martin BR . Enhancing Impact: The Value of Public Sector R & D: Summary report. Council for Industry and Higher Education . 2012 , August http://www.ncub.co. uk/impact. Accessed 17 Mar 2017 .
112. Daviter F. The political use of knowledge in the policy process . Policy Sci . 2015 ; 48 ( 4 ): 491 - 505 .
113. Milat AJ , Bauman AE , Redman S. A narrative review of research impact assessment models and methods . Health Res Policy Sys . 2015 ; 13 ( 1 ): 18 .
114. Grant J . Policy resource: Measuring the benefits from research . RAND Europe report RB-9202-RE . 2016 .
115. Bozeman B , Fay D , Slade CP . Research collaboration in universities and academic entrepreneurship: the-state-of-the-art . J Technol Transfer . 2013 ; 38 ( 1 ): 1 - 67 .
116. Wells R , Whitworth J . Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Aust New Zealand Health Policy . 2007 ; 4 : 14 .
117. Greenhalgh T , Fahy N. Research impact in the community-based health sciences: an analysis of 162 case studies from the 2014 UK Research Excellence Framework . BMC Med . 2015 ; 13 ( 1 ): 232 .
118. Osei-Kyei R , Chan APC . Review of studies on the critical success factors for public-private partnership (PPP) projects from 1990 to 2013 . Int J Proj Manag . 2015 ; 33 ( 6 ): 1335 - 46 .
119. Turner JR , Müller R. The project manager's leadership style as a success factor on projects: a literature review . Proj Manag J . 2005 ; 36 ( 1 ): 49 - 61 .
120. De Pinho CK , Norman CD , Jadad AR . Product development public-private partnerships for public health: a systematic review using qualitative data . Soc Sci Med . 2011 ; 73 ( 7 ): 986 - 94 .
121. Belassi W , Tukel OI . A new framework for determining critical success/failure factors in projects . Int J Proj Manag . 1996 ; 14 ( 3 ): 141 - 51 .
122. Buse K , Harmer AM . Seven habits of highly effective global public-private health partnerships: practice and potential . Soc Sci Med . 2007 ; 64 ( 2 ): 259 - 71 .
123. Liu J , Love PED , Smith J , Regan M , Sutrisna M . Public-private partnerships: a review of theory and practice of performance measurement . Int J Product Perform Manag . 2014 ; 63 ( 4 ): 499 - 512 .
124. Liu J , Love PED , Davis PR , Smith J , Regan M. Conceptual framework for the performance measurement of public-private partnerships . J Infrastruct Syst . 2014 ; 4014023 : 1 - 15 .
125. Goldman M , Wittelsberger A , De Magistris M-T. The innovative medicines initiative moves translational immunology forward . Eur J Immunol . 2013 ; 43 ( 2 ): 298 - 302 .
126. Carrillo MC , Sanders CA , Katz RG . Maximizing the Alzheimer's Disease Neuroimaging Initiative II . Alzheimers Dement . 2009 ; 5 ( 3 ): 271 - 5 .
127. Buse K , Tanaka S . Global public-private health partnerships: lessons learned from ten years of experience and evaluation . Int Dent J . 2011 ; 61 ( Suppl . 2): 2 - 10 .
128. Structural Genomics Consortium. What is the SGC? http:// www .thesgc.org/about/what_is_the_sgc. Accessed 29 Jan 2017 .
129. Donovan C , Hanney S. The Bpayback framework^ explained . Res Eval . 2011 ; 20 ( 3 ): 181 - 3 .
130. Kaplan W , Wirtz VJ , Mantel-Teeuwisse A , Stolk P , Duthey B , Laing R . Priority medicines for Europe and the world 2013 update . World Health Organization 2013 . http://www.who.int/ medicines/areas/priority_medicines/MasterDocJune28_ FINAL_Web.pdf. Accessed 17 Mar 2017 .
131. Carrillo MC , Bain LJ , Frisoni GB , Weiner MW . Worldwide Alzheimer's disease neuroimaging initiative . Alzheimers Dement . 2012 ; 8 ( 4 ): 337 - 42 .
132. Weiner MW , Veitch DP , Aisen PS , Beckett LA , Cairns NJ , Cedarbaum J , et al. Impact of the Alzheimer's Disease Neuroimaging Initiative , 2004 to 2014. Alzheimers Dement. 2015 ; 11 ( 7 ): 865 - 84 .
133. Jones-Davis DM , Buckholtz N. The impact of the Alzheimer's Disease Neuroimaging Initiative 2: what role do public-private partnerships have in pushing the boundaries of clinical and basic science research on Alzheimer's disease? Alzheimers Dement . 2015 Jul; 11 ( 7 ): 860 - 4 .
134. Aisen PS. Q& a: the Alzheimer's disease neuroimaging initiative . BMC Med . 2011 ; 9 ( 1 ): 101 .
135. Snyder HM , Kim H , Bain LJ , Egge R , Carrillo MC . Alzheimer's disease public-private partnerships: update 2014 . Alzheimers Dement . 2014 ; 10 ( 6 ): 873 - 80 .
136. Salloway S . New lessons from the Alzheimer's Disease Neuroimaging Initiative . Arch Neurol . 2011 ; 68 ( 1 ): 19 - 21 .
137. Snyder HM , Bain LJ , Egge R , Carrillo MC . Alzheimer's disease public-private partnerships: a landscape of the global nonprofit community . Alzheimers Dement . 2013 ; 9 ( 4 ): 466 - 71 .
138. Mayeux R , Stern Y. Genetic epidemiology of Alzheimer disease . Alzheimer Dis Assoc Disord . 2006 ; 20 ( 3 Suppl 2 ): S58 - 62 .
139. Schneider LS , Mangialasche F , Andreasen N , Feldman H , Giacobini E , Jones R , et al. Clinical trials and late-stage drug development for Alzheimer's disease: an appraisal from 1984 to 2014 . J Intern Med . 2014 ; 275 ( 3 ): 251 - 83 .
140. ADNI. ADNI data usage statistics . http://adni.loni.usc.edu/datasamples/adni -data-usage-stats/ . Accessed 2 Jan 2017 .
141. ADNI. ADNI publication overview . http://adni.loni.usc.edu/ news-publications/publications/. Accessed 2 Jan 2017 .
142. Hendrix JA , Finger B , Weiner MW , Frisoni GB , Iwatsubo T , Rowe CC , et al. The worldwide Alzheimer's Disease Neuroimaging Initiative: an update . Alzheimers Dement . 2015 ; 11 ( 7 ): 850 - 9 .
143. Committee for Medicinal Products for Human Use (CHMP). Draft guideline on the clinical investigation of medicines for the treatment of Alzheimer's disease and other dementias. 2016; European Medicines Agency (EMA): EMA /CHMP/539931/ 2014 .
144. US Food and Drug Administration (FDA). Guidance for Industry Alzheimer's Disease: Developing Drugs for the Treatment of Early Stage Disease . Draft guidance; 2013 .
145. Committee for Medicinal Products for Human Use (CHMP), Q u a l i f i c a t i o n o p i n i o n o f A l z h e i m e r ' s d i s e a s e n o v e l methodologies/biomarkers for the use of CSF AB 1-42 and ttau and/or PET-amyloid imaging (positive/ negative) as biomarkers for enrichment, for use in regulatory clinical trials in mild and moderate Alzheimer's disease. 2012; European Medicines Agency (EMA): EMA /CHMP/SAWP/893622/ 2011 .
146. Kozauer N , Katz R . Regulatory innovation and drug development for early-stage Alzheimer's disease . N Engl J Med . 2013 ; 368 ( 13 ): 1169 - 71 .
147. Brady LS , William Z , Potter WZ . Public-private partnerships to revitalize psychiatric drug discovery . Expert Opin Drug Discovery . 2014 ; 9 ( 1 ): 1 - 8 .
148. Clark J , McGoey L . The black box warning on philanthrocapitalism . Lancet . 2016 ; 388 ( 10059 ): 2457 - 9 .