Reflections on the Future of Pharmaceutical Public-Private Partnerships: From Input to Impact

Pharmaceutical Research, Jun 2017

Public Private Partnerships (PPPs) are multiple stakeholder partnerships designed to improve research efficacy. We focus on PPPs in the biomedical/pharmaceutical field, which emerged as a logical result of the open innovation model. Originally, a typical PPP was based on an academic and an industrial pillar, with governmental or other third party funding as an incentive. Over time, other players joined in, often health foundations, patient organizations, and regulatory scientists. This review discusses reasons for initiating a PPP, focusing on precompetitive research. It looks at typical expectations and challenges when starting such an endeavor, the characteristics of PPPs, and approaches to assessing the success of the concept. Finally, four case studies are presented, of PPPs differing in size, geographical spread, and research focus.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

Reflections on the Future of Pharmaceutical Public-Private Partnerships: From Input to Impact

Pharm Res Reflections on the Future of Pharmaceutical Public-Private Partnerships: From Input to Impact Remco L. A. de Vrueh 0 1 2 Daan J. A. Crommelin 0 1 2 D SGC SME TI Pharma WHO 0 1 2 0 Research & development Structural genomics consortium Small and medium-sized enterprise Top institute pharma World health organization 1 Department of Pharmaceutics, Utrecht Institute for Pharmaceutical Sciences, UIPS Utrecht University , Utrecht , The Netherlands 2 Lygature , Utrecht , The Netherlands 3 Daan J. A. Crommelin Public Private Partnerships (PPPs) are multiple stakeholder partnerships designed to improve research efficacy. We focus on PPPs in the biomedical/ pharmaceutical field, which emerged as a logical result of the open innovation model. Originally, a typical PPP was based on an academic and an industrial pillar, with governmental or other third party funding as an incentive. Over time, other players joined in, often health foundations, patient organizations, and regulatory scientists. This review discusses reasons for initiating a PPP, focusing on precompetitive research. It looks at typical expectations and challenges when starting such an endeavor, the characteristics of PPPs, and approaches to assessing the success of the concept. Finally, four case studies are presented, of PPPs differing in size, geographical spread, and research focus. key drivers; performance evaluation; public-private-partnerships; R&D business models ABBREVIATIONS AD Alzheimer’s disease ADNI Alzheimer’s disease neuroimaging initiative EMA European medicines agency FDA Food and drug administration IMI European innovative medicines initiative PPP Public private partnerships INTRODUCTION Bilateral R&D interactions between academic and industrial scientists have a long tradition. In the last decade, alongside these bilateral (‘vertical’) interactions, the multiple stakeholder partnerships called Public-Private Partnerships (PPPs) have emerged. These are R&D networks that do not work on a one-to-one basis but rather involve a range of stakeholders. As well as traditional academia and industry stakeholders, such a PPP may include charities, patient organizations, and even national competent authorities (‘regulators’). The scope of this review is a discussion of the role and added-value of PPPs in facilitating precompetitive multistakeholder collaborative research. First, we will focus on the key drivers for different stakeholders as they embrace the multi-stakeholder collaborative research model. Second, we will provide insight into the challenges of managing multi-stakeholder collaborative research projects and how different PPPs have dealt with these challenges. Third, we will elaborate more on current models for monitoring and evaluating the performance of precompetitive PPPs. Finally, given the growing number of PPPs, we conclude by discussing their value for society and the need to justify the public and private investments being made ( 1,2 ). SHIFT IN THE PHARMACEUTICAL BUSINESS MODEL: FROM FULLY INTEGRATED TO OPEN INNOVATION The process of bringing new treatments from Bbench to bedside^ has been described as a translation continuum, where various types of resources and areas of knowledge are involved when moving from basic fundamental research to proven clinical application and, eventually, medical practice (and back) ( 3 ). Major advances in a.o. molecular biology, functional genomics, and genetics have resulted in enhancing our scientific understanding of disease pathophysiology ( 4 ). Furthermore, the different ‘omics’ fields have enabled largescale measurements of biological processes ( 5 ). Around the turn of the century, the general expectation by the global biomedical community, both public and private, was that this increased knowledge base would fuel the development of a new generation of innovative therapies, impacting public health ( 5 ). Despite the promise, these major advances in science have not yet increased the success rate of moving a compound from lab bench to first-in- man to approval and into medical practice ( 6,7 ). In fact, the rate of development by pharmaceutical companies of innovative medicinal products which were approved by regulatory authorities slowed down ( 8,9 ), indicating that a number of chasms continue to exist in the translational process. One explanation for the high rate of attrition in drug development mentioned in literature is that industry is increasingly targeting more complex diseases. Another is that regulatory authorities have become increasingly demanding ( 10–12 ). The pharmaceutical industry is already confronted with the patent expiry of many blockbuster drugs, the so-called ‘patent cliff’ ( 13 ). The disappointing rate of development attrition has been a strong motivator, especially for large companies, to thoroughly review drug development strategies and accompanying processes ( 14 ). Pharma companies know that if the trend remains unaltered, costs of developing a drug will continue to increase and ultimately render the drug development process untenable ( 15 ). This would jeopardize the longterm profitability that allows them to reinvest in future R&D. As a first response, a wave of mergers and acquisitions has taken place to save costs and to counter the weakening pipeline. However, several studies have shown that these mergers and acquisitions have not resulted in improvement of R&D productivity ( 9,16,17 ). What seems to have been a more effective Bproductivity improvement^ strategy has been the increased focus of the pharmaceutical sector on developing medicinal products for rare diseases, so-called orphan drugs ( 18 ). Recent 10-year approval/positive opinion overviews presented by the FDA reveal an upward trend in the number of drug approvals/positive opinions since 2010 ( 19 ). In 2015, almost half of the approvals/positive opinions in the US were for orphan drugs ( 20 ), compared with one third in 2005 ( 21 ). A similar trend is observed for Europe ( 22 ). Alongside an increased focus on rare diseases, the role of large pharmaceutical companies within the field of pharmaceutical innovation has been changing over the past 2–3 decades ( 23 ). Companies continue to move away from the traditional Bfully integrated discovery and development company^ model, towards the adoption and implementation of various open innovation strategies such as strategic alliances, purchase of scientific services and in-licensing ( 24,25 ). In the various models, companies tend to focus internally more on late (phase II/III) clinical development and distribution of products. At the same time, the open innovation models allow them to access efficiently the best science ( 26,27 ). Consequently, the discovery of potentially new therapies and subsequent pre-clinical and early (phase I/IIa) clinical evaluation have increasingly become the domain of academic parties and small and medium-sized enterprises (SMEs). Ultimately, the translation of basic scientific research into a potential treatment modality is then driven by Bfully integrated discovery and development networks^, in which expertise and knowledge from academic parties, large pharmaceutical companies, SMEs, and other stakeholders converge in partnerships ( 25,28,29 ). As exemplified in Table 1, there is increasing focus on better understanding open innovation models in the bio-pharmaceutical sector and their impact on R&D productivity. This paper specifically focuses on the emergence of PPPs and their socio-economic impact. UNIVERSITY-INDUSTRY COLLABORATION TO SPUR BIOMEDICAL R&D Collaboration in R&D between academia and the pharmaceutical industry is not itself a novel concept. It has been around for a while, with the aim of sustaining pharmaceutical innovation and advancing new product development ( 32,33 ). The US is a prime example, where the Bayh-Dole Act of 1980 is considered an important spur for numerous collaborations between academic research groups and small biotech companies or large pharmaceutical companies ( 34,35 ). This development is part of a shift that universities have made, especially in the US and the EU, through which they consider technology transfer and commercialization as an integral part of their mission, a trend dubbed by D’Este and Perkman as the BEntrepreneurial University^ (36). University-industry collaboration within biomedical R&D over the last decades has moved beyond vertical, bilateral arrangements ( 37 ) such as contract research, consulting, and research agreements with an option to license drug candidates and technologies. It has evolved into more horizontal, multistakeholder public-private partnerships (PPPs) ( 38,39 ). One important reason is that the multi-stakeholder PPP model Article Organizational modes for Open Innovation in the biopharmaceutical industry: An exploratory analysis Measuring Open Innovation in the Michelino et al. ( 30 ) Bio-Pharmaceutical Industry Models for open innovation in the pharmaceutical industry Schuhmacher et al. ( 31 ) Authors Concise description Based on two rounds of interviews with 20 industry experts, a model describing the adoption of inbound and outbound open Innovation by bio-pharmaceutical companies was presented Bio-pharmaceutical firms employ a mix of open innovation modes (i.e. licensing agreements, alliances and supply/provision of technical and scientific services) to engage with large pharmaceutical companies, biotech firms and universities with the aim to acquire or commercially exploit technologies and knowledge. Focus on degree of openness of 126 global top R&D spending companies (Period 2008–2012) Model built from three different perspectives: inbound versus outbound processes, economic versus financial transactions and the nature of the traded entities: research and development, intellectual property and know-how Negative correlation of openness degree with firm age, dimension and efficiency, with biotech companies being more open than pharmaceutical ones. Based on an analysis of their R&D models, 13 multinational pharmaceutical companies were categorized with respect to their preference in innovation management (introverted or extroverted) and proportion of externally acquired R&D projects (low or high). Based on the analysis and categorization, four types of open innovators were proposed: ‘knowledge creator’, ‘knowledge integrator’, ‘knowledge translator’ and ‘knowledge leverager’ Examples of Research Focusing on Better Understanding the Open Innovation Concept in The Bio-pharmaceutical Sector generates the necessary common ground where governmental bodies, universities, patient organizations, health foundations, and the private sector can combine resources and expertise. PPPs allow these stakeholders to address jointly the grand challenges within the field of pharmaceutical innovation that are of mutual interest. In simple terms, PPPs have the potential to make things happen that would not be possible in their absence ( 40–42 ). Van Ham and Koppenjan defined multi-stakeholder PPPs as Bthe cooperation of some sort of durability between public and private actors in which they jointly develop products and services and share risks, costs and resources which are connected with these products^ ( 43 ). Within the area of pharmaceutical R&D, there are essentially two main types of multi-stakeholder collaborative research initiatives or PPPs ( 44 ): & & Product Development PPPs, which have positioned themselves to address the global prosperity gap by developing pharmaceutical solutions (vaccine, diagnostic, drug) for low and middle income countries with an alternative (not-for-profit) business model ( 45–47 ). Precompetitive PPPs, which aim to generate novel scientific concepts (e.g. disease targets and research models) and infrastructures (e.g. databases) through effective collaboration between multiple public and private entities based on mutual trust, pooling of complementary expertise and knowledge, and sharing of rewards. Potential disputes over issues such as intellectual property are avoided by limiting activities to the precompetitive space ( 48 ). Also important, and closely related to these two types of PPPs, is the group of Access PPPs. These aim to improve accessibility to a specific treatment modality through massdrug administration programs, in order to alleviate disease burden in low and middle income countries and to overcome obstacles in the distribution system of treatments ( 49 ). An interesting initiative to increase patient access to expensive biologics in low and middle income countries is the joint development of affordable, high quality biosimilars by a consortium of companies from low and middle income countries. This development is undertaken in collaboration with academic partners and coordinated by the Utrecht Centre for Affordable Biotherapeutics. Table 2 shows examples of precompetitive and product development PPPs. According to Lim, the average number of multistakeholder PPPs launched per year has grown from 8 in 2001–2003 to 54 in 2011–2013 ( 38 ). A major contributor to this growth has been the launch of various high-profile precompetitive public-private initiatives, involving multiple consortia, such as the European Innovative Medicines Initiative (IMI), the Dutch Top Institute Pharma, and the US Foundation for National Institutes of Health. Faster Cures has included in its consortiapedia catalogue more than 350 consortia profiles, both public-public as well as publicprivate ( 38 ). The literature clearly indicates that the concept of PPP has matured. A non-exhausting bibliometric analysis performed through PubMed (searching by BPublic-Private Sector Partnerships^[Mesh] OR Bpublic-private partnership^[TIAB] OR Bpublic-private partnerships^[TIAB]) revealed over 2000 citations, of which more than 300 could be directly linked to the concept of multi-stakeholder collaborative research. Closer evaluation reveals that as successfully launched PPPs grew over the last decade, the number of citations on the concept of Bbiomedical R&D^ PPP also increased, from around 35 between 2001–2005 to around 225 in 2011–2015. Precompetitive Critical Path Institute Innovative Medicines Initiative Top Institute Pharma Structural Genomics Initiative Product development Medicines for Malaria Venture Drugs for Neglected Diseases initiative International AIDS Vaccine Initiative Pediatric Praziquantel Consortium Mission To foster development of new evaluation tools and standards for drug therapy trials, which accelerates regulatory qualification and medical product approval and adoption. To improve health by speeding up the development of, and patient access to, innovative medicines, particularly in areas where there is an unmet medical or social need. To establish, support and manage public-private collaborations between academia and the (inter-) national pharmaceutical industry to create ‘health & wealth’. To catalyze research in new areas of human biology and drug discovery research by focusing on less-well-studied domains of the human genome. To reduce the burden of malaria in disease-endemic countries by discovering, developing and facilitating delivery of new, effective and affordable antimalarial drugs. To develop new drugs or new formulations of existing drugs for people living with neglected diseases. To ensure the development of safe, effective, accessible, preventive HIV vaccines for use throughout the world. To develop, register and provide access to a suitable pediatric praziquantel formulation for treating schistosomiasis in preschool-age children. Ref. ( 50,51 ) ( 52,53 ) ( 54 ) ( 55,56 ) (57) ( 58 ) ( 59 ) ( 60 ) Such a high number of citations demonstrates that the concept of PPP has been adopted fully by funders, academia, and the private sector as an important tool that unites two intrinsically different systems of knowledge creation. Within academia, the emphasis is on curiosity-driven, conceptual, publicly accessible research with a long horizon. Within industry, competitive advantage is pursued by shielding in-house research findings and disclosure through patents ( 61 ). This intrinsic difference was once considered as being the core of the obstacles to successful public-private collaboration in biomedical R&D ( 62–64 ). Organizational aspects of boundaryspanning activities between universities and industry ( 62 ), such as conflicts over IP and university administration ( 64 ), have also been mentioned by industry as an important barrier towards successful collaboration with universities ( 65,66 ). It appears that PPPs in the precompetitive stage, where scientific concepts are developed, offer a perfect level playing field for all stakeholders to facilitate translational medicine, and ultimately to improve pharmaceutical R&D productivity. climate that would allow for such a strategy, public-private partnerships have been proposed.^ (67). We fully agree with this statement, but full exploitation of basic science into clinical utility also requires more. It needs the lifting of existing boundaries between scientific disciplines and stakeholders, resulting from traditions, policies, and bureaucracies ( 68 ). Apart from an increased level of Btransdisciplinarity^, exploitation depends on integrating the contributions of the major stakeholders involved in translational medicine: academia, governmental bodies, small and large biopharmaceutical industry, health foundations, and patient organizations. The successful integration of resources and expertise from multiple stakeholders, especially between universities and industry, is built on a clear understanding and alignment of their missions and needs within biomedical R &D. Furthermore, public and private stakeholders play different roles in the biomedical research translation continuum ( 3 ), so an awareness of the intrinsic motivation of the various stakeholders seeking a multi-stakeholder PPP collaborative model will help to value properly public-private collaboration. WHAT ARE THE MAIN DRIVERS FOR KEY STAKEHOLDERS TO EMBRACE THE MULTI-STAKEHOLDER PPP MODEL? As former NIH director Elias Zerhouni put it, "The way forward in multidisciplinary research is to engage in predictive, personalized, preemptive and participatory medicine. For the creation of the optimal innovation UNIVERSITIES Public organizations, mainly universities, focus primarily on technology push. Their footprint can be seen at all stages of the research and development continuum. They are the major providers of basic science concepts, and universities, especially in the US and the EU, now consider commercialization of their research activities an integral part of their mission, as mentioned above. Although commercialization has been identified as a reason for universities to engage with industry ( 36,69 ), academics in particular are driven by a number of motivators: 1) the opportunity to learn about industry challenges and research activities, with feedback on applicability of research and the chance to become part of a network; 2) access to in-kind resources (materials, research expertise and equipment); and 3) access to alternative sources of public and private funding. The term Btechnology^ encompasses, among other things, biochemical findings and new molecular entities, implying an innovation which is pushed through R&D to the market. Here, academic research groups act based on curiosity and on their passion for performing hypothesis-driven research. Crowley mentions the intrinsic level of unpredictability as an important and also attractive characteristic of curiosity-driven research ( 70 ). There are various examples of important solutions to clinical problems that were derived from basic science that could never have been predicted at their outset, such as the successful treatment of leukemia with Bcr-Abl tyrosine kinase inhibitors and the clinical use of bisphosphonates ( 70 ). PHARMACEUTICAL INDUSTRY Private entities, mainly from the industrial side, focus on market pull. They react to the needs of patients through market analysis, and have the appropriate resources, knowledge and capabilities at their disposal to bring new, promising and often innovative therapies to the market. As discussed above, the pharmaceutical industry has increasingly embraced different models of open innovation, including PPPs ( 25 ), because of the need to target diseases of greater complexity, and because they are faced with a high attrition rate in drug development. Various scholars ( 63,71,72 ) have listed key incentives for industry seeking collaboration with universities. These include gaining access to basic knowledge; improving the ability to solve problems; gaining access to new tools and techniques for the development of new technologies; improving a firm’s reputation in the labor market and among potential partners; entering into the academic network; and exploiting opportunities for public funding. Several studies and comments have pointed out that funding of clinical research by the pharmaceutical industry is strongly associated with pro-industry results ( 73–76 ). We do not take this concern about the influence of the pharmaceutical industry on the development and use of medicines lightly. However, we also believe that influence in the basic science stage, which is by its nature precompetitive and concept-driven, will be limited for several reasons. First, using data from the Carnegie Mellon Survey on industrial R&D, and investigating over 30 manufacturing industries, Cohen et al. showed that the pharmaceutical industry is unique and highly dependent on basic science (77). The latter has been confirmed by other studies ( 78 ). Second, as summarized by Perkmann ( 71 ), there are several studies showing that university-industry collaboration doesn’t negatively impact university’s research productivity, nor does it result in shifting away from basic science and more towards applied research. Finally and perhaps most importantly, as mentioned above, the intrinsic motivation of academics to engage with industry goes beyond mere financial gain ( 36 ). This is especially true in a PPP setting, in which funding of public partners and SMEs is in general provided by governmental bodies and/or health foundations ( 52,79 ). Taking the various positions of both public and private entities into consideration reveals that they are really interdependent (80). As stated by Crowley ( 70 ), BAcademia and industry need each other to effect substantive improvements in health. Without substantive collaborations between the two, maximum advances in healthcare for the US public are unlikely^. HEALTH FOUNDATIONS Apart from universities and industry, other stakeholders have also started to embrace this type of partnership to reach their objectives. Health foundations or charities increasingly demonstrate that PPPs can be instrumental in obtaining maximum benefit for the patients they serve ( 79 ). In reviewing their research portfolio, these organizations could see that there was considerable progress towards advancing basic disease understanding. However, it was also observed that translation of this knowledge into clinical proof of concept was lagging considerably, with a need for alternative strategies. Of course, health foundations do not possess the budgets of larger pharmaceutical companies or governmental funders ( 81,82 ), but, based on a survey amongst 11 major US health foundations, Chang identified a number of non-monetary value drivers that health foundations consider of added value in a partnership (83). First, health foundations can provide the necessary clinical network and access to patients. Second, because of their central position, foundations have relationships with and unfettered access to the world’s leading academic experts who can facilitate understanding of the basic science, but who can also play a role in clinical trial design and evaluation. Third, as already mentioned, foundations can help to attract additional funding to a partnership, and place great emphasis on collaboration and information sharing. Finally, although knowledge sharing amongst partners in a PPP can be cumbersome due to concerns about IP/trade secrets and/or other adverse competitive consequences, health foundations have demonstrated that they can act as a trusted intermediary or broker. They can ensure proper storage, handling, analysis and/or mediated revealing of information (e.g. aggregation and anonymization of information before it is distributed within the PPP). PATIENT ORGANIZATIONS Numerous health foundations, such as the Cystic Fibrosis Foundation ( 84 ) and the Polycystic Kidney Foundation ( 85 ), were originally founded by patients (or their carers) with a key objective: Bto find treatments and a cure^ for a specific disease or group of diseases ( 86,87 ). A review of 75 Innovative Medicines Initiative (IMI) projects reveals that European or international patient associations (such as EURORDIS, the European Patient Forum and the International Alliance of Patients’ Organizations) are participating in a total of 16 IMI PPPs (88). Although not all 16 projects provide detail on organizations’ roles in the work program, closer examination reveals that patient participation occurs at three levels ( 89 ). The lowest level of participation is to support partnerships with the dissemination of project results to patients and public. Examples in this respect are the IMI projects EPAD ( 90 ) and PharmaCog ( 91 ) projects. The next level of patient participation, best exemplified by GetReal ( 92 ), Protect ( 93 ), and U-Biopred ( 94 ), is to ensure that the perspective of patients and patients’ organizations is included from the start of the work program via focus groups, patient input platforms, and memberships to other project governance bodies. Finally, EUPATI is a prime example of the highest level of patient participation, a patient-led project. It focused on training close to 100 patients or their representatives in all aspects of medicines development, and on developing an extensive, multilanguage training toolbox to be rolled out across Europe ( 95 ). REGULATORY AGENCIES Finally, regulatory and health technology assessment agencies, such as the EMA and the Dutch Medicines Evaluation Board, also see the PPP model as a platform that allows them to share data, expertise, and resources, and to enhance their understanding of other stakeholders’ perspectives ( 50,51,96–98 ). Nowadays, these public bodies participate in a variety of PPPs, including those focusing on the development of new evaluation tools and standards (Adapt-SMART, C-Path, Escher ( 97–100 )); training programs in drug regulatory sciences e.g. PharmaTrain (101); and new approaches for incorporating real life data into drug development (e.g. GetReal ( 92 )). HOW TO EVALUATE THE PERFORMANCE & IMPACT OF PRECOMPETITIVE RESEARCH PPPS With so many PPPs launched in the last two decades, we agree with Freire that it is time to Bmove beyond the buzzword^ of PPPs as engines that simply facilitate the translation of basic fundamental research, looking also at their impact on proven clinical application and eventually, medical practice ( 102 ). Like biomedical research in general, there is a growing demand to better understand the socio-economic value of biomedical R&D PPPs ( 103,104 ). Apart from understanding socio-economic value to ensure continued funding support, other identified reasons for this move towards evaluating the (research) impact of PPPs include the requirement for accountability. Research organizations need to judge and manage their performance and to improve understanding of which research (model) ultimately leads to the desired impact (103). In the last decade, various scholars have independently investigated indicators to evaluate the performance of research PPPs ( 2,105 ) and of university-industry alliances in general ( 62,63,106 ). As depicted in Table 3, the proposed frameworks for a performance measurement system are comparable, and follow the basics of the logical framework method. This involves defining objectives and securing the necessary inputs through to producing a concrete output and eventually generating the desired impact (107). The adoption of unambiguous definitions for each stage, generally accepted by all stakeholders, will be essential, because this ensures a clear distinction between input, process, output, short term (or intermediate) outcomes and long term outcomes/impact. Building on existing literature ( 2,62,108,109 ), we propose the definitions that follow below as a starting point for further discussion. The input stage revolves around Bthe PPP’s ability to bring together human, financial and physical resources, public and private researchers’ capabilities and motivation, and the partners’ knowledge and experience^. These inputs provide the basis for the following process stage, which reflects Bthe PPP’s ability to pursue a highquality research program, as well as educational and/or other services that are relevant to all stakeholders, and to create an environment of genuine trust that allows ample sharing of knowledge and resources throughout the PPP^. The process stage will result in the generation of various concrete outputs or Bthe PPP’s ability to generate immediate tangible scientific knowledge, products and/or services^. Depending on Bthe PPP’s ability to encourage their use in the short term^, these outputs will translate into short term (or intermediate) outcomes. Finally, long term outcome/impact can be defined as Bthe longer term economical and health impact of the PPP’s generated output, for society, patient as well as the stakeholders involved^. As depicted in Table 4, a number of performance indicators have been suggested for each stage of the proposed logical framework, to allow the implementation of PPP performance measurement in practice ( 2,62,105 ). A number of indicators are also included for biomedical research output and outcome that are appropriate for research PPPs ( 110 ). Finally, building on top of work by Denee et al. and Thonon et al. ( 2,110 ), we suggest classifying the indicators into five categories or domains: Networks and collaboration; research activity and Logical framework key stages Perkmann et al. (62) Denee et al. ( 2 ) Intermediate outcomes Final outcomes Innovative Medicines Initiative ( 105 ) # Resources, processes and facilities that can link inputs to outcomes knowledge; knowledge sharing and dissemination; human capital; and financials and operations. Establishing a framework, a clear definition of the key stages, and a set of performance indicators provides a good starting point for implementation of a meaningful PPP performance measurement system. However, several challenges, especially around outcomes and long-term impact, have been reported that will need to be taken into consideration to ensure that a PPP performance measurement system is implemented that is endorsed by all the stakeholders involved. To start with, the lengthy timeline associated with pharmaceutical R&D, combined with research becoming more and more multidisciplinary in nature (and consequently involving many different processes, individuals, and organizations), makes it challenging to attribute directly a specific contribution from a group or PPP to a specific health or socio-economic impact ( 103,111 ). On top of this, ‘knowledge creep’ is considered a challenge when assessing the impact of research, because policy makers have a tendency to respond slowly to translating accumulating evidence into policy and/or guidelines, and often do so without recognition of the contributing research (112). In line with Penfield et al. ( 103 ), given that the type of impact stakeholders anticipate varies according to the type and scope of a PPP, impact-specific challenges may prohibit a fair comparison of impact between PPPs. Linking the results of a clinical trial (output) to clinical benefit (outcome) will obviously be more straightforward than discovery of a new drug candidate or a specific training program to clinical benefit ( 1 ). A recent systematic review by Thonon et al. revealed a total of 57 indicators to measure output and outcome of medical research. Most identified indicators were in fact the many different indexes (e.g. h-index) currently available to measure research production and impact. With research being complex, non-linear, and unpredictable in nature ( 113 ), there is a propensity to ‘count what can be easily measured’ ( 114,115 ), rather than ‘measuring what counts’ in terms of significant, enduring changes ( 114–116 ). We concur with this, however the underlying problem is not the availability of possible indicators, but rather the measurability of indicators, especially outcome ones. A series of expert workshops, using IMI and TI Pharma as examples, revealed that, with regard to output, 9 out of a total of 10 suggested indicators were considered measurable; however, with regard to outcome/impact, only 2 out of a total of 12 suggested indicators were considered measurable (109). This difficulty of measuring outcome, especially in disciplines such as social and policy science, was also reported in a recent analysis of 162 impact case studies from the 2014 UK Research Excellence ( 117 ). As highlighted by Perkmann et al., the intangibility of many outcome indicators will require the identification of suitable and measurable proxies ( 62 ). This is especially true when it comes to multistakeholder public-private collaborations, for which various soft elements of critical success factors have been identified, such as trust ( 118 ), value recognition ( 119,120 ), communication ( 34,119,121 ), and stakeholder involvement ( 119–122 ). Research impact assessment studies are all part of a relatively new scientific endeavor. With PPPs becoming increasingly multidisciplinary, involving multiple stakeholders with competing interests and generating a multidirectional exchange of knowledge, a recent review of the normative literature on infrastructural PPPs suggests that the logic model of impact assessment (cf. Table 3) doesn’t fully capture the interaction between researchers, funders, innovation intermediaries and end-users ( 123 ). Currently, other models have been developed and are under evaluation, which take into consideration the multiple-stakeholder complexity mentioned above ( 124 ). Several scholars have started to argue that in the end, research impact assessment is most likely to be assessed through a multi-indicator, multi-method approach ( 113,114 ) involving a mix of quantitative analysis (e.g. publications, citations, patents); interviews with key stakeholders; peer assessment; case studies; and ultimately routine engagement with the end-users of research. PRECOMPETITIVE BIOMEDICAL PPPS STARTING TO GENERATE TANGIBLE OUTCOMES A quick glance of the scientific literature reveals a number of reviews highlighting that considerable outputs, such as publications, patent applications, tools, in silico and animal models, training modules, and open-source databases have been generated by the various biomedical R&D PPPs initiated in the last two decades ( 40,42,55,97,125–127 ). However, as mentioned above, the true success of the precompetitive multistakeholder public-private collaboration model ultimately goes beyond mere outputs. Success is about its ability to generate tangible outcomes, and in the long run a sustainable socioeconomic impact. Questions that must be addressed include: Which research has translated into novel disease targets, or in policy and/or guideline changes? To what extent have developed tools and models been adopted by relevant stakeholders? What is the number of end-users of the various infrastructures and databases that have been generated? How many PPPs continue beyond their original term? In contrast to the increasing number of publications on the concept of biomedical R&D PPPs mentioned above, publications focusing specifically on (short-term) outcomes and impact of biomedical R&D PPPs supported by tangible evidence are still limited. CASE STUDIES In the following sections four case studies differing in size, geographical spread and research focus will be discussed. The Structural Genomics Consortium (SGC) In 2014, RAND Europe and the Institute on Governance evaluated the SGC model of operation ( 56 ). SGC is a precompetitive public-private partnership that was founded in 2003, and operates from six academic institutions - the University of Toronto (Canada); the University of Oxford (UK); Karolinska Institute (Sweden); University of North Carolina (US); the State University of Campinas (Brazil); and the Goethe University Frankfurt (Germany). The partnership comprises an open collaborative network of scientists in hundreds of universities around the world and in nine global pharmaceutical companies. Its main goal is to accelerate Bresearch in human biology and drug discovery by making all of its research output freely available to the scientific community^ ( 128 ). The focus is on providing a boost to drug discovery by determining 3D protein structures in biomedically relevant areas in a costeffective manner. Apart from perspectives on the approach, strengths and weaknesses of the model and lessons learnt, part of the evaluation was to extract the value of knowledge in line with the framework presented above: outputs, outcomes and impacts of the SGC. In 2013, the SGC had produced a total of 1195 protein structures and 83 sequences; 17 chemical probes and compound tools; and 98 antibodies. It made these publicly accessible through established databases (Protein Data Bank, Uniprot). More than 1400 clones were provided to both the private and academic sectors. This considerable output was accompanied by 452 peer-reviewed publications, including Nature, PNAS and PLoS One, and by attendance and presentations at well over 250 conferences. With regard to outcomes, defined as Bthe utility of those outputs in delivering additional impacts^, the report focused on three aspects: the reach of the SGC research, the influence of SGC researchers and the economic impact of SGC developments ( 56 ). Using the list of indicators and categories depicted in Table 4 as a point of reference, the reach of the SGC was identified in the area of human capital (collaboration with over 500 scientists and training of over 300 scholars). Its influence was acknowledged in the areas of human capital (over 100 staff moved to academia, industry and business schools) and knowledge sharing & dissemination (around 100 meetings with industry, policy maker discussion and workshops). Assessing the economic impact of the SGC, such as the costs of new products or valuation of new companies, was difficult. Apart from a $15 m financing of sp in -out Tensha Th erap eu tics, a second spin-ou t (1DegreeBio) estimated a potential $1bn cost-savings in R&D expenditure relating to the identification of effective antibodies ( 56 ). Finally, based on several assumptions, the revenue of products related to the SGC was appraised at a value of over $60 m CAD. Important to note is that apart from these tangible deliverables, more intangible ones were also mentioned. As mentioned by private sector representatives in interviews, the SGC also provided a platform that allowed pooling of resources and sharing of expertise, saving costs on bureaucracy and generating a more efficient research processes. The Innovative Medicines Initiative (IMI) The difficulty to attribute scientific outcomes to input mentioned earlier was confirmed by the SGC evaluators as a key limitation for a robust analysis. Research is not performed in splendid isolation, but is part of a complex network of interactions that contribute jointly to a specific scientific breakthrough ( 129 ). This complexity, and also the existing lengthy timelines for pharmaceutical R&D ( 5 ), was also mentioned in a recent report on the socio-economic impact of the IMI ( 105 ). With the aim of improving the competitive situation of the European Union in the field of pharmaceutical research, the European Commission (EC) and the European Federation of Pharmaceutical Industries and Associations (EFPIA) implemented the IMI in 2008 ( 52,98 ). With a total budget of over EUR 5 billion (IMI1: 2 billion in 2008–2013; IMI2: 3.3 billion in 2014–2024), IMI is generally considered the largest biomedical PPP in the world. It aims to support the development of next generation vaccines, medicines and treatments, focusing on, among others, the improvement of the current drug development process and reduction of the time to reach clinical proof of concept. Currently, IMI has funded over 80 public-private consortia spanning the complete drug development cycle, including training, post-marketing and regulatory science. In 2016, the IMI appointed an expert group to assess the socio-economic impact of the first nine IMI projects (total value >EUR 200 million; completed early 2016) of which eight were focused on precompetitive research and one on training. Although in the report the complexity of interactions between various measures of inputs, outputs, outcomes and impact was acknowledged, the expert group chose to adhere to the logical framework (See Table 3) to assess the performance of the project. In terms of output, the nine projects combined generated over 550 scientific publications (proxy for research activity and knowledge sharing); cell-based or animal models in five different (disease) areas; three biobanks (diabetes, medicine-induced injuries, pain); novel imaging techniques in the area of diabetes; new or improved biomarkers; a patient cohort database; and novel tools to test biomarkers. In addition, a plethora of novel targets, processes, approaches, and methods were reported that could incite ongoing and new collaborative research. Tangible outcomes that were mentioned in the report were four start-ups/spinoffs; patents; several assay products; tools and animal models being commercialized; and a training program. Taking into consideration the complexity of pharmaceutical R&D and its lengthy timelines, the expert group considered tangible socio-economic impacts, such as delivery of novel treatments; cost and time savings; reductions in risk and attrition rate; and the reduced need for animal testing beyond the scope of the IMI. The impact IMI projects can generate is primarily related to improvements and amendments within the medicines development process itself, also defined by the expert group as the Bpathway to socio-economic impact^ ( 105 ). However, generating true tangible socio-economic impact will rely heavily on recognition of the benefits arising from the IMI, and subsequent actions by pharmaceutical companies, regulators, payers and policy makers through adoption of novel processes, the implementation of novel or updated guidelines and policies, and so on. The Dutch top Institute Pharma (TI Pharma) This PPP (total budget: EUR 274 million) ran between 2006 and 2014 (motto: ‘health and wealth’) and built pharmaceutical research and development networks in five disease areas: (auto)immune diseases, cardio-vascular diseases, cancer, infectious diseases, and brain diseases. Enabling technologies included therapeutic target finding; validation & animal models; lead selection & in-silico modelling; predictive drug disposition & toxicology; biomarkers & bio-sensoring; and drug formulation, delivery & targeting. In addition to contributions by Dutch universities, Dutch SMEs and international big pharma companies, the Dutch government supported this PPP with a total contribution of EUR 135 million. TI Pharma operated as an independent body, and the only condition for the Dutch government’s contribution was that projects were in line with the recommendations from the ‘WHO Priority Medicines report’ published in 2004 and (updated in 2013 ( 130 )). A total of 26 academic institutions, 43 SMEs, and 20 big pharma companies were partnering via 74 TI Pharma projects. At the end of the 2013, TI Pharma’s output consisted of 470 trained PhD and post-doctoral fellows; almost 750 publications; 41 lead compounds, lead series and libraries; 18 novel formulations; 11 biomarkers; 33 preclinical models; 28 clinical models; 11 research databases; and 87 research tools (assays, discovery models) ( 54 ). Several tangible early outcomes were reported. Regarding human capital, of the 257 TI Pharma fellows who had finished their project by the end of 2013, an impressive 98% continued their career in industry, academia and other pharma (e.g., regulatory) and non-pharma organizations. In the area of research activity and knowledge, clear benefit for patients is anticipated through the delivery of a safe morphine dosage regimen for newborns; new clinical protocols for COPD; a vaccine candidate against acute myeloid leukemia; and a disease registry for rare metabolic diseases, which will facilitate development of diagnosis and treatment for this specific group of diseases. A total of 74 follow-up projects could also be identified, which built on the outputs of various TI Pharma projects (Networks & collaboration). Finally, in 2015 TI Pharma merged with another Dutch PPP, resulting in Lygature and continues to successfully manage international PPP projects (current portfolio: >15 projects). The Alzheimer’s Disease Neuroimaging Initiative (ADNI) One of the most well-described precompetitive biomedical R&D PPPs is the Alzheimer’s Disease Neuroimaging Initiative or ADNI ( 131–137 ). ADNI was initiated in 2004, and with a total budget of more than $150 million is considered the biggest PPP in Alzheimer disease (AD) research. Alzheimer disease is a complex disorder affecting tens of millions of people around the world ( 138 ), and translation of disease understanding into viable treatment options remains disappointing ( 139 ). Apart from its intrinsic complexity, one major limitation faced by both academic and pharmaceutical partners during clinical trials was that outcome measures were limited to clinical and cognitive ones. Through ADNI, US leading Alzheimer research centers, the National Institute on Aging, 13 pharmaceutical companies and two not-forprofit foundations all combined resources and expertise to facilitate development of effective AD treatments, by developing, validating, and qualifying a novel set of imaging and other biomarkers for clinical trials ( 132,133 ). Originally planned as a five-year project (ADNI-1), with an important coordinating effort by the Foundation for the National Institutes of Health, the partnership has been successful in: 1) attracting additional funding to extend (ADNI-GO) and renew (ADNI-2 & ADNI-3) the overall project until mid-2021; 2) increasing the number of partners to more than 30; and 3) attracting the Canadian Institutes for Health Research to join (133). To date, there have been no successful clinical trials with ADmodifying drugs. However, Weiner et al. and Jones-Davis and Buckholtz recently summarized the key outputs and outcomes of ADNI, which in essence started out as a multi-site, longitudinal study of normal cognitive aging, mild cognitive impairment, and early AD ( 132,133 ). All data (PET, MRI, clinical, biospecimens, genetics) acquired during the ADNI project to date is housed in a database hosted by the Laboratory of Neuroimaging, and is open to the entire scientific research community. This has resulted in over 14 million downloads, and more than 750 publications citing the use of ADNI data ( 132,140 ), of which a number are in high-impact journals, such as Nature, PNAS, and PLOS One (141). Populating the database continues, with the upload of data derived from the use of ADNI samples. Apart from ensuring its own sustainability, ADNI has also looked beyond the US border ( 132 ). It has been a key driver in promoting the ADNI concept worldwide, resulting in ADNI projects based in North America, Europe, Japan, Australia, Korea, and Argentina, w hich are united through Worldwide-ADNI ( 142 ). The ADNI concept has also inspired other initiatives focusing on closely related aspects of AD (e.g., traumatic brain injury as an AD risk factor) and other PPPs (e.g., developing biomarkers for Parkinson’s disease). By making data and samples available, the infrastructures and databases described above have also made an important contribution to enhancing the basic understanding of various complex diseases. As well as providing open-access to data and sharing of samples, ADNI has also made available a set of protocols and methods related to the study itself, to MRI and PET analysis and acquisition, and to biomarker and proteomic analysis. This allows comparison of data gathered across multiple sites. According to Weiner et al. Bpharmaceutical companies developing disease-modifying treatments for AD and studies funded by the National Institutes of Health and private foundations have used ADNI methods in virtually all their clinical trials^ ( 132 ). The knowledge of the progression of AD pathology, of biomarker interrelationship, and of genetic risk factors for the disease that ADNI has generated, is also being acknowledged in recently issued draft regulatory guidelines on clinical investigation of medicines for the treatment of AD and other dementias ( 143,144 ). In addition, to helping enrich recruitment in regulatory clinical trials for Mild and Moderate AD, the first two biomarkers (CSF Aβ and tau protein; and low hippocampal volume) have been or are in the process of being qualified by the EMA and FDA ( 145,146 ). CONCLUDING REMARKS Despite the intrinsic difficulties involved in measuring the impact of biomedical PPPs, recent evaluations of the SGC, IMI, TI Pharma and ADNI highlighted in this review provide clear evidence that precompetitive biomedical PPPs have started to generate tangible outcomes. These outcomes are in areas where not only the pharmaceutical industry and academia, but also health foundations, patient organizations and regulatory agencies share a mutual interest. These include: 1) providing the necessary infrastructure and databases to facilitate the discovery of novel disease targets and leads; 2) increasing basic understanding of complex diseases, including novel models and standards; 3) capacity building through jointly designed training programs; and 4) providing a platform to allow exchange of different perspectives on, for example, the regulatory system and regulatory science ( 38,44,147 ). Considering the long timelines for uptake of validated new developments in the biomedical/pharmaceutical field, continuation of PPPs beyond their original time span (often 4– 6 years) will certainly contribute to PPPs reaching their full potential. This can be achieved either through a continuation of funding (ADNI, IMI) or by changing the business model (TI Pharma, IMI-Pharmatrain). As highlighted above, pharmaceutical R&D is complex and characterized by lengthy timelines, and one could easily argue that the challenges mentioned in this review make it very difficult to properly assess the long-term impact of the multi-stakeholder PPP concept on medical practice. However, we should continue to improve the metrics. Given the growing number of PPPs, the availability of funding and resources will become a limiting factor, and will Bforce^ stakeholders to become even more prudent in selecting which PPP initiatives to back or join, and which ones to avoid ( 1 ). In simple terms, partners, funders, and civil society will increasingly seek confirmation of the incremental value achieved through partnerships. This paper highlights a link between the emergence of the PPP concept and the growing ‘popularity’ of open innovation in the biomedical/pharmaceutical R&D world. One may wonder how this open innovation model will evolve, and both whether and how the format of PPPs will change with it. We have already seen changes since the time when a typical PPP was based on an academic and industrial pillar, with governmental or other third party funding as ‘carrot’. Over time, health foundations, patient organizations and regulatory scientists have regularly joined in ( 79,85,94,96 ). So, what is next? More funding and ‘research priority setting’ from philanthropists, also coined as philanthrocapitalism (148)? The future will tell. ACKNOWLEDGEMENTS AND DISCLOSURES Nothing to declare Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. 1. Pardoe D , Hunter J , Cooke R , Barrett P . Assessing the value of R & D partnerships . Drug Discov World. 2010 /11;Winter: 9 - 17 . 2. Denee TR , Sneekes A , Stolk P , Juliens A , Raaijmakers JA , Goldman M , et al. Measuring the value of public-private partnerships in the pharmaceutical sciences . Nat Rev Drug Discov . 2012 ; 11 ( 5 ): 419 . 3. Drolet BC , Lorenzi NM . Translational research: understanding the continuum from bench to bedside . Transl Res . 2011 ; 157 ( 1 ): 1 - 5 . 4. Price AL , Spencer CC , Donnelly P . Progress and promise in understanding the genetic basis of common diseases . In Proc. R. Soc. B 2015 Dec 22 (Vol. 282 , no. 1821 , p. 20151684 ). The Royal Society. 5. Ioannidis JP . Expectations, validity, and reality in omics . J Clin Epidemiol . 2010 ; 63 ( 9 ): 945 - 9 . 6. Kola I , Landis J . Can the pharmaceutical industry reduce attrition rates? Nat Rev Drug Discov . 2004 ; 3 ( 8 ): 711 - 6 . 7. Mervis J. Productivity counts - but the definition is key . Science . 2005 ; 309 ( 5735 ): 726 - 7 . 8. Scannell JW , Blanckley A , Boldon H , Warrington B. Diagnosing the decline in pharmaceutical R&D efficiency . Nat Rev Drug Discov . 2012 ; 11 ( 3 ): 191 - 200 . 9. Munos B . Lessons from 60 years of pharmaceutical innovation . Nat Rev Drug Discov . 2009 ; 8 ( 12 ): 959 - 68 . 10. Chabner BA . Defining a new role for the National Cancer Institute cooperative groups: more science, fewer trials . Oncologist . 2014 ; 19 : 1113 - 4 . 11. Moors EHM , Cohen AF , Schellekens H. Towards a sustainable system of drug development . Drug Discov Today . 2014 ; 19 ( 11 ): 1711 - 20 . 12. Schuhmacher A , Gassmann O , Hinder M. Changing R & D models in research-based pharmaceutical companies . J Transl Med . 2016 ; 14 ( 1 ): 105 . 13. Denoon A , Vollebregt E. Can regenerative medicine save big Pharma's business model from the patent cliff? Regen Med . 2010 ; 5 : 687 - 90 . 14. Gilbert J , Henske P , Singh A . Rebuilding big pharma's business model . In Vivo-New York Then Norwalk . 2003 ; 21 ( 10 ): 73 - 80 . 15. Kaitin KI . Deconstructing the drug development process: the new face of innovation . Clin Pharmacol Ther . 2010 ; 87 ( 3 ): 356 - 61 . 16. LaMattina JL . The impact of mergers on pharmaceutical R&D. Nat Rev Drug Discov . 2011 ; 10 ( 8 ): 559 - 60 . 17. Comanor WS , Scherer FM . Mergers and innovation in the pharmaceutical industry . J Health Econ . 2013 ; 32 ( 1 ): 106 - 13 . 18. Dolgin E. Big pharma moves from Bblockbusters^ to Bniche busters^ . Nat Med . 2010 ; 16 ( 8 ): 837 . 19. FDA . Novel drugs 2015 summary . January 2016 . US Food and Drug Administration . 20. Regulatory Affairs Professionals Society . EMA carries 2014 momentum, recommends 39 new drugs and sets orphan record . 23890/ EMA-Carries- 2014 -Momentum-Recommends- 39 -NewDrugs- and - Sets- Orphan-Record/#sthash. cgU5Cbie.dpuf. Accessed 17 Mar 2017 . 21. Owens J. 2006 drug approvals: finding the niche . Nat Rev Drug Discov . 2007 ; 6 ( 2 ): 99 - 101 . 22. EMA . Human medicines highlights - November 2015. Information on Medicines 2015 . 23. Cockburn IM . The changing structure of the pharmaceutical industry . Health Aff . 2004 ; 23 ( 1 ): 10 - 22 . 24. Bianchi M , Cavaliere A , Chiaroni D , Frattini F , Chiesa V . Organisational modes for open innovation in the biopharmaceutical industry: an exploratory analysis . Technovation . 2011 ; 31 ( 1 ): 22 - 33 . 25. Dixon J. RIPCO , FIPCO, NRDO, FIPNET, VIPC. Nature Biotechnology . 2011 . 2011/05/31/ripco-fipco -nrdo-fipnet-vipco . Accessed 17 Mar 2017 . 26. Hunter J . Challenges for pharmaceutical industry: new partnerships for sustainable human health . Philos Trans a Math Phys Eng Sci . 2011 ; 369 ( 1942 ): 1817 - 25 . 27. Rafols I , Hopkins MM , Hoekman J , Siepel J , O'Hare A , PerianesRodriguez A , et al. Big Pharma, little science? A bibliometric perspective on big Pharma's R&D decline . Technol Forecast Soc Change . 2014 ; 81 ( 1 ): 22 - 38 . 28. Wang L , Plump A , Ringel M. Racing to define pharmaceutical R&D external innovation models . Drug Discov Today . 2015 ; 20 ( 3 ): 361 - 70 . 29. Gehr S , Garner CC . Rescuing the lost in translation . Cell . 2016 ; 165 ( 4 ): 765 - 70 . 30. Michelino F , Lamberti E , Cammarano A , Caputo M. Measuring open innovation in the bio-pharmaceutical industry . Creativity and Innovation Management . 2015 ; 24 ( 1 ): 4 - 28 . 31. Schuhmacher A , Germann PG , Trill H , Gassmann O . Models for open innovation in the pharmaceutical industry . Drug Discov Today . 2013 ; 18 ( 23 ): 1133 - 7 . 32. Blumenthal D , Causino N , Campbell E , Louis KS . Relationships between academic institutions and industry in the life sciences-an industry survey . N Engl J Med . 1996 ; 334 ( 6 ): 368 - 73 . 33. Blumenthal D , Campbell E , Causino N , Louis KS . Participation of life-science faculty in research relationships with industry . N Engl J Med . 1996 ; 335 ( 23 ): 1734 - 9 . 34. Bagley CE , Tvarno CD . Pharmaceutical public-private partnerships in the United States and Europe: moving from the bench to the bedside . Harvard Bus Law Rev . 2013 ; 4 : 373 - 400 . 35. Milne CP , Malins A . Academic-industry partnerships for biopharmaceutical research & development: advancing medical science in the US . Tuft Center for the Study of Drug Development . 2012 Apr.; 36. D'Este P , Perkmann M. Why do academics engage with industry? The entrepreneurial university and individual motivations . J Technol Transf . 2011 ; 36 ( 3 ): 316 - 39 . 37. Stuart TE , Ozdemir SZ , Ding WW . Vertical alliance networks: the case of university-biotechnology-pharmaceutical alliance chains . Res Policy . 2007 ; 36 ( 4 ): 477 - 98 . 38. Lim MD . Consortium sandbox: building and sharing resources . Sci Transl Med . 2014 ; 6 ( 242 ): 242cm6 . 39. Yildirim O , Gottwald M , Schüler P , Michel MC . Opportunities and challenges for drug development: public-private partnerships . Adaptive Designs and big Data Front Pharmacol . 2016 ; 7 : 461 . 40. Stolk P. Priority Medicines for Europe and the World B A Public Health Approach to Innovation ^ Update on 2004 Background Paper Background Paper 8.1 Public Private Partnerships . 2013 ; 1 - 25 . 41. Reypens C , Lievens A , Blazevic V . Leveraging value in multistakeholder innovation networks: a process framework for value co-creation and capture . Ind Mark Manag . 2014 ; 56 : 40 - 50 . 42. van Tulder R , Seitanidi MM , Crane A , Brammer S. Enhancing the impact of cross-sector partnerships: four impact loops for channeling partnership studies . J Bus Ethics . 2015 ; 135 ( 1 ): 1 - 17 . 43. Hodge GA , Greve C. PPPs: the passage of time permits a sober reflection . Econ Aff . 2009 ; 29 ( 1 ): 33 - 9 . 44. Stevens H , Van Overwalle G , Van Looy B , Huys I . Perspectives and opportunities for precompetitive public-private partnerships in the biomedical sector . Biotechnol Law Rep . 2013 ; 32 ( 3 ): 131 - 9 . 45. Croft SL . Public-private partnership: from there to here . Trans R Soc Trop Med Hyg . 2005 ; 99 ( Suppl 1 ): S9 - 14 . 46. De Pinho CK , Norman CD , Jadad AR . Product development public-private partnerships for public health: a systematic review using qualitative data . Soc Sci Med . 2011 ; 73 ( 7 ): 986 - 94 . 47. Wheeler C , Berkley S . Initial lessons from public-private partnerships in drug and vaccine development . Bull World Health Organ . 2001 ; 79 ( 8 ): 728 - 34 . 48. Goldman M. New frontiers for collaborative research . Sci Transl Med . 2013 ; 5 ( 216 ): 216ed22 . 49. Widdus R . Public-private partnerships: an overview . Trans R Soc Trop Med Hyg . 2005 Oct; 99 ( Suppl 1 ): S1 - 8 . 50. Woodcock J , Woosley R . The FDA critical path initiative and its influence on new drug development . Annu Rev Med . 2008 ; 59 : 1 - 12 . 51. Woosley RL , Cossman J . Drug development and the FDA's critical path initiative . Clin Pharmacol Ther . 2007 ; 81 ( 1 ): 129 - 33 . 52. Goldman M. The innovative medicines initiative: a European response to the innovation challenge . Clin Pharmacol Ther . 2012 ; 91 ( 3 ): 418 - 25 . 53. Stevens H , Van Overwalle G , Van Looy B , Huys I. Innovative medicines initiative (IMI) case study analysis reveals the true added value of early-phase public-private partnerships (PPPs) . Biotechnol Law Rep . 2015 ; 34 ( 4 ): 153 - 65 . 54. TI Pharma . TI Pharma new tracks to medicines . 2014 . http:// es/TI%20Pharma% 20Report % 20 -% 20New%20Tracks%20to%20Medicines.pdf. Accessed 17 March 2017 . 55. Edwards A . Team up with industry . Nature . 2016 ; 531 ( 7594 ): 299 - 301 . 56. Morgan Jones M , Castle-Clarke S , Brooker S , Nason E , Huzair F , Chataway J. Genomics Consortium A knowledge platform for drug discovery . 2014 ; RAND Europe with The Institute on Governance:RR-512-SGC . dam/rand/pubs/research_reports/RR500/RR512/RAND_ RR512.pdf. Accessed 17 Mar 2017 . 57. Nwaka S , Riopel L , Ubben D , Craft JC . Medicines for Malaria Venture new developments in antimalarials . Travel Med Infect Dis . 2004 ; 2 ( 3 -4): 161 - 70 . 58. Maxmen A. Big Pharma's cost cutting challenger . Nature . 2016 ; 536 : 388 - 90 . 59. Holmes D. Profile: IAVI-searching for the Achilles' heel of HIV . Lancet. 2013 ; 381 ( 9883 ): 2072 . 60. Kourany-Lefoll E , Keiser J , De Vrueh R . Case study: schistosomiasis . EBR . 2015 , January. P10 - 14 . 61. Murray F , O'Mahony S. Exploring the foundations of cumulative innovation: implications for Organization science . Organ Sci . 2007 ; 18 : 1006 - 21 . 62. Perkmann M , Neely A , Walsh K. How should firms evaluate success in university-industry alliances? A performance measurement system . R D Manag . 2011 ; 41 ( 2 ): 202 - 16 . 63. Piva E , Rossi-Lamastra C . Systems of indicators to evaluate the performance of university-industry alliances: a review of the literature and directions for future research . Meas Bus Excell . 2013 ; 17 ( 3 ): 40 - 54 . 64. Bruneel J , Este PD , Salter A . Investigating the factors that diminish the barriers to university - industry collaboration . Res Policy . 2010 ; 39 ( 7 ): 858 - 68 . 65. Grimaldi R , Kenney M , Siegel DS , Wright M. 30 years after Bayh-dole: reassessing academic entrepreneurship . Res Policy . 2011 ; 40 ( 8 ): 1045 - 57 . 66. Jacobsson S , Lindholm-Dahlstrand Å , Elg L. Is the commercialization of European academic R&D weak? - a critical assessment of a dominant belief and associated policy responses . Res Policy . 2013 ; 42 ( 4 ): 874 - 85 . 67. Luijten PR , van Dongen GAMS , Moonen CT , Storm G , Crommelin DJA . Public-private partnerships in translational medicine: concepts and practical examples . J Control Release . 2012 ; 161 ( 2 ): 416 - 21 . 68. Yamamoto K. Time to play ball . Science . 2013 ; 340 ( 6139 ): 1375 . 69. Ramos-Vielba I , Sanchez-Barrioluengo M , Woolley R . Scientific research groups' cooperation with firms and government agencies: motivations and barriers . J Technol Transf . 2016 ; 41 ( 3 ): 558 - 85 . 70. Crowley WF . Translation of basic research into useful treatments: how often does it occur? Am J Med . 2003 ; 114 ( 6 ): 503 - 5 . 71. Perkmann M. University - industry relations. In: Audretsch D , Hayter C , Link A , editors. Concise guide to entrepreneurship, technology and innovation . Cheltenham: Edward Elgar; 2015 . p. 227 - 33 . 72. Monaghan BJ , Malek A -M, Simson H . Public-private partnerships in healthcare: criteria for success . Healthc Manag Forum . 2001 ; 14 ( 4 ): 44 - 9 . 73. Angell M , Relman AS . Patents, profits & American medicine: conflicts of interest in the testing & marketing of new drugs . Daedalus . 2002 ; 131 ( 2 ): 102 - 11 . 74. Lexchin J , Bero LA , Djulbegovic B , Clark O . Pharmaceutical industry sponsorship and research outcome and quality: systematic review . BMJ . 2003 ; 326 ( 7400 ): 1167 - 70 . 75. Bekelman JE , Li Y , Gross CP . Scope and impact of financial conflicts of interest in biomedical research: a systematic review . JAMA . 2003 ; 289 ( 4 ): 454 - 65 . 76. Sismondo S. How pharmaceutical industry funding affects trial outcomes: causal structures and responses . Soc Sci Med . 2008 ; 66 ( 9 ): 1909 - 14 . 77. Cohen WM , Nelson RR , Walsh JP . Links and impacts: the influence of public research on Industrial R & D. Manag Sci . 2002 ; 48 ( 1 ): 1 - 23 . 78. Toole AA . The impact of public basic research on industrial innovation: evidence from the pharmaceutical industry . Res Policy . 2012 ; 41 ( 1 ): 1 - 12 . 79. De Vrueh RL , Awad W , Stolk A , Dijcks FA , Rijnders TW , Janssen JW . Deal watch: roles and strategies for health foundations in public-private partnerships . Nat Rev Drug Discov . 2014 ; 13 ( 6 ): 406 . 80. Chin-Dusting J , Mirzahi J , Jennings G , Fitzgerald D . Finding improved medicines: the role of academic-industrial collaboration . Nat Rev Drug Discov . 2005 ; 4 ( 11 ): 891 - 7 . 81. Technopolis group. Effects of NGI investments . Study commissioned by the Netherlands Genomics Initiative . Jun 26 2012. 05/1481_NGI-Multiplier -Report.pdf. Accessed 17 Mar 2017 . 82. Research America. U.S. Investments in Medical and Health Research and Development 2013 -2015. Fall 2016 . https:// R%26D _report.pdf. Accessed 17 Mar 2017 . 83. Chang J. Best Practices for Venture Philanthropy Collaborations between Disease- Focused Foundations and For-Profit Life Science Companies . June 2010. Master Thesis; Massachusetts Institute of Technology http://hdl.handle. net/1721.1/58400. Accessed 17 Mar 2017 . 84. Marshall BC , Penland CM , Hazle L , Ashlock M , Wetmore D , Campbell PW , et al. Cystic fibrosis foundation: achieving the mission . Respir Care . 2009 ; 54 ( 6 ): 788 - 95 . 85. Lim MD , Brumfield MA , Goldman M . Philanthropies as partners for drug development in public-private partnerships . J Med Dev Sci . 2016 ; 1 ( 1 ): 10 - 3 . 86. Anderson M , Kimberly MCK . On the path to a science of patient input . Sci Transl Med . 2016 ; 8 ( 336 ): 336ps11 . 87. Anderson M , McCleary KK . From passengers to co-pilots: Patient roles expand . Sci Transl Med . 2015 ; 7 ( 291 ): 291fs25 . 88. Innovative Medicines Initiative. IMI Project portfolio . http:// Accessed 1 Feb 2017 . 89. De Wit M , Bloemkolk D , Teunissen T , van Rensen A. Voorwaarden voor succesvolle betrokkenheid van patiënten/ cliënten bij medisch wetenschappelijk onderzoek . Tijdschrift Voor Gezondheidswetenschappen . 2016 ; 94 ( 3 ): 91 - 100 . 90. EPAD website. European Prevention of Alzheimer's Dementia (EPAD) Consortium http ://ep-ad. org/project-structure/. Accessed 1 Feb 2017 . 91. Alzheimer Europe. PharmaCog. http://www.alzheimer-europe. org/Research/PharmaCog. Accessed 1 Feb 2017 . 92. Panayidou K , Gsteiger S , Egger M , Kilcher G , Carreras M , Efthimiou O , et al. GetReal in mathematical modelling: a review of studies predicting drug effectiveness in the real world . Res Synth Methods . 2016 ; 7 ( 3 ): 264 - 77 . 93. Reynolds RF , Kurz X , de Groot MCH , Schlienger RG , Grimaldi-Bensouda L , Tcherny-Lessenot S , et al. Theories of attempted suicide: should they differ from theories of completed suicide? Pharmacoepidemiol Drug Saf 2016 ; 25 ( Suppl . 1): 5 - 10 . 94. Supple D , Roberts A , Hudson V , Masefield S , Fitch N , Rahmen M , et al. From tokenism to meaningful engagement: best practices in patient involvement in an EU project . Res Involv Engagem . 2015 ; 1 ( 1 ): 5 . 95. Pushparajah D , Geissler J , Westergaard N. EUPATI: collaborating between patients, academia and industry to champion the informed patient in medicines research and development . J Med Dev Sci . 2015 ; 1 ( 1 ): 74 - 80 . 96. Goldman M , Seigneuret N , Eichler H-G. The innovative medicines initiative: an engine for regulatory science . Nat Rev Drug Discov . 2015 Jan; 14 ( 1 ): 1 - 2 . 97. Parekh A , Buckman-Garner S , McCune S , ONeill R , Geanacopoulos M , Amur S , et al. Catalyzing the critical path initiative: FDA's progress in drug development activities . Clin Pharmacol Ther . 2015 Mar; 97 ( 3 ): 221 - 33 . 98. Goldman M , Compton C , Mittleman BB . Public-private partnerships as driving forces in the quest for innovative medicines . Clin Transl Med . 2013 ; 2 ( 1 ): 2 . 99. Ermisch M , Bucsics A , Bonanno PV , Arickx F , Bybau A , Bochenek T , et al. Payers' views of the changes arising through the possible adoption of adaptive pathways . Front Pharmacol . 2016 ; 7 : 305 . 100. Spindler P , Bach KF , Schmiegelow M , Bedlington N , Eichler HG . Innovation of medical products: the evolution of regulatory science, research, and education . Ther Innov Regul Sci . 2015 ; 50 ( 1 ): 44 - 8 . 101. Hardman M. European initiatives for better training in medicines development . J med Dev Sci . 2015 ; 1 : 1 - 9 . 102. Freire MC . Partnerships: beyond the buzzword . Sci Transl Med . 2014 , 6 : 242ed15 . 103. Penfield T , Baker MJ , Scoble R , Wykes MC . Assessment evaluations and definition of research impact . Res Eval . 2013 ; 23 ( 1 ): 1 - 12 . 104. Molas-Gallart J , D'Este P , Llopis O , Rafols I . Towards an alternative framework for the evaluation of translational research initiatives . Res Eval . 2016 ; 25 ( 3 ): 235 - 43 . 105. Innovative Medicines Initiative. IMI Socio-economic Impact Assessment Expert Group Final Report . 2016 , May. Document Reference: IMI2/OUT/2016-01541 http://www.imi.europa. eu/sites/default/files/uploads/documents/Publications/ SocioeconomicImpactAssessment_FINALMay2016.pdf. Accessed 17 Mar 2017 . 106. Seppo M , Lilles A . Indicators measuring university cooperation. Discuss Est Econ Policy . 2010 ; 20 ( 1 ): 204 - 25 . 107. Baccarini D. The logical framework method for defining project success . Int J Confl Manag . 1999 ; 10 ( 2 ): 130 - 53 . 108. UK Evaluation Forum . Report: Medical research: assessing the benefits to society . 2016, May. ISBN No. I-903401-12-7. 109. Dittrich K. How valuable is your partnership? A framework for valuing public-private research partnerships in biomedical sciences . DRUID Society Meetings 2012 . acc_papers/mnsc0qm81x1fx27vt2leqq9fji0i.pdf. Accessed 17 Mar 2017 . 110. Thonon F , Boulkedid R , Delory T , Rousseau S , Saghatchian M , Van Harten W , et al. Measuring the outcome of biomedical research: a systematic literature review . PLoS One . 2015 ; 10 ( 4 ): e0122239 . 111. Hughes A , Martin BR . Enhancing Impact: The Value of Public Sector R & D: Summary report. Council for Industry and Higher Education . 2012 , August uk/impact. Accessed 17 Mar 2017 . 112. Daviter F. The political use of knowledge in the policy process . Policy Sci . 2015 ; 48 ( 4 ): 491 - 505 . 113. Milat AJ , Bauman AE , Redman S. A narrative review of research impact assessment models and methods . Health Res Policy Sys . 2015 ; 13 ( 1 ): 18 . 114. Grant J . Policy resource: Measuring the benefits from research . RAND Europe report RB-9202-RE . 2016 . 115. Bozeman B , Fay D , Slade CP . Research collaboration in universities and academic entrepreneurship: the-state-of-the-art . J Technol Transfer . 2013 ; 38 ( 1 ): 1 - 67 . 116. Wells R , Whitworth J . Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Aust New Zealand Health Policy . 2007 ; 4 : 14 . 117. Greenhalgh T , Fahy N. Research impact in the community-based health sciences: an analysis of 162 case studies from the 2014 UK Research Excellence Framework . BMC Med . 2015 ; 13 ( 1 ): 232 . 118. Osei-Kyei R , Chan APC . Review of studies on the critical success factors for public-private partnership (PPP) projects from 1990 to 2013 . Int J Proj Manag . 2015 ; 33 ( 6 ): 1335 - 46 . 119. Turner JR , Müller R. The project manager's leadership style as a success factor on projects: a literature review . Proj Manag J . 2005 ; 36 ( 1 ): 49 - 61 . 120. De Pinho CK , Norman CD , Jadad AR . Product development public-private partnerships for public health: a systematic review using qualitative data . Soc Sci Med . 2011 ; 73 ( 7 ): 986 - 94 . 121. Belassi W , Tukel OI . A new framework for determining critical success/failure factors in projects . Int J Proj Manag . 1996 ; 14 ( 3 ): 141 - 51 . 122. Buse K , Harmer AM . Seven habits of highly effective global public-private health partnerships: practice and potential . Soc Sci Med . 2007 ; 64 ( 2 ): 259 - 71 . 123. Liu J , Love PED , Smith J , Regan M , Sutrisna M . Public-private partnerships: a review of theory and practice of performance measurement . Int J Product Perform Manag . 2014 ; 63 ( 4 ): 499 - 512 . 124. Liu J , Love PED , Davis PR , Smith J , Regan M. Conceptual framework for the performance measurement of public-private partnerships . J Infrastruct Syst . 2014 ; 4014023 : 1 - 15 . 125. Goldman M , Wittelsberger A , De Magistris M-T. The innovative medicines initiative moves translational immunology forward . Eur J Immunol . 2013 ; 43 ( 2 ): 298 - 302 . 126. Carrillo MC , Sanders CA , Katz RG . Maximizing the Alzheimer's Disease Neuroimaging Initiative II . Alzheimers Dement . 2009 ; 5 ( 3 ): 271 - 5 . 127. Buse K , Tanaka S . Global public-private health partnerships: lessons learned from ten years of experience and evaluation . Int Dent J . 2011 ; 61 ( Suppl . 2): 2 - 10 . 128. Structural Genomics Consortium. What is the SGC? http:// www Accessed 29 Jan 2017 . 129. Donovan C , Hanney S. The Bpayback framework^ explained . Res Eval . 2011 ; 20 ( 3 ): 181 - 3 . 130. Kaplan W , Wirtz VJ , Mantel-Teeuwisse A , Stolk P , Duthey B , Laing R . Priority medicines for Europe and the world 2013 update . World Health Organization 2013 . medicines/areas/priority_medicines/MasterDocJune28_ FINAL_Web.pdf. Accessed 17 Mar 2017 . 131. Carrillo MC , Bain LJ , Frisoni GB , Weiner MW . Worldwide Alzheimer's disease neuroimaging initiative . Alzheimers Dement . 2012 ; 8 ( 4 ): 337 - 42 . 132. Weiner MW , Veitch DP , Aisen PS , Beckett LA , Cairns NJ , Cedarbaum J , et al. Impact of the Alzheimer's Disease Neuroimaging Initiative , 2004 to 2014. Alzheimers Dement. 2015 ; 11 ( 7 ): 865 - 84 . 133. Jones-Davis DM , Buckholtz N. The impact of the Alzheimer's Disease Neuroimaging Initiative 2: what role do public-private partnerships have in pushing the boundaries of clinical and basic science research on Alzheimer's disease? Alzheimers Dement . 2015 Jul; 11 ( 7 ): 860 - 4 . 134. Aisen PS. Q& a: the Alzheimer's disease neuroimaging initiative . BMC Med . 2011 ; 9 ( 1 ): 101 . 135. Snyder HM , Kim H , Bain LJ , Egge R , Carrillo MC . Alzheimer's disease public-private partnerships: update 2014 . Alzheimers Dement . 2014 ; 10 ( 6 ): 873 - 80 . 136. Salloway S . New lessons from the Alzheimer's Disease Neuroimaging Initiative . Arch Neurol . 2011 ; 68 ( 1 ): 19 - 21 . 137. Snyder HM , Bain LJ , Egge R , Carrillo MC . Alzheimer's disease public-private partnerships: a landscape of the global nonprofit community . Alzheimers Dement . 2013 ; 9 ( 4 ): 466 - 71 . 138. Mayeux R , Stern Y. Genetic epidemiology of Alzheimer disease . Alzheimer Dis Assoc Disord . 2006 ; 20 ( 3 Suppl 2 ): S58 - 62 . 139. Schneider LS , Mangialasche F , Andreasen N , Feldman H , Giacobini E , Jones R , et al. Clinical trials and late-stage drug development for Alzheimer's disease: an appraisal from 1984 to 2014 . J Intern Med . 2014 ; 275 ( 3 ): 251 - 83 . 140. ADNI. ADNI data usage statistics . -data-usage-stats/ . Accessed 2 Jan 2017 . 141. ADNI. ADNI publication overview . news-publications/publications/. Accessed 2 Jan 2017 . 142. Hendrix JA , Finger B , Weiner MW , Frisoni GB , Iwatsubo T , Rowe CC , et al. The worldwide Alzheimer's Disease Neuroimaging Initiative: an update . Alzheimers Dement . 2015 ; 11 ( 7 ): 850 - 9 . 143. Committee for Medicinal Products for Human Use (CHMP). Draft guideline on the clinical investigation of medicines for the treatment of Alzheimer's disease and other dementias. 2016; European Medicines Agency (EMA): EMA /CHMP/539931/ 2014 . 144. US Food and Drug Administration (FDA). Guidance for Industry Alzheimer's Disease: Developing Drugs for the Treatment of Early Stage Disease . Draft guidance; 2013 . 145. Committee for Medicinal Products for Human Use (CHMP), Q u a l i f i c a t i o n o p i n i o n o f A l z h e i m e r ' s d i s e a s e n o v e l methodologies/biomarkers for the use of CSF AB 1-42 and ttau and/or PET-amyloid imaging (positive/ negative) as biomarkers for enrichment, for use in regulatory clinical trials in mild and moderate Alzheimer's disease. 2012; European Medicines Agency (EMA): EMA /CHMP/SAWP/893622/ 2011 . 146. Kozauer N , Katz R . Regulatory innovation and drug development for early-stage Alzheimer's disease . N Engl J Med . 2013 ; 368 ( 13 ): 1169 - 71 . 147. Brady LS , William Z , Potter WZ . Public-private partnerships to revitalize psychiatric drug discovery . Expert Opin Drug Discovery . 2014 ; 9 ( 1 ): 1 - 8 . 148. Clark J , McGoey L . The black box warning on philanthrocapitalism . Lancet . 2016 ; 388 ( 10059 ): 2457 - 9 .

This is a preview of a remote PDF:

Remco L. A. de Vrueh, Daan J. A. Crommelin. Reflections on the Future of Pharmaceutical Public-Private Partnerships: From Input to Impact, Pharmaceutical Research, 2017, 1985-1999, DOI: 10.1007/s11095-017-2192-5