The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies

et al. (2014) The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies. PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474 The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies Stephen A. Gallo 0 Afton S. Carpenter 0 David Irwin 0 Caitlin D. McPartland 0 Joseph Travis 0 Sofie Reynders 0 Lisa A. Thompson 0 Scott R. Glisson 0 Lutz Bornmann, Max Planck Society, Germany 0 1 American Institute of Biological Sciences - Scientific Peer Advisory and Review Services Division, Reston, Virginia, United States of America, 2 Florida State University, Department of Biological Science , Tallahassee, Florida , United States of America There is a paucity of data in the literature concerning the validation of the grant application peer review process, which is used to help direct billions of dollars in research funds. Ultimately, this validation will hinge upon empirical data relating the output of funded projects to the predictions implicit in the overall scientific merit scores from the peer review of submitted applications. In an effort to address this need, the American Institute of Biological Sciences (AIBS) conducted a retrospective analysis of peer review data of 2,063 applications submitted to a particular research program and the bibliometric output of the resultant 227 funded projects over an 8-year period. Peer review scores associated with applications were found to be moderately correlated with the total time-adjusted citation output of funded projects, although a high degree of variability existed in the data. Analysis over time revealed that as average annual scores of all applications (both funded and unfunded) submitted to this program improved with time, the average annual citation output per application increased. Citation impact did not correlate with the amount of funds awarded per application or with the total annual programmatic budget. However, the number of funded applications per year was found to correlate well with total annual citation impact, suggesting that improving funding success rates by reducing the size of awards may be an efficient strategy to optimize the scientific impact of research program portfolios. This strategy must be weighed against the need for a balanced research portfolio and the inherent high costs of some areas of research. The relationship observed between peer review scores and bibliometric output lays the groundwork for establishing a model system for future prospective testing of the validity of peer review formats and procedures. - Data Availability: The authors confirm that all data underlying the findings are fully available without restriction. All relevant data are within the paper and its Supporting Information files. Funding: The authors have no funding or support to report. Competing Interests: The authors have declared that no competing interests exist. Some form of peer review is used at the majority of research granting organizations to determine the most meritorious applications to consider for funding. As such, peer review makes a significant contribution to how billions of dollars in research grants are awarded, influencing the very direction of science itself. However, this process has been increasingly questioned, particularly with regard to how well peer review results predict the ultimate impact of the funded research [15]. While several studies suggest that the process of peer review of scientific manuscripts has some success in identifying what will later become highly cited, high-impact publications, only a handful of publications have dealt with the predictive accuracy of the outcomes of peer review of grant applications [69]. Of these, a few have reported results supporting the validity of the peer review outcome [1013]. However, in terms of direct comparison between peer review scores (or percentile ranking) and bibliometric data, several publications from program directors at the NIGMS and NHLBI have indicated either a modest (but statistically significant) correlation or no correlation between publication impact and peer review scores, with both data sets displaying a substantial amount of variation in impact among grants with similar peer review scores [14,15]. Thus, the few studies in the literature that do exist provide inconsistent results at best and contradictory results at worst. In addition, the sources of the large degree of variability in the data from these studies remain unexplored, as has the dynamic relationship of publication impact and peer review output of a funding program over time. Understanding the factors that influence the inputs and outputs of funded research programs is crucial for two reasons. First, the results of such analyses can be used to develop a working model of the peer review process with which to validate evaluation procedures. Second, the results could inform funding agencies on how to optimize their funding strategies to promote the maximal scientific impact of their programs. The American Institute of Biological Sciences (AIBS) has conducted a retrospective analysis of peer review and project output data over an 8-year period for a discrete funding research program and examined whether correlations exist among the assessment of scientific merit using a peer review system and the scientific output from this program. AIBS conducts scientific peer review for federal and non-federal clients and in doing so has accumulated data that speak to the predictive ability of the peer review process. For one such program, referred to as PrX in this manuscript, AIBS has collected peer review scoring data and post-funding citation output data from applications reviewed between 1999 and 2006. PrX is an extramural program designed to support a wide variety of research topic areas, including vision, drug abuse, nutrition, blood-related cancer, kidney disease, autoimmune diseases, malaria, tuberculosis, osteoporosis, arthritis, and autism research, among others. Topic areas were not static, changing from year to year in both type and number (1431 distinct areas per year), and very few were continuous throughout the 19992006 period of study. However, after an initial rise, the total number of topic areas did stabilize after 2001 to an average level of 27. In every program year there was a significant proportion of both applied and basic research applications, with many applications encompassing varying degrees of both basic and applied research in their specific aims. However, topic area descriptions were general and brief, with research scopes largely open to interpretation by the applicants (e.g., one such topic area was Drug Abuse; no further definition was provided). In general, the research submitted was overwhelmingly biomedical in nature over the full review period (19992006). The program began in 1999 with a funding level of $5.5 M, which increased to $40. (...truncated)