Counting Highly Cited Papers for University Research Assessment: Conceptual and Technical Issues

Citation: Rodrguez-Navarro A ( Counting Highly Cited Papers for University Research Assessment: Conceptual and Technical Issues Alonso Rodrguez-Navarro 0 Eshel Ben-Jacob, Tel Aviv University, Israel 0 Centro de Biotecnolog a y Geno mica de Plantas, Universidad Polite cnica de Madrid , Madrid , Spain A Kuhnian approach to research assessment requires us to consider that the important scientific breakthroughs that drive scientific progress are infrequent and that the progress of science does not depend on normal research. Consequently, indicators of research performance based on the total number of papers do not accurately measure scientific progress. Similarly, those universities with the best reputations in terms of scientific progress differ widely from other universities in terms of the scale of investments made in research and in the higher concentrations of outstanding scientists present, but less so in terms of the total number of papers or citations. This study argues that indicators for the 1% high-citation tail of the citation distribution reveal the contribution of universities to the progress of science and provide quantifiable justification for the large investments in research made by elite research universities. In this tail, which follows a power low, the number of the less frequent and highly cited important breakthroughs can be predicted from the frequencies of papers in the upper part of the tail. This study quantifies the false impression of excellence produced by multinational papers, and by other types of papers that do not contribute to the progress of science. Many of these papers are concentrated in and dominate lists of highly cited papers, especially in lower-ranked universities. The h-index obscures the differences between higher- and lower-ranked universities because the proportion of h-core papers in the 1% high-citation tail is not proportional to the value of the h-index. - Government policy-makers, corporate research managers, and university administrators need valid and reliable S&T indicators for a variety of purposes: for example, to measure the effectiveness of research expenditures, identify areas of strength and excellence, set priorities for strategic planning, monitor performance relative to peers and competitors, and target emerging specialties and new technologies for accelerated development. So begins a paper by Garfield and Welljams-Dorof [1], and the essence of this idea can be found in the introductions to countless papers published before and since. Consistent with this idea, many research indicators have been developed (see, for example [24]). It is unclear, however, whether the indicators currently used accurately measure all that governments and research administrators need to know, or whether such indicators are always correctly interpreted and applied by governments and research administrators [57]. The Spanish government, for example, recently announced that the quality of scientific research in Spain had overtaken that of Switzerland [8], but this statement is inconsistent with the role research plays in the economic realities of the respective countries. While the responsibility for any misstatement would lay exclusively with its author, if it is based on a research indicatorin this case, citation counts of all published papersthe validity of that indicator as a numerical measure of research performance should be brought into question. The use of indicators of research performance based on measures such as the numbers of all papers published and their subsequent citations contributes to not only misleading conclusions about a countrys research performances but also to the notion that elite research institutions are not using their research investment funds in an efficient manner. Considering the highest- and lowest-ranked universities in [9], MITs research revenues exceed 1.3 billion US dollars per year (http://web.mit.edu/facts/financial.html, accessed on August 2011), whereas the equivalent figure for Complutense University of Madrid is 21 million euros (approximately, 27 million US dollars; http://www.redotriuniversidades.net/, accessed on August 2011). The comparison of these figures leads to the conclusion that the difference in research performance between these two universities is not best indicated by the ratio of the number of publications or of the other indicators based on the total number of publications, which may have values of 23 (for example: Academic Ranking of World Universities 2010, PUB score, http://www. arwu.org/ARWU2010.jsp, accessed on August 2011; Excellence Rate Report, http://www.scimagoir.com, downloaded November, 2011). It is, of course, difficult to make comparisons regarding research funding and output [10], and the differences in the accounting methods across institutions must be corrected before making comparisons. However, even having compensated for those differences, the investment ratio between MIT and Complutense University is still greater than 10:1. In addition, the high number of researchers having received Nobel prizes or other awards [11] and with a high number of citations [12] at MIT suggests that the institution tends to hire high-level researchers in contrast with the suboptimal methods for researcher selection to which Spanish universities adhere [13]. All this suggests that differences in research performance between MIT and Complutense University should be even greater than those suggested by the differences in funding and ought not to be expressed as simply a ratio of 3.0. Another way to view this issue is calculating the ratio of research investment per paper for the two institutions. This calculation reveals that the MITs cost of one paper is seven times higher than in Complutense University, which suggests that those papers coming out of MIT are not comparable to those coming out of Complutense University. These examples illustrate that some research indicators may be problematic at both country and university levels. Although the problems can be investigated at both levels, the university level is easier to investigate because universities are research units more homogeneous than countries, where very different institutions coexist. Moreover, it is easy to select a sample of universities that are very different in research activity and similar in size to simplify the analysis of the results. To evaluate overall research performance, the x-index was recently formulated using a statistical procedure which optimized the correlation of the indicator with the number of Nobel Prize achievements [9]. The x-index only considers the papers that are included in the worlds top 1% of cited papers [9] and not all these papers. In particular, multinational and review papers are not counted, and a subtraction term is included in the formula to statistically discount the papers that report methods, clinical trials, and statistics (MCTS papers). Although the statistic (...truncated)