How do NIHR peer review panels use bibliometric information to support their decisions?
Scientometrics
DOI 10.1007/s11192-017-2417-8
How do NIHR peer review panels use bibliometric
information to support their decisions?
Salil Gunashekar1 • Steven Wooding2 • Susan Guthrie1
Received: 25 April 2017
The Author(s) 2017. This article is an open access publication
Abstract Bibliometrics is widely used as an evaluation tool to assist prospective R&D
decision-making. In the UK, for example, the National Institute for Health Research
(NIHR) has employed bibliometric analysis alongside wider information in several
awarding panels for major funding schemes. In this paper, we examine various aspects of
the use of bibliometric information by members of these award selection panels, based on
interviews with ten panel members from three NIHR panels, alongside analysis of the
information provided to those panels. The aim of the work is to determine what influence
bibliometrics has on their decision-making, to see which types of bibliometric measures
they find more and less useful, and to identify the challenges they have when using these
data. We find that panel members broadly support the use of bibliometrics in panel
decision-making, and that the data are primarily used in the initial individual assessment of
candidates, playing a smaller role in the selection panel meeting. Panel members felt that
the most useful measures of performance are normalised citation scores and the number or
proportion of papers in the most highly cited X% (e.g. 5, 10%) for the field. Panel members
expressed concerns around the comparability of bibliometrics between fields, but the
discussion suggested this largely represents a lack of understanding of bibliometric techniques, confirming that effective background information is important. Based on the evidence around panel behaviour and concerns, we set out guidance around providing
bibliometrics to research funding panels.
Keywords Bibliometrics Peer review Review panels Grant funding
& Salil Gunashekar
1
RAND Europe, Cambridge CB4 1YG, UK
2
Centre for Science and Policy, University of Cambridge, Cambridge CB2 1QA, UK
123
Scientometrics
Introduction
Bibliometrics is increasingly used in the assessment of research, both for impact evaluation
and for awarding research funding. In the UK, the National Institute for Health Research
(NIHR) has employed bibliometric analysis as part of a wider set of information in several
awarding panels including the NIHR Senior Investigators, Collaborations for Leadership in
Applied Health Research and Care, and Biomedical Research Centres/Units competitions.
We discuss the specific details about the use of bibliometrics within these three competitions in the next section. Furthermore, the Research Excellence Framework (REF)
exercise in the UK drew on bibliometrics in its assessment of the quality of research
produced by UK higher education institutions across all disciplines (REF 2014a). For the
REF, universities were asked to submit up to four research outputs (e.g. journal articles,
monographs, book chapters, etc.) for each member of staff included in their submissions.
These outputs were peer-reviewed by expert sub-panels1 in order to assess the quality of
the outputs in terms of ‘‘originality, significance and rigour.’’ Some of the sub-panels also
used citation information provided by the REF team, which was not intended to be used as
a primary tool of assessment but rather as a ‘‘positive indicator of the academic significance’’ to inform the decisions arrived at by the REF sub-panels (REF 2012). The subpanels that used bibliometric data were primarily those affiliated to the health, physical and
life sciences which traditionally have good coverage within bibliometric databases and
journal-based outputs.2
In Australia, a similar national research assessment exercise is conducted. Excellence in
Research for Australia (ERA), the first three rounds of which took place in 2010, 2012 and
2015 aims to identify and evaluate the quality of research at Australian higher education
institutes (ERA 2015). Results of citation analyses have been explicitly used as indicators
of research quality in the exercise in addition to the peer review of a sample of research
outputs, with citation information predominantly used in the science and medical disciplines. Unlike the REF, funding is not directly allocated based on the outcomes of the ERA
process.
Given the influence of bibliometrics, it is important to understand how panels use the
information provided and how it can best support their decision making. Previous articles
have explored the reliability of bibliometrics as an alternative to peer review (Nederhof and
van Raan 1993; Aksnes and Taxt 2004). A study commissioned by the Higher Education
Funding Council for England (HEFCE) reviewed the role of metrics in research assessment
(HEFCE 2015a), finding that there is still some cynicism among the research community
around the wider use of metrics to evaluate research (Wilsdon et al. 2015). The report
concluded that metrics should not replace peer review but that in some cases, peer-review
based decision-making could be complemented with the use of ‘‘carefully selected’’
quantitative indicators (HEFCE 2015b). In this article we explore the use of bibliometrics
as a supplementary source of information to inform peer review. The role that bibliometrics plays in decision-making in this context, and the suitability of the metrics provided,
are not well understood. Typically, bibliometrics are only one part of the evidence
1
Specifically, 36 subject-based units of assessment (UOAs) working under 4 main panels.
2
The following REF sub-panels made use of citation data: Clinical Medicine; Public Health, Health
Services and Primary Care; Allied Health Professions, Dentistry, Nursing and Pharmacy; Psychology,
Psychiatry and Neuroscience; Biological Sciences; Agriculture, Veterinary, and Food Science; Earth Systems and Environmental Sciences; Chemistry; Physics; Computer Science and Informatics; and Economics
and Econometrics.
123
Scientometrics
provided to support decision-making by panels, and as such it is difficult to test the impact
they make on the conclusions the panels reach.
The literature that examines the use of bibliometric data by selection panels is sparse.
One notable exception is a study by Lewison et al. (1999) that looks at the bibliometrics
used to inform a panel that selected neuroscience grants. The study examined the results
from three surveys, two of selection panel members and one of applicants. The aim was to
establish panel members’ and applicants’ knowledge of bibliometrics and to determine
which indicators they found to be most useful. The authors found that more than two-thirds
of the respondents were in favour of using bibliometrics to inform the decision-making
process. With regard to specific bibliometric indicators, the respondents felt that citationbased scores and journal-impact category rankings were the most helpful (...truncated)
