The Evaluation Scale: Exploring Decisions About Societal Impact in Peer Review Panels
Minerva (2016) 54:75–97
DOI 10.1007/s11024-016-9290-0
The Evaluation Scale: Exploring Decisions About
Societal Impact in Peer Review Panels
Gemma E. Derrick1 • Gabrielle N. Samuel1
Published online: 9 February 2016
The Author(s) 2016. This article is published with open access at Springerlink.com
Abstract Realising the societal gains from publicly funded health and medical
research requires a model for a reflexive evaluation precedent for the societal impact
of research. This research explores UK Research Excellence Framework evaluators’
values and opinions and assessing societal impact, prior to the assessment taking
place. Specifically, we discuss the characteristics of two different impact assessment
extremes – the ‘‘quality-focused’’ evaluation and ‘‘societal impact-focused’’ evaluation. We show the wide range of evaluator views about impact, and that these
views could be conceptually reflected in a range of different positions along a
conceptual evaluation scale. We describe the characteristics of these extremes in
detail, and discuss the different beliefs evaluators had which could influence where
they positioned themselves along the scale. These decisions, we argue, when considered together, form a dominant definition of societal impact that influences the
direction of its evaluation by the panel.
Keywords Societal impact Peer review Evaluation frameworks
Impact Qualitative
Introduction
The 2006 Cooksey review of publicly funded health research stated that the UK was
at risk of ‘‘failing to reap the full economic, health and social benefits of public
investment in health research’’ due to health research not being adequately
translated into health outcomes (Cooksey 2006). Similar calls have been made in
other countries where the economic benefit of investing in health and medical
& Gemma E. Derrick
1
Educational Research, Lancaster University, Lancaster LA1 4YL, UK
123
76
G. E. Derrick, G. N. Samuel
research have been realised by government policymakers and academics alike
(Donovan 2008). One such strategy to increase the societal returns from publicly
funded research is to include a formal assessment of research returns in peer review
evaluation processes and link the outcomes to the allocation of funding. As such,
there are currently moves by public funding bodies to evaluate research in terms of
both scientific and societal impact (Smith 2001) as part of research’s social contract
with society (Gibbons et al. 1994; Nowotny et al. 2001; Wolfendale 1993).
However, without a strong precedent for formal, reflexive (Dahler-Larsen 2012), expost evaluation of societal impact, questions remain about how evaluators would
navigate the peer review process of these outcomes in terms of resolving their
values about what constitutes excellence in the societal returns from research.
Whereas a number of conceptual models and theories have been proposed to
understand the process of impact realisation and, in turn, used to guide its
evaluation, actual empirical investigations of the assessment of societal impact,
where the results are linked to funding outcomes, are rare (Bornmann 2012, 2013;
Holbrook and Hrotic 2013). Problems associated with access to peer-review panel
deliberations, and a lack of formal frameworks incorporating criteria of the societal
impact of research with which to investigate, have made conducting this type of
empirical research difficult. However, the UK’s 2014 Research Excellence
Framework (REF2014) for the first time included a formal assessment of societal
impact. Therefore, by using this framework, this article provides one of the first
empirical investigations of how evaluators expected to weigh the main concerns
expressed in the literature about the evaluation of the societal impact of research,
when faced with the task of evaluating this criterion formally.
In particular, this study describes five separate decisions expressed by evaluators
prior to the evaluation process taking place that must be resolved within peer review
group discussions about formally assessing the societal impact of research. This
article links hypothesised issues about impact evaluation, which have already been
discussed widely in the literature (Bornmann 2013), with broader concerns about
how evaluator definitions (Huutoniemi 2012; Langfeldt 2006), biases (Langfeldt
2004) and behavioural tendencies (Langfeldt 2001) contribute to group-based peer
review processes. In the absence of a firm, reflexive precedent for evaluation,
questions are raised regarding the reliability of review outcomes and the interplay of
evaluator viewpoints. It is important to explore these questions prior to the
assessment process so as to gain an insight into the baseline values evaluators hold
regarding societal impact evaluation, and so to understand the nature of these
tensions independent to the development of an evaluation culture that evaluators
quickly acquire during evaluation panel discussions (Olbrecht et al. 2007; Langfeldt
2001). This is also important as it is this committee culture that ultimately influences
the review outcomes (Kerr et al. 1996; Langfeldt 2004), and the future evaluation
behaviours of peer reviewers in similar situations that will require them to use their
experience of evaluating the societal impact of research from this situation. In
addition, by describing the range of tensions that exist prior to the formal evaluation
process taking place, this research provides an insight into what tensions will
dominate the group discussions within the review process, and hypothesise about
how these may be resolved by the panel.
123
Exploring Decisions About Societal Impact in Peer Review Panels
77
Although the results presented here may provide a guide to interpret the
REF2014 impact evaluation results, this is not its primary goal. Instead, the results
discussed in this article aim to use the process of impact assessment as a way of
understanding the range of tensions faced by evaluators regarding the assessment of
societal impact, in the absence of prior experience or methods of benchmarking this
measure.
In the next section, we discuss the relevant literature regarding the process of
peer review and, in particular, the concerns regarding the evaluation of societal
impact, as opposed to the more traditional, scientific impact of publicly funded
research. In the absence of prior studies (Bornmann 2012, 2013; Holbrook and
Hrotic 2013), we discuss research of panel-based peer review processes that will be
used to interpret our results. The methods section will describe the approach
employed for this study. In particular, it will describe the REF2014’s formal, expost ‘‘impact’’ criterion. The results section will discuss the analysis of 62 semistructured interviews with REF2014 evaluators for health, medical and biomedical
research submissions, and prior to the formal evaluation process taking place. This
section will also introduce the conceptual mo (...truncated)
