Scholars as government-appointed research evaluators: Do they create congruence between their professional quality standards and political demands?

PLOS ONE RESEARCH ARTICLE Scholars as government-appointed research evaluators: Do they create congruence between their professional quality standards and political demands? Hendrik Woiwode ID* President’s Research Group, Berlin Social Science Center, Berlin, Germany a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Woiwode H (2020) Scholars as government-appointed research evaluators: Do they create congruence between their professional quality standards and political demands? PLoS ONE 15(10): e0239336. https://doi.org/10.1371/ journal.pone.0239336 Editor: Cindy Sing Bik Ngai, Hong Kong Polytechnic University, HONG KONG * Abstract All across the globe politically initiated research evaluations are based on “informed peer review” procedures. Scholars are appointed as evaluators and can apply self-defined quality standards in order to overcome shortcomings of standardized measures. Even though there are no binding criteria in these procedures and the quality standards of the scholars’ disciplines vary, studies suggest that scholars, in their role as government-appointed research evaluators, assess research uniformly.By drawing on a small-N investigation, this study compares the quality standards scholars apply as government-appointed research evaluators with quality standards they follow as researchers. The study points to a paradox: Criteria scholars refer to while describing the excellence of their own research and criteria they use as evaluators differ and contradict each other. The results are discussed from different angles. Received: May 13, 2020 Accepted: September 4, 2020 Published: October 14, 2020 Peer Review History: PLOS recognizes the benefits of transparency in the peer review process; therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. The editorial history of this article is available here: https://doi.org/10.1371/journal.pone.0239336 Copyright: © 2020 Hendrik Woiwode. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: All relevant data are within the manuscript and its Supporting Information files. Introduction The mutual assessment of research-quality by qualified scientific peers—the so-called peer review—is the most widely accepted instrument for assessing research. It fulfills an elementary selection function within the scientific community as the review of manuscripts in the publication system, the awarding of academic prizes, or the review of research funding applications are based on peer review [1, 2]. However, peer review is not exclusively used for self-governing purposes within the scientific community. As the demand for control, accountability and evaluation of research performance has increased significantly in recent decades [3–5], peer review becomes increasingly used as a formalized management tool in politically initiated evaluations–mostly in combination with bibliometric information in so-called “informed peer review” procedures [6, 7]. Political initiated evaluations based on informed peer review–among them the Research Excellence Framework (REF) in the UK, or the Excellence in Research for Australia initiative (ERA)–were introduced with the aim to ensure an efficient allocation of scarce public funds and to stimulate better research performance. They influence structural and financial PLOS ONE | https://doi.org/10.1371/journal.pone.0239336 October 14, 2020 1 / 14 PLOS ONE Funding: The publication of this article was funded by the Open Access Fund of the Leibniz Association. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist. Scholars as government-appointed research evaluators conditions of academic knowledge production and are thus seen as a regulative pressure on professional self-regulation because they enforce politically defined quality standards of scientific work [8–10]. However,—due to the integration of a qualitative peer review mechanism -, active researchers review the quality of research during these evaluations. Taking over a role as evaluator enables scholars to enforce an understanding of quality that does justice to the special characteristics of their core activities that bibliometric data cannot reflect. Judgments about the (future) quality of research outputs are thus not made only based on citation information [7]. Definitions of high-quality research vary between (sub-)disciplines due to different epistemological orientations [11–13]. Standardized research evaluations are criticized because they focus only on specific output measures and thus cannot reflect this heterogeneity. Qualitative peer review procedures are therefore included in government-initiated evaluations. They should reflect the heterogeneity of scientific outputs and overcome the deficits of standardized evaluation procedures [7]. However, research suggests that in their role as governmentappointed evaluators scholars from different disciplines emphasize the characteristics of highquality research in nearly the same way–even though they are able to conduct the quality assessments themselves and without mandatory criteria [14]. This allows the assumption that– despite the integration of a qualitative peer review mechanism–the evaluation procedures do not reflect epistemic heterogeneity adequately and raises the question regarding the overall functionality of qualitative peer reviews in political initiated research evaluations. To the best of the author’s knowledge no study so far has systematically compared the quality standards scholars apply during political initiated qualitative research evaluations in socalled “informed peer review” procedures with quality standards they apply in their role as researchers. The majority of studies concerned with peer review focus on the assessment procedures in the publication sector and use quantitative methods to examine the procedures reliability, validity and fairness [15]. These studies give rise to criticism of peer review by indicating, for example, a low reliability [16, 17], a low prognostic quality [18] or opportunistic behavior of reviewers and authors [19]. The number of studies concerned with qualitative peer review in political initiated evaluations, however, is relatively small [20]. The present study thus asks: Do scholars as government-appointed research evaluators create congruence between their professional quality standards and political demands? The question is answered by focusing on the work of a federal state government agency in Germany who evaluated research performances of all subjects at univers (...truncated)