Demystifying the Effectiveness of the Impact Factor and Proposing Alternative Research Assessment Means

International Journal of Aquatic Research and Education Volume 6 | Number 1 2-1-2012 Demystifying the Effectiveness of the Impact Factor and Proposing Alternative Research Assessment Means Stathis Avramidis Hellenic Center for Disease Control and Prevention, Greece, Follow this and additional works at: https://scholarworks.bgsu.edu/ijare Recommended Citation Avramidis, Stathis (2012) "Demystifying the Effectiveness of the Impact Factor and Proposing Alternative Research Assessment Means," International Journal of Aquatic Research and Education: Vol. 6 : No. 1 , Article 3. DOI: 10.25035/ijare.06.01.03 Available at: https://scholarworks.bgsu.edu/ijare/vol6/iss1/3 This Letter to the Editors is brought to you for free and open access by ScholarWorks@BGSU. It has been accepted for inclusion in International Journal of Aquatic Research and Education by an authorized editor of ScholarWorks@BGSU. Article 3 Avramidis: Demystifying the Effectiveness of the Impact Factor and Proposing Letter to the Editor International Journal of Aquatic Research and Education, 2012, 6, 8-10 © 2012 Human Kinetics, Inc. Demystifying the Effectiveness of the Impact Factor and Proposing Alternative Research Assessment Means The increasing growth of scientific literature the last decades in all fields of academia, including aquatics, has led to the inevitable discussion of how scholarly work should be reviewed and evaluated and where it should be submitted. For example, experienced water safety authorities are concerned about the inclusion of “non-research” manuscripts labeled as “research” or low-quality contributions as a solution for continuing to operate the journal. Instead, they correctly suggest a more rigid evaluation of what is submitted as research, using peer reviewers familiar with current scientific standards, ideally people who have themselves been published in recognized scientific journals. The remainder of submissions that are accepted for publication can be classified as opinion, or similar, but conflicts should nevertheless be very clearly stated. While this may reduce the amount of content labeled as research in the International Journal of Aquatic Research and Education (IJARE), it will serve to increase its integrity, which will inure to its benefit when the next applications are submitted to PubMed and Medline. (Brewster, 2011, p. 376) Journal editors may face the problem of how submitted scholarly work should be evaluated to guarantee high standards and increase the likelihood to be included in databases that accommodate high rated research publications (see Langendorfer, 2011). Similarly, prospective authors face this dilemma: “Should I choose to submit my manuscript to IJARE, which is dedicated to peer-reviewed aquatic scholarly work but does not have an impact factor (IF) or to another journal that may be less relevant to noncompetitive aquatics and water safety but has an IF?” Given the above concerns from experts, editors, and scholars, I suspect that many of us anticipate the moment when IJARE will eventually have its own IF as evidence of quality. So far, IJARE has earned considerable respect and reputation within the aquatic and water safety community internationally without having an IF, because it acts as a forum and a force for change in aquatics (see Langendorfer, 2007) using a developmental approach for each manuscript submission (Langendorfer, 2007, 2010). Because IJARE is already 5 years old (2007-2011), it is only a matter of time before it gets its own IF. Whether or not this happens, it is essential to be aware of two crucial issues. First, we need to know the limitations of this rating method. Second, we need to base our decision on which journal we will choose as the destination for our manuscript submissions as well as how to value one’s work as reviewers or faculty committee members with alternatively effective methods. The IF is perhaps the most widely known means for rating research, but its effectiveness has been questioned as “notoriously invalid and unreliable” (Langen- Published8by ScholarWorks@BGSU, 2012 1 International Journal of Aquatic Research and Education, Vol. 6, No. 1 [2012], Art. 3 Letter to the Editor   9 dorfer, 2011, p. 372) when assessing the value of a research portfolio of scholars because it can be manipulated in several ways. For example, in 2007, a journal raised its IF from 0.66 to 1.44, because it published an editorial that cited all its articles from 2005 to 2006 protesting against the absurd use of the IF (see Agrawal, 2005; The PLoS Medicine Editors, 2006). In 2008, another paper included a sentence that instructed readers to cite it and received over 6,600 citations raising the journal’s IF from 2.051 to 49.926, more than Nature (31.434) and Science (28.103; see Wikipedia, 2011; Schuttea & Svec, 2007). Similarly, the IF has been manipulated by publishing review papers because they are cited more than pure research studies (Seglen, 1997) and by changing the fraction of citable items compared to frontmatter in the denominator of the IF equation (Arnold & Fowler, 2011). As shown above, the use of IF is not the answer and therefore alternative solutions need to be suggested. In relation to those acting as reviewers and faculty members that are responsible for assessing journal publications, such alternative methods may be the use of the h-index (Rieder, Bruse, Michalski, Kleeff, & Friess, 2010), a developmental rubric (Langendorfer, 2011), or the direct and qualitative evaluation of the articles using the Boyer/Carnegie six common evaluation criteria in the form of a checklist (Glassick, Huber, & Maeroff, 1996). Each of these alternative methods of rating papers would be preferred rather than rating the journals in which the work is published. Finally, in relation to those wishing to submit their aquatic-oriented scholarly work to a journal, I personally believe that their decision should be based on whether this journal is relevant to their field, is orchestrated by an international editorial board, and is peer reviewed using a developmental approach. If the manuscript is accepted, it will be read by the most appropriate readership. If it is rejected, it will benefit by the constructive review comments and will have the opportunity to be revised and submitted elsewhere in an improved form. Given that IJARE is governed by these principles, I encourage aquatics and water safety scholars to continue choosing this journal as a destination of their work. Through IJARE, their work will have a higher impact to the society regardless of the journal’s impact factor. Stathis Avramidis, PhD Associate, Hellenic Centre for Disease Control and Prevention (Greece) Visiting Research Fellow, Leeds Metropolitan University (UK) Task Force Member, The Lifesaving Foundation (Ireland) References Agrawal, A. (2005). Corruption of journal impact factors. Trends in Ecology and Evolution 20(4), 157. Arnold, D.N. & Fowler, K.K. (2011). (...truncated)