Home institution bias in the New England Journal of Medicine? A noninferiority study on citation rates

Scientometrics, Nov 2017

Recently, in the four top journals of humanities, an institutional bias towards publication of authors from Harvard and Yale was shown. The New England Journal of Medicine (NEJM) is today the highest ranked general medical journal. It is unknown if there exists institutional bias favoring publication of articles originating from Harvard University, since the NEJM is produced by the Massachusetts Medical Society with close connections to the Harvard University. We examined if studies originating from the Harvard University published in the NEJM were noninferior in terms of citation rates compared to articles with an origin outside Harvard University. We evaluated original research articles published in the NEJM in 2000 up until June 2001. A two-sample noninferiority test based on the primary endpoint of citations was performed. Twenty-two studies were affiliated to the Harvard University and 280 studies were not affiliated to the Harvard University. The mean number of citations for Harvard affiliated studies was 625 (95% CI 358–952, median 354) and for non–Harvard affiliated studies 493 (95% CI 421–569, median 303). The mean difference was not statistically different between affiliations, but fulfilled the requirements for noninferiority [132 (95% CI − 138–402, P = 0.343), Δ 200]. In summary, citation rates were comparable between studies origination from the Harvard University compared to non-Harvard Institutions. Based on these results there appears to be low risk of institutional bias in the publishing process of original studies in the NEJM.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://link.springer.com/content/pdf/10.1007%2Fs11192-017-2584-7.pdf

Home institution bias in the New England Journal of Medicine? A noninferiority study on citation rates

Home institution bias in the New England Journal of Medicine? A noninferiority study on citation rates Alberto Falk Delgado 0 1 Anna Falk Delgado 0 1 0 Department of Clinical Neuroscience, Karolinska Institute , Stockholm , Sweden 1 Department of Surgical Sciences , Plastic and Reconstructive Surgery , Uppsala University , Ing. 78/79 Plastikmottagningen, Akademiska Sjukhuset, 75185 Uppsala , Sweden Alberto Falk Delgado and Anna Falk Delgado have contributed equally. Citations rates; Institutional bias of articles Introduction Scientific journals are ranked according to their impact factor (IF), where the IF is based upon the number of citations to that journals recent published articles divided by the number of published articles the two preceding years. According to this, the New England Journal of Medicine (NEJM) is today the highest ranked general medical journal, with a steadily increasing impact factor (IF) over the last 20 years that has now reached 72 in IF (Garfield 1996; Reuters 2017) . This increasing IF might reflect that only the scientifically most rigorous and influential medical papers are published in the NEJM. Furthermore, 58% of the randomized controlled trials published in NEJM are entirely funded by a for-profit organization (Delgado and Delgado 2017). The academic publication process in the NEJM should optimally ensure publication of only the most impactful papers, and the base for publication decision should rely primarily on scientific merit and novelty. Recently, in the four top journals of humanities, an institutional bias towards publication of authors from Harvard and Yale was shown (Piper and Wellman 2017) . A finding of possible institutional inequality in the process of academic publishing. The NEJM implemented a policy Too close to call in 2001 to prevent ‘‘insider bias’’ among the editors (Curfman and Drazen 2001) , when editors submitted work to their own journal. However, it is unclear if there exists institutional bias favoring publication of articles originating from Harvard University, since the NEJM is produced by the Massachusetts Medical Society with close connections to the Harvard University. How can one determine that only the most influential medical articles are published in the NEJM? One method to assess the impact of a paper is simply by determining its citation rate, rather than quality assessment methods (Wang et al. 2014) . In this study, we examined if studies originating from the Harvard University published in the NEJM were noninferior in terms of citation rates compared to articles with an origin outside Harvard University. Methods We evaluated original research articles published in the NEJM during 2000 and 2001. Articles from year 2000 and 2001 were chosen in order to have a fair chance of being cited up until 2017. Brief reports were excluded, since brief reports in general are less cited compared to original articles (unpublished results). The articles were gathered consecutively and manually from the webpage www.nejm.org. Institutional affiliation based on first and last author were registered and categorized as Harvard or non-Harvard affiliation. The affiliation to Harvard of either the first or last author categorized the research as pertaining to Harvard. Citation rates, as of August 2017, for each article were retrieved from www.nejm.org. Data was extracted by one author. Study sample size was determined through power calculation based on pilot data from the first 3 months of 2000 in the Journal. A two-sample noninferiority test based on the primary endpoint of citation rates was performed using a power of 90% with a 2.5% error rate and a delta value of 200. Implementing a sampling ratio of 5 the numbers of articles needed in each group was 14 articles from the Harvard University and 280 non-Harvard based studies. We estimated that we needed to include articles from January 2000 up to June 30 2001. A two-sample noninferiority test based on the primary endpoint of citations was performed with a power of 90% and type I error rate of 2.5% with non-inferiority (D) set to 200 citations. Nonparametric Mann–Whitney’s U test was used to compare differences between continuous variables. Analyses were performed in STATA Version 13.1 (StataCorp, USA). Descriptive summary statistics were presented with 95% confidence interval. Results A total of 302 original articles published in the NEJM year 2000 up until June 2001 was identified. Twenty-two studies were affiliated to the Harvard University and 280 studies were not affiliated to the Harvard University. The mean number of citations for Harvard affiliated studies was 625 (95% CI 358–952, median 354, Fig. 1a, b) and for non–Harvard affiliated studies 493 (95% CI 421–569, median 303). The mean difference was not statistically different between affiliations, but fulfilled the requirements for noninferiority (132 (95% CI - 138–402, P = 0.343), D 200). Discussion Discuss the findings In total, eight percent of all articles published in the NEJM had a first or last author affiliated to the Harvard University. Based on these results there appears to be a low risk of institutional inequality in terms of impact of the articles published in NEJM publications from the Harvard University compared to publications to other institutions. Data from the humanities have suggested an institutional bias with a higher proportion of the authors affiliated to Yale and Harvard more likely to publish in the top ranked journals (Piper and Wellman 2017) . Previous data from the humanities does however not reveal if the studies originating from Yale or Harvard are inferior compared to other institutions. Our study reveal that eight percent of NEJM publications are affiliated to Harvard, this skewed proportion with origin from NEJMs home institution might suggest evidence for institutional equality. Previous studies related to citation rates have focused on factors related to higher citation, where e.g. for-profit funded research and articles related to cancer/cardiovascular disease are associated with higher citation rates (Kulkarni et al. 2007) , and length of the title (Jacques and Sebire 2010) . Currently, few reports have assessed the influence of home institution bias in the editorial process (Weller 2002) . Previous research has provided support for a national institutional publication bias, were authors or referees from the same country are favored for publication (Daniel 1993) . Studies from Harvard were non-inferior and not superior to studies not affiliated to Harvard in terms of citations. This suggest that only the most impactful articles actually are published in NEJM despite its home institution. Furthermore, this study ensure a rigid editorial process for the acceptance of articles in NEJM. Strengths and limitations The strength of this study is pertinent to the fact that the study was powered for the primary endpoint—noninferiority in citation rates. This is to our knowledge the first study to assess home institutional bias related to citation rates. Furthermore, we chose articles published from 2000 and 2001 to ensure a sufficient time to be cited, and less vulnerable to trends in citing. Few general medical journals are so strongly linked as NEJM to a particular institution. It is unclear if these finding can be generalized to other fields, or more specialized journals. Conclusion We conclude that eight percent of the studies in NEJM were associated with Harvard University. However, we found no evidence of home institution bias between studies affiliated to Harvard and those not affiliated to the Harvard University with regards to citations rates. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Curfman , G. D. , & Drazen , J. M. ( 2001 ). Too close to call . New England Journal of Medicine , 345 ( 11 ), 832 . Daniel , H. -D. ( 1993 ). Guardians of science: Fairness and reliability of peer review (p . xiv , 118 ). Cambridge: Weinheim: VCH. Delgado , A. F. , & Delgado , A. F. ( 2017 ). The association of funding source on effect size in randomized controlled trials: 2013-2015-A cross-sectional survey and meta-analysis . Trials , 18 ( 1 ), 125 . Garfield , E. ( 1996 ). How can impact factors be improved? BMJ , 313 ( 7054 ), 411 - 413 . Jacques , T. S. , & Sebire , N. J. ( 2010 ). The impact of article titles on citation hits: An analysis of general and specialist medical journals . JRSM Short Reports , 1 ( 1 ), 2 . Kulkarni , A. V. , Busse , J. W. , & Shams , I. ( 2007 ). Characteristics associated with citation rate of the medical literature . PLoS ONE , 2 ( 5 ), e403 . Piper , A. , & Wellman , C. ( 2017 ). Publication, power, and patronage: On inequality and academic publishing . Critical Inquiry. Reuters , T. ( 2017 ). Incites journal citation reports . https://jcr.incites.thomsonreuters.com/. Accessed Sept 20 . Wang , Y. X. , et al. ( 2014 ). Implications of Web of Science journal impact factor for scientific output evaluation in 16 institutions and investigators' opinion . Quantitative Imaging in Medicine and Surgery , 4 ( 6 ), 453 - 461 . Weller , A. C. ( 2002 ). Editorial peer review: Its strengths and weaknesses . Medford, NJ: Information Today, Inc.


This is a preview of a remote PDF: https://link.springer.com/content/pdf/10.1007%2Fs11192-017-2584-7.pdf

Alberto Falk Delgado, Anna Falk Delgado. Home institution bias in the New England Journal of Medicine? A noninferiority study on citation rates, Scientometrics, 2017, 1-5, DOI: 10.1007/s11192-017-2584-7