Home institution bias in the New England Journal of Medicine? A noninferiority study on citation rates
Home institution bias in the New England Journal
of Medicine? A noninferiority study on citation rates
Alberto Falk Delgado 0 1
Anna Falk Delgado 0 1
0 Department of Clinical Neuroscience, Karolinska Institute , Stockholm , Sweden
1 Department of Surgical Sciences , Plastic and Reconstructive Surgery , Uppsala University , Ing. 78/79 Plastikmottagningen, Akademiska Sjukhuset, 75185 Uppsala , Sweden
Alberto Falk Delgado and Anna Falk Delgado have contributed equally.
Citations rates; Institutional bias of articles
Scientific journals are ranked according to their impact factor (IF), where the IF is based
upon the number of citations to that journals recent published articles divided by the
number of published articles the two preceding years. According to this, the New England
Journal of Medicine (NEJM) is today the highest ranked general medical journal, with a
steadily increasing impact factor (IF) over the last 20 years that has now reached 72 in IF
(Garfield 1996; Reuters 2017)
. This increasing IF might reflect that only the scientifically
most rigorous and influential medical papers are published in the NEJM. Furthermore, 58%
of the randomized controlled trials published in NEJM are entirely funded by a for-profit
organization (Delgado and Delgado 2017).
The academic publication process in the NEJM should optimally ensure publication of
only the most impactful papers, and the base for publication decision should rely primarily
on scientific merit and novelty. Recently, in the four top journals of humanities, an
institutional bias towards publication of authors from Harvard and Yale was shown
and Wellman 2017)
. A finding of possible institutional inequality in the process of
academic publishing. The NEJM implemented a policy Too close to call in 2001 to prevent
‘‘insider bias’’ among the editors
(Curfman and Drazen 2001)
, when editors submitted
work to their own journal. However, it is unclear if there exists institutional bias favoring
publication of articles originating from Harvard University, since the NEJM is produced by
the Massachusetts Medical Society with close connections to the Harvard University. How
can one determine that only the most influential medical articles are published in the
NEJM? One method to assess the impact of a paper is simply by determining its citation
rate, rather than quality assessment methods
(Wang et al. 2014)
. In this study, we examined
if studies originating from the Harvard University published in the NEJM were noninferior
in terms of citation rates compared to articles with an origin outside Harvard University.
We evaluated original research articles published in the NEJM during 2000 and 2001.
Articles from year 2000 and 2001 were chosen in order to have a fair chance of being cited
up until 2017. Brief reports were excluded, since brief reports in general are less cited
compared to original articles (unpublished results). The articles were gathered
consecutively and manually from the webpage www.nejm.org. Institutional affiliation based on
first and last author were registered and categorized as Harvard or non-Harvard affiliation.
The affiliation to Harvard of either the first or last author categorized the research as
pertaining to Harvard. Citation rates, as of August 2017, for each article were retrieved
from www.nejm.org. Data was extracted by one author. Study sample size was determined
through power calculation based on pilot data from the first 3 months of 2000 in the
Journal. A two-sample noninferiority test based on the primary endpoint of citation rates
was performed using a power of 90% with a 2.5% error rate and a delta value of 200.
Implementing a sampling ratio of 5 the numbers of articles needed in each group was 14
articles from the Harvard University and 280 non-Harvard based studies. We estimated that
we needed to include articles from January 2000 up to June 30 2001. A two-sample
noninferiority test based on the primary endpoint of citations was performed with a power
of 90% and type I error rate of 2.5% with non-inferiority (D) set to 200 citations.
Nonparametric Mann–Whitney’s U test was used to compare differences between continuous
variables. Analyses were performed in STATA Version 13.1 (StataCorp, USA).
Descriptive summary statistics were presented with 95% confidence interval.
A total of 302 original articles published in the NEJM year 2000 up until June 2001 was
identified. Twenty-two studies were affiliated to the Harvard University and 280 studies
were not affiliated to the Harvard University. The mean number of citations for Harvard
affiliated studies was 625 (95% CI 358–952, median 354, Fig. 1a, b) and for non–Harvard
affiliated studies 493 (95% CI 421–569, median 303). The mean difference was not
statistically different between affiliations, but fulfilled the requirements for noninferiority
(132 (95% CI - 138–402, P = 0.343), D 200).
Discuss the findings
In total, eight percent of all articles published in the NEJM had a first or last author
affiliated to the Harvard University. Based on these results there appears to be a low risk of
institutional inequality in terms of impact of the articles published in NEJM publications
from the Harvard University compared to publications to other institutions.
Data from the humanities have suggested an institutional bias with a higher proportion of
the authors affiliated to Yale and Harvard more likely to publish in the top ranked journals
(Piper and Wellman 2017)
. Previous data from the humanities does however not reveal if
the studies originating from Yale or Harvard are inferior compared to other institutions.
Our study reveal that eight percent of NEJM publications are affiliated to Harvard, this
skewed proportion with origin from NEJMs home institution might suggest evidence for
institutional equality. Previous studies related to citation rates have focused on factors
related to higher citation, where e.g. for-profit funded research and articles related to
cancer/cardiovascular disease are associated with higher citation rates
(Kulkarni et al.
, and length of the title
(Jacques and Sebire 2010)
. Currently, few reports have
assessed the influence of home institution bias in the editorial process
Previous research has provided support for a national institutional publication bias, were
authors or referees from the same country are favored for publication
Studies from Harvard were non-inferior and not superior to studies not affiliated to Harvard
in terms of citations. This suggest that only the most impactful articles actually are
published in NEJM despite its home institution. Furthermore, this study ensure a rigid editorial
process for the acceptance of articles in NEJM.
Strengths and limitations
The strength of this study is pertinent to the fact that the study was powered for the primary
endpoint—noninferiority in citation rates. This is to our knowledge the first study to assess
home institutional bias related to citation rates. Furthermore, we chose articles published
from 2000 and 2001 to ensure a sufficient time to be cited, and less vulnerable to trends in
citing. Few general medical journals are so strongly linked as NEJM to a particular
institution. It is unclear if these finding can be generalized to other fields, or more
We conclude that eight percent of the studies in NEJM were associated with Harvard
University. However, we found no evidence of home institution bias between studies
affiliated to Harvard and those not affiliated to the Harvard University with regards to
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution,
and reproduction in any medium, provided you give appropriate credit to the original author(s) and the
source, provide a link to the Creative Commons license, and indicate if changes were made.
Curfman , G. D. , & Drazen , J. M. ( 2001 ). Too close to call . New England Journal of Medicine , 345 ( 11 ), 832 .
Daniel , H. -D. ( 1993 ). Guardians of science: Fairness and reliability of peer review (p . xiv , 118 ). Cambridge: Weinheim: VCH.
Delgado , A. F. , & Delgado , A. F. ( 2017 ). The association of funding source on effect size in randomized controlled trials: 2013-2015-A cross-sectional survey and meta-analysis . Trials , 18 ( 1 ), 125 .
Garfield , E. ( 1996 ). How can impact factors be improved? BMJ , 313 ( 7054 ), 411 - 413 .
Jacques , T. S. , & Sebire , N. J. ( 2010 ). The impact of article titles on citation hits: An analysis of general and specialist medical journals . JRSM Short Reports , 1 ( 1 ), 2 .
Kulkarni , A. V. , Busse , J. W. , & Shams , I. ( 2007 ). Characteristics associated with citation rate of the medical literature . PLoS ONE , 2 ( 5 ), e403 .
Piper , A. , & Wellman , C. ( 2017 ). Publication, power, and patronage: On inequality and academic publishing . Critical Inquiry.
Reuters , T. ( 2017 ). Incites journal citation reports . https://jcr.incites.thomsonreuters.com/. Accessed Sept 20 .
Wang , Y. X. , et al. ( 2014 ). Implications of Web of Science journal impact factor for scientific output evaluation in 16 institutions and investigators' opinion . Quantitative Imaging in Medicine and Surgery , 4 ( 6 ), 453 - 461 .
Weller , A. C. ( 2002 ). Editorial peer review: Its strengths and weaknesses . Medford, NJ: Information Today, Inc.