Online distribution channel increases article usage on Mendeley: a randomized controlled trial
Online distribution channel increases article usage on Mendeley: a randomized controlled trial
Paul Kudlow 0 2 3 4 5 6 7 8 9
Matthew Cockerill 0 2 3 4 5 6 7 8 9
Danielle Toccalino 0 2 3 4 5 6 7 8 9
Devin Bissky Dziadyk 0 2 3 4 5 6 7 8 9
Alan Rutledge 0 2 3 4 5 6 7 8 9
Aviv Shachak 0 2 3 4 5 6 7 8 9
Roger S. McIntyre 0 2 3 4 5 6 7 8 9
Arun Ravindran 0 2 3 4 5 6 7 8 9
Gunther Eysenbach 0 2 3 4 5 6 7 8 9
Paul Kudlow 0 2 3 4 5 6 7 8 9
0 Data Science Team, TrendMD Inc., MaRS Discovery District, West Tower, 661 University Avenue
1 465 , Toronto M5G 1M1, ON , Canada
2 Institute of Medical Science, University of Toronto , Toronto, ON , Canada
3 Department of Psychiatry, Clinician-Investigator Program, University of Toronto , Toronto, ON , Canada
4 Centre for Global eHealth Innovation, Techna Institute, University Health Network , Toronto, ON , Canada
5 Department of Psychiatry, University of Toronto , Toronto, ON , Canada
6 Centre for Addiction and Mental Health , Toronto, ON , Canada
7 Mood Disorders Psychopharmacology Unit, University Health Network , Toronto, ON , Canada
8 University of Toronto , Toronto, ON , Canada
9 Institute of Health Policy , Management and Evaluation , University of Toronto , Toronto, ON , Canada
Prior research shows that article reader counts (i.e. saves) on the online reference manager, Mendeley, correlate to future citations. There are currently no evidenced-based distribution strategies that have been shown to increase article saves on Mendeley. We conducted a 4-week randomized controlled trial to examine how promotion of article links in a novel online cross-publisher distribution channel (TrendMD) affect article saves on Mendeley. Four hundred articles published in the Journal of Medical Internet Research were randomized to either the TrendMD arm (n = 200) or the control arm (n = 200) of the study. Our primary outcome compares the 4-week mean Mendeley saves of articles randomized to TrendMD versus control. Articles randomized to TrendMD showed a 77% increase in article saves on Mendeley relative to control. The difference in mean Mendeley saves for TrendMD articles versus control was 2.7, 95% CI (2.63, 2.77), and statistically significant (p \ 0.01). There was a positive correlation between pageviews driven by TrendMD and article saves on Mendeley (Spearman's rho r = 0.60). This is the first
-
randomized controlled trial to show how an TrendMD enhances article saves on Mendeley.
While replication and further study are needed, these data suggest that cross-publisher
article recommendations via TrendMD may enhance citations of scholarly articles.
Randomized controlled trial Article
TrendMD Knowledge dissemination
Background
As global research output continues to increase, the competition for readers’ attention
amongst scholarly publishers, journals, and, authors is becoming tougher. Traditionally,
scholarly publishers promoted issues of journals containing multiple articles, but with the
increasing dominance of electronic publishing, there is growing interest in promoting
individual articles (Fox et al. 2016a). It remains common practice for authors to promote
their research articles by presenting at conferences; however there is scant evidence to
suggest that such tactics are actually effective at enhancing scholarly article impact
(de
Leon and McQuillin 2014)
. Many journals engage in online tactics, such as promoting
scholarly article links in social media channels to attract and engage readers. There is
robust data on the effectiveness of online advertising [i.e. Google AdWords (Turnbull and
Bright 2008) and social media (Hollis 2005)] for driving purchases of consumer goods and
building brands (Tiago and Ver´ıssimo 2014). However, there is a paucity of data on the
efficacy of social media and other online channels to distribute scholarly content and drive
impact of individual articles (Fox et al. 2016b).
Prior research has yielded inconclusive results as to whether social media can enhance
pageviews and/or article impact
(Fox et al. 2014, 2016b; Dixon et al. 2015; Thoma et al.
2015; Hand et al. 2016)
. For example, a study completed in 2014 by Fox et al. (2014),
found no differences in median 30-day pageviews for articles randomized (n = 121) to a
social media promotion strategy that involved articles receiving posts on Twitter and
Facebook containing a toll-free link to the full-text version of the article. A key limitation,
however, was that investigators did not examine the effects of paid tweets or sponsored
Facebook posts on article pageviews.
The Fox et al. study stirred controversy by some groups in the medical publishing
community, citing that the social media strategy was not intensive enough, and that their
conclusions were not generalizable
(Dixon et al. 2015; Thoma et al. 2015)
. In their
response letter, Thoma et al. (2015) cited their experience running a comprehensive social
media campaign, which lead to a 289% increase in traffic to the Annals of Emergency
Medicine when compared with the prior calendar year. However, as Fox et al. (2016b)
pointed out, these data are from an ecological association, and the traffic increase cannot be
attributed to the social media campaign based on the observational study design. The
increase in traffic could have been due to organic changes in traffic to the journal, rather
than the social media campaign. In another response letter,
Dixon et al. (2015)
found that
article pageviews increased from 3234 to 6768 in the 7 days following the posting of a
blog article on Radiopaedia.org containing a summary of a manuscript. These data,
however, are potentially confounded by selection bias; did the blog post lead to enhanced
visibility, or was the manuscript blogged about noteworthy to begin with?
To address the concerns, Fox et al. (2016b) completed a follow-on randomized
controlled trial (n = 152 articles; 74 social media; 77 control) in 2016 that utilized a more
intensive social media promotion strategy. Investigators retweeted posts of articles on
Twitter to encourage online interaction. To increase the viewership on Facebook,
investigators sponsored Facebook posts for 24 h for a total of $10 USD for each post. Despite
the increased intensity of social media, there were no differences in 30-day article
pageviews between intervention (499.5 median pageviews) and control (450.5 median
pageviews) (Fox et al. 2016b). These data however, are still limited by the fact that paid Tweets
were not used, and that no other article metrics were reported aside from 30-day
pageviews, which have not been found to strongly correlate to future impact (Perneger 2004).
The choice of what metrics to use when assessing the effectiveness of online
distribution tactics is an area of active research. Though citation counts remain the gold standard
of measuring scholarly article impact, citations take a long time to accrue on articles, and
are therefore not well suited to assess the immediate impact of distribution tactics. Several
studies have examined early indicators of impact, known as ‘‘altmetrics’’
(Eysenbach 2011;
Li et al. 2011; Thelwall et al. 2013; Fox et al. 2016a)
. Altmetrics include the number of
times a journal article is viewed (pageviews), downloaded, mentioned, or discussed on
social media, or saved by various citation manager programs such as Mendeley. The
Almetric score is a popular article impact metric that reflects aggregate mentions of articles
on many social media channels (e.g. Twitter, Facebook, etc.), Wikipedia, news, and blogs
(Altmetric 2016)
. However, the literature on the relationship between Altmetric scores and
traditional measures of impact, such as citations is mixed. One study suggests that tweets
of articles on Twitter correlate to citations (Eysenbach 2011). In contrast, other studies
have not found a relationship between tweets or Altmetric scores to citations (Priem et al.
2012; Thelwall et al. 2013).
Replicated studies have found that the most robust early predictor of citations is article
saves/reader counts of scholarly articles on reference managers, such as Mendeley
(Priem
et al. 2012; Lin and Fenner 2013; Zahedi et al. 2014, 2015; Ebrahimy et al. 2016; Maflahi
and Thelwall 2016; Thelwall and Wilson 2016; Li and Thelwall 2012)
. This makes
intuitive sense; as a common practice, scholars save articles in bibliographic software such
as Mendeley in advance of creating other work (i.e. during literature reviews) (Pautasso
2013). Accordingly, a 2014 study revealed that 63% of Web of Science articles from 2005
to 2011 had at least one Mendeley save by April 2013
(Zahedi et al. 2014)
and found a
moderate Spearman Rho correlation (r = 0.49) between Mendeley saves and citation
counts
(Zahedi et al. 2014)
. These findings were replicated by a recent large systematic
review of 90,728 articles published in 7 PloS journals between 2009 and 2013. The study,
which utilized a path analysis method to assess causal relationships, found that Mendeley
article saves preceded, and explained 69% of the variance in article citation counts
(Ebrahimy et al. 2016)
. Investigators found that visibility (as measured by pageviews and
article downloads) was necessary, but not sufficient to drive article saves on Mendeley
(Ebrahimy et al. 2016)
. In other words, the more an article is seen, the higher the
probability for it to be saved in Mendeley; but many articles with high pageview and download
counts did not go on to be highly saved on Mendeley or cited. In contrast, the study
(Ebrahimy et al. 2016)
found that other altmetrics such as article mentions on Twitter,
F1000 recommendations, Facebook posts, and Altmetric scores were not predictive of
future citations
(Ebrahimy et al. 2016)
. These data were largely consistent with other
observational data suggesting article saves on Mendeley is the best early predictor of future
citations
(Priem et al. 2012; Thelwall and Wilson 2016)
.
Notwithstanding, the primary issue facing scholarly content producers is that there are
currently no evidenced-based strategies that have been shown to enhance article saves on
Mendeley. We previously reported that distribution of article links in the cross-publisher
content recommendation network (TrendMD), augment pageviews (Kudlow et al. 2016);
however, we do not know if this increased visibility affects article Mendeley saves. The
purpose of this study was to examine the impact of distributing article links in a
crosspublisher recommendations network on article saves on Mendeley.
Methods
We conducted a 4-week randomized controlled trial that included 400 Open Access articles
published in the Journal of Medical Internet Research (JMIR) between October 1 2014 and
April 30 2016. JMIR is a leading health informatics and health services/health policy
journal (ranked first by impact factor in these disciplines). It focuses on emerging
technologies in health, medicine, and biomedical research (Harriman 2004). We selected
articles published between 6 months and 2 years prior to the beginning of the trial rather
than newly published articles because there is less variation in pageviews, and Mendeley
saves for older articles, which made it more efficient to detect possible effects of the
intervention. Articles were randomized to either the TrendMD arm (n = 200) or the
control arm (n = 200) of the study and outcomes were measured at 4-week. The overall
study design is presented in Fig. 1.
Exclusions:
1. Author promoted articles
on TrendMD
2. All articles published in the
Journal of Medical Internet
Research were disabled from
appearing in the widget
Exclude 3 articles
due to missing data
Overall study design
n = 407 original articles published in the Journal
of Medical Internet Research between October
2014 and April 2016
n = 400 articles randomized
Intervention
For background, TrendMD (www.trendmd.com) is a cross-publisher article
recommendations and distribution platform that is embedded on over 3300 journals and websites
from 300 publishers and seen by approximately 80 million readers per month (TrendMD
Inc. 2017). Participating publishers use TrendMD to distribute their published article links
within the article recommendations displayed on articles within their journals
(non-sponsored recommendations) or on third-party journals within the TrendMD Network
(sponsored recommendations) (Fig. 2). TrendMD’s content distribution model is benchmarked
to similar services in the consumer web, where the leading networks Outbrain (www.
outbrain.com) and Taboola (www.taboola.com) generate the ‘‘From the web’’ and ‘‘You
may like’’ recommendations seen alongside the content on many popular websites like
CNN or New York Times (Kudlow et al. 2016). Among the chief possible reasons why
TrendMD may be an effective distribution channel, is that TrendMD is recommending
articles to readers directly in context, when they are reading other scholarly material
(Fig. 2).
The intervention consisted of exposure of original articles published in JMIR in the
TrendMD Network between November 14 and December 14 2016. Articles included in
the TrendMD Network are displayed as recommended article links (Kudlow et al.
2016). Links to articles randomized to TrendMD were displayed as both non-sponsored
recommended links on online journals published by JMIR Publications Inc. (n = 14)
and sponsored recommended links on third-party publications participating in the
TrendMD Network (n = 3300 journals, 80 million readers per month as of November
14, 2016). The frequency of both non-sponsored and sponsored article link placements
are determined by a relevancy score based on the following: relatedness (i.e. keyword
overlap), collaborative filtering (similar to Amazon’s ‘‘people who bought this item also
bought that item’’), and user clickstream analysis (the Netflix approach, basing
recommendations on the users’ interests expressed through their online history) (Kudlow
et al. 2014, 2016). As a result of the relevancy scoring system, some articles
randomized to TrendMD were seen more often (i.e. accrued more link impressions) than
others in the TrendMD Network. The publisher was charged a cost-per-click fee when
their sponsored article links were clicked. The publishers sponsored links are displayed
in the TrendMD Network so long as they are relevant and the publisher’s account
balance is greater than $0. The 200 articles randomized to TrendMD received a
maximum total budget of $500 at a cost-per-click of $0.4 USD for 1250 sponsored
TrendMD clicks. The actual amount spent by the publisher was $421.60 (of the total
allocated budget of $500) over the 4-week trial (1054 sponsored clicks received by the
200 article randomized to TrendMD at $0.4 cost per click). There is no fee for clicks on
the publisher’s non-sponsored links displayed in the JMIR journals. A summary of how
TrendMD works is presented in Fig. 3.
TrendMD Network
3,300 journals
80M unique readers per month
*Non-sponsored click
Articles randomized
to TrendMD (n=200)
*Sponsored click
Sponsored click
TrendMD recommends links
to schola rly articles across a
cross-publisher network of
>3,300 journals and 80M
unique readers per month
*Sponsored clicks lead to sponsored pageviews; non-sponsored clicks lead to non-sponsored pageviews.
Measured outcomes
Article pageviews
Mendeley saves
Altmetric scores
Control
Articles randomized to control (n = 200) received no promotion in the TrendMD Network
or any other social media networks. Traffic received by articles randomized to control was
by organic means only (e.g. Google, Google Scholar, PubMed, etc.).
Primary outcome (Table 1)
The primary outcome compares the mean Mendeley reader counts (i.e. saves) over the
4-week trial for articles randomized to TrendMD versus control. A Mendeley reader is
counted when an article has been saved to a Mendeley user library account (Mendeley).
Mendeley saves were selected because this metric has been shown to correlate to future
article citations
(Ebrahimy et al. 2016; Thelwall and Wilson 2016)
. Mendeley reader
counts were abstracted through the Altmetric Explorer and cross-referenced with the
Mendeley API (Mendeley).
Secondary outcomes (Table 1)
Mean differences in total pageviews, organic pageviews, and Altmetric scores for articles
randomized to TrendMD versus control were selected as secondary outcomes. Altmetric
scores were included as a secondary outcome because they do not include Mendeley saves
or article pageviews
(Altmetric 2015, 2016)
. In addition we collected engagement metrics,
which include bounce rates, mean time spent on article pages, and pages per session for
readers who clicked on TrendMD sponsored links versus organic pageviews of control
articles. See Table 1 for definitions of outcomes collected. Article pageview and
engagement data were abstracted through JMIR Google Analytics account, including
HTML and PDF pageviews. Altmetric scores data were abstracted through Altmetric
Explorer.
Statistical methods
We performed an a priori power calculation to determine necessary sample size. Based on
our groups’ prior research (Kudlow et al. 2014, 2016), we assumed that both primary and
secondary outcomes had a log-normal distribution. We assumed that the 4-week difference
in mean Mendeley article saves between the control group and TrendMD would be 5, with
a standard deviation of 20. Therefore, assuming a log-normal distribution for 4-week mean
Mendeley saves, an effect of our intervention could be detected at 80% power using a
2-sided (alpha = 0.05) by a sample size of 195 papers in each group (Kadam and Bhalerao
2010).
Baseline characteristics of articles were tabulated and compared on log-transformed
data across randomized study arms using the 2-sample t test for independence. We
categorized articles by publication date and used the Chi-square to test for independence. The
primary analysis compares 4-week mean Mendeley saves by 2-sample t test on the
logtransformed data. Our secondary analysis also uses the 2-sample t test on the
log-transformed data to compare means in pageviews, engagement metrics, and Altmetric scores
between TrendMD and control. We calculated the effect size using Cohen’s d
(Cohen
1977)
. Lastly we performed a multivariate regression analysis using TrendMD sponsored
and non-sponsored pageviews, as well as organic pageviews to predict a change in
Definition
A Mendeley reader is counted when an article
has been saved to a Mendeley user library
account
An instance of a page being loaded in a
browser. Pageviews is a metric defined as
the total number of pages viewed. For
articles randomized to TrendMD, total
pageviews is equal to the sum of organic
pageviews and TrendMD total pageviews
Organic pageviews are equal to total
pageviews for articles randomized to
control. For articles randomized to
TrendMD, organic pageviews is equal to the
difference between total pageviews and
TrendMD total pageviews (i.e. a measure of
total pageviews after subtracting TrendMD
total pageviews)
The total number of pageviews from
TrendMD. It is equal to the sum of
TrendMD non-sponsored and sponsored
pageviews
TrendMD pageviews from clicks on JMIR
non-sponsored article links displayed in
TrendMD recommendations on JMIR Inc.
journals. For the purpose of this
investigation, we assume that 1 TrendMD
non-sponsored click leads to 1 TrendMD
non-sponsored pageview
TrendMD pageviews from clicks on JMIR
sponsored article links displayed in
TrendMD recommendations on
participating publisher sites (3300 as of
November 14, 2016). For the purpose of this
investigation, we assume that 1 TrendMD
sponsored click leads to 1 TrendMD
sponsored pageview
Altmetric.com collects, scores, and weights
mentions of academic articles on social
media platforms (Twitter, Facebook, etc.),
news outlets, and, blog posts. The Altmetric
score is mutually exclusive to Mendeley
saves and pageviews. The Altmetric score is
a proprietary metric from Altmetric.com
The percentage of single-page visits (i.e. visits
in which the person left a website from the
entrance page without interacting with the
page)
Mendeley saves over the 4-week trial. R version 3.3.2 was used to complete the statistical
analysis.
Baseline characteristics (Table 2)
Overall, 400 articles were randomized: 200 to the TrendMD arm and 200 to the control
arm. A Kolmogorov–Smirnov test of the pageview (p = 0.26), Mendeley saves
(p = 0.15), and Altmetric score (p = 0.57) data confirmed that the distributions were
lognormal within the control and TrendMD arms. As shown in Table 2, there were no
differences in article total pageviews (p = 0.40), Mendeley saves (p = 0.35), Altmetric
scores (p = 0.46), or publication date (p = 0.92) at the study onset for articles randomized
to TrendMD versus control.
Primary outcome (Table 3)
Articles randomized to the TrendMD arm received a 77% increase in mean saves on
Mendeley relative to control over the 4-week trial. The mean Mendeley saves for articles
randomized to TrendMD was 6.2 (median = 5; SD = 5.7), compared to 3.5 (median = 2;
Mean difference
(95% confidence
interval)
p value (2- Cohen’s
sample d
t test)
SD = 4.3) for articles randomized to control (Fig. 4). The difference in mean Mendeley
saves for articles randomized to TrendMD versus control was 2.7 saves, 95% CI
(2.63–2.77). The effect size of TrendMD on article Mendeley saves was moderate
(Cohen’s d = 0.53) and statistically significant (p \ 0.01). The cumulative distribution of
article Mendeley saves over the 4-week trial is shown in Fig. 5.
Secondary outcomes (Table 3)
Pageviews
Articles randomized to the TrendMD arm received a 95% increase in mean total pageviews
relative to control over the 4-week trial. The mean total pageviews for articles randomized
to TrendMD was 35.9 (median = 30; SD = 27.1), whereas control articles had a mean of
18.4 total pageviews (median = 13; SD = 28.1). The difference in mean total pageviews
for articles randomized to TrendMD versus control was 17.5 pageviews, 95% CI
(17.11–17.89) (Fig. 6). The effect size of TrendMD on total pageviews was
moderate-tolarge (Cohen’s d = 0.64) and statistically significant (p \ 0.01).
Thirty-percent of the mean total pageviews (mean = 10.8; median = 9; SD = 8.5) of
articles randomized to the intervention were due to clicks on either TrendMD
non-sponsored or sponsored article links. TrendMD non-sponsored clicks lead to a mean of 5.5
pageviews (median = 4; SD = 5.4) and TrendMD sponsored clicks lead to a mean of 5.4
pageviews (median = 4; SD = 5.9). Figure 7 is a histogram of the traffic received by
articles randomized to TrendMD. In Table 4, we examined the top 20 publishers and
journals in the TrendMD Network who sent traffic to articles randomized to TrendMD
through sponsored links.
Lastly, articles randomized to TrendMD received a 37% increase in mean organic
pageviews relative to control over the 4-week trial (Fig. 8). The mean organic pageviews
for articles randomized to TrendMD was 25.2 (median = 17; SD = 24.4), whereas control
articles had a mean of 18.4 organic pageviews (median = 13; SD = 28.1). The difference
in mean organic pageviews for articles randomized to TrendMD versus control was 6.8
pageviews, 95% CI (6.43–7.17). The effect size of TrendMD on organic pageviews was
small (Cohen’s d = 0.26) and statistically significant (p \ 0.01).
TrendMD had a small and statistically significant effect on Altmetric scores. The mean
Altmetric score for articles randomized to TrendMD was 0.44 (median = 0; SD = 2.0),
whereas articles randomized to control had a mean of 0.16 (median = 0; SD = 0.54). The
difference in mean Altmetric scores for articles randomized to TrendMD versus control
was 0.28, 95% CI (0.26–0.30). The effect size of TrendMD on Altmetric scores was small
(Cohen’s d = 0.19) and was a statistically significant (p = 0.031).
TrendMD non- TrendMD
sponsored pageviews sponsored
pageviews
People who visited JMIR by TrendMD sponsored links to articles in the intervention group
were more engaged when compared to those who accessed control articles via organic
means (i.e. PubMed, Google Scholar, Google, etc.). This is evidenced by the fact that in
comparison to who viewed control articles, those who accessed the articles randomized to
TrendMD had lower bounce rates, and visited a greater number of pages per session
(Table 5). There however was no statistical difference in mean session duration between
the two groups.
Multivariate regression
We completed a multivariate regression model for the effects of TrendMD sponsored and
non-sponsored pageviews, as well as organic pageviews on Mendeley article saves over the
4-week trial. The parameters of the model include:
•
•
MS: Mendeley article saves.
TS: TrendMD sponsored pageviews.
TrendMD: sponsored visitors
(n = 1054)
Mean (SD)
9.79% (28.7%)
4.82 (2.17)
•
•
TN: TrendMD non-sponsored pageviews.
O: Organic pageviews.
The linear regression model can be expressed as:
MS ¼ 0:42
TS þ 0:381
TN þ 0:057
O þ e ð
Alexopoulos 2010
Þ
Our model was a good fit; both TrendMD driven pageviews and organic pageviews
predicted 46% of the variation in Mendeley saves. All predictor variables in the model
were statistically significant (p \ 0.0001). Shown in Fig. 9 is a correlation graph between
TrendMD article pageviews and article saves on Mendeley (Spearman’s rho r = 0.60;
r squared = 0.394).
Since we found a statistically significant difference between mean organic pageviews of
articles randomized to TrendMD versus control, we completed a secondary regression
model to examine if pageviews through TrendMD predicted organic pageviews. Though
there was a relationship between TrendMD pageviews and organic pageviews
(Beta = 0.503; p \ 0.0001), TrendMD pageviews only predicted 3.0% (r squared = 0.03)
of the variation in organic pageviews. However, when we removed 2 outlying articles
which received greater than three-times the standard deviation from the mean (10.2196/
jmir.3652, 10.2196/jmir.4052), TrendMD pageviews predicted 12.2% of the variation in
organic pageviews (Beta = 0.722; p \ 0.0001). Lastly, we found no correlation between
total article pageviews and Mendeley saves for articles randomized to control over the
4-week trial (Fig. 10; R squared control 0.011).
This is the first rigorous investigation to show how an online cross-publisher distribution
channel (TrendMD) can be used to increase scholarly article Mendeley saves, pageviews,
and, Altmetric scores. TrendMD had statistically significant effects on all outcomes
measured, with the strongest effect size on pageviews, followed by Mendeley saves, organic
pageviews, and a very small effect size on Altmetric scores. Our study significantly adds to
the relatively scant corpus of literature that examines the efficacy of online strategies to
distribute peer-reviewed content. Prior research has yielded inconclusive results as to whether
online distribution strategies, such as social media can enhance pageviews and/or impact of
scholarly literature
(Fox et al. 2014, 2016b; Dixon et al. 2015; Thoma et al. 2015; Hand et al.
2016)
. These data address an important unmet need of scholarly content providers for
evidenced-based methods to effectively distribute individual peer-reviewed articles.
These findings are consistent with our prior findings, in which we showed how
crosspublisher distribution via TrendMD lead to a 49% increase in weekly article pageviews
relative to baseline traffic over a 3-week period (Kudlow et al. 2016). Our prior findings
were limited however by the crossover design (i.e. we had no control group), relatively
small TrendMD Network size (1100 and 12 million readers per month), and outcome
measure of pageviews. The current study addresses these limitations through the
randomized controlled trial study design, larger TrendMD cross-publisher network ([3300
journals and websites 80 million readers per month), and more robust impact measure of
differences in mean Mendeley saves.
Several findings were interesting to note from the data presented herein. One key
finding was the statistically significant difference in mean organic pageviews between
TrendMD and control. This suggests that individuals arrived at articles randomized to
TrendMD more frequently via the Internet compared to control articles. One possible
explanation is that discovery of articles via TrendMD lead to secondary effects, which lead
to individuals visiting articles randomized to TrendMD more frequently. Some of these
secondary effects could be readers coming back to articles independently (e.g. saving them
as bookmarks on an Internet browser and visiting it later), sharing articles with their
colleagues over email, or spreading via word of mouth. Though we have no methods of
directly measuring attribution of the additional organic pageviews, our data indicates that
TrendMD visitors were more engaged when compared to control. This could indicate that
more secondary effects, such as more sharing of articles took place. Increased independent
return of visitors and sharing is also supported by the fact that TrendMD pageviews were
correlated to organic pageviews. Another secondary effect could have been that TrendMD
lead to enhancements to Search Engine Optimization (SEO) on Google (i.e. TrendMD
articles ranked higher in Google Search results due to more backlinks from TrendMD
recommended links) (Killoran 2013). Therefore, enhancements to SEO could have lead to
more organic pageviews in the TrendMD group versus control.
Another interesting finding was that the standard deviation in organic pageviews was
lower, and the median organic pageviews was higher in articles randomized to TrendMD
versus control. This indicates that TrendMD lead to more evenly distributed visibility of
articles. These data could indicate that TrendMD may encourage discoverability of articles
that aren’t normally seen and/or not normally searched for. The general rule is scholars
tend to read current articles more often than older articles, as part of their strategy for
keeping up to date (Tenopir et al. 2012). However, electronic availability of articles and
better search technology has prompted more reading of older articles by U.S. science
faculty
(Tenopir et al. 2009; Acharya et al. 2014)
. Based on the data collected herein,
TrendMD more evenly distributed visibility of articles, even beyond the effects of
electronic publishing and enhancements in search technology.
Strengths of this study include the rigorous trial design that was adequately powered for
our primary outcome (mean Mendeley saves). The primary outcome was objective and
unbiased between the control and intervention arms. However, some limitations warrant
mention. Firstly, authors PK, MC, DT, DBD, AR, and, GE all have conflicts of interest
with the results presented herein. Risk of bias, however, was mitigated by our
singleblinded, randomized-controlled trial design as well as inclusion of authors, RM, AS, and,
AR, who do not have a conflict of interest. Furthermore, this study was completed with
articles published in an Open Access journal with a potentially technology savvy audience.
Replication is needed in different academic disciplines and including closed-access content
to determine if these data are generalizable. Notwithstanding, the referring publisher data
indicate that visitors came from journals that publish content across academic disciplines,
which suggests that these findings may be generalizable to other disciplines. Another
limitation is that we did not capture any citation data. While prior research suggests that
Mendeley saves are a robust predictor of future citations, we currently do not have any data
to support that our increase in Mendeley usage will lead to future citations. Another
possible limitation of this current investigation is length of the study; it’s possible that
TrendMD’s effect size on Mendeley saves may saturate and diminish over a longer period
of time. Future studies with a longer duration are planned to test the possibility of
saturation effects of cross-publisher distribution via TrendMD on Mendeley saves. Lastly, our
results were limited by the fact that no other online interventions, including social media,
as well as single-publisher article recommendations were tested. Future studies are planned
to test the effects of distribution of articles via both paid and unpaid social media channels,
as well as single-publisher recommendations, in parallel with cross-publisher
recommendations via TrendMD.
Notwithstanding, herein we show how a cross-publisher online distribution channel
(TrendMD) can be used to increase scholarly article saves on Mendeley, pageviews, and,
Altmetric scores. Replicated data has shown that Mendeley article saves correlate to future
citations
(Priem et al. 2012; Lin and Fenner 2013; Zahedi et al. 2014; Ebrahimy et al. 2016;
Maflahi and Thelwall 2016; Thelwall and Wilson 2016; Li and Thelwall 2012)
. Therefore,
while replication and further study are needed, TrendMD may be an online distribution
channel that can be used to increase citations of scholarly articles.
Acknowledgements This research was supported in part by two investment grants made by the Ontario
Centres of Excellence (OCE), a Canadian government-funding program. The grants were, OCE Market
Readiness CC (Grant Number 22292) and OCE Market Readiness CB (Grant Number 23811). We also want
to thank members of the TrendMD team for help with data acquisition.
Compliance with ethical standards
Conflict of interest Dr. Paul Kudlow, Alan Rutledge, and Dr. Gunther Eysenbach are co-founders and
owners of TrendMD Inc. Dr. Gunther Eysenbach is also the founder and owner of JMIR Publications Inc.;
the publisher of the Journal of Medical Internet Research. Dr. Matthew Cockerill is employed by TrendMD
Inc. as an advisor and independent contractor. Danielle Toccalino and Devin Bissky Dziadyk are full-time
employees of TrendMD Inc. Drs. Roger S. McIntyre, Arun Ravindran, and Aviv Shachak are co-supervising
Dr. Paul Kudlow for his graduate work, but declare no financial conflict of interest with TrendMD Inc. or the
data presented herein.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution,
and reproduction in any medium, provided you give appropriate credit to the original author(s) and the
source, provide a link to the Creative Commons license, and indicate if changes were made.
Acharya , A. , Verstak , A. , Suzuki , H. , et al ( 2014 ) Rise of the rest: The growing impact of non-elite journals . arXiv cs.DL.
Alexopoulos , E. C. ( 2010 ). Introduction to multivariate regression analysis . Hippokratia , 14 , 23 - 28 .
Altmetric. ( 2015 ). The donut and Altmetric attention score . https://www.altmetric.com/about-our -data/thedonut-and-score/ . Accessed December 27 , 2016 .
Altmetric. ( 2016 ). How is the Altmetric attention score calculated? https://help .altmetric.com/support/ solutions/articles/6000060969-how -is-the-altmetric-attention-score-calculated . Accessed April 29 , 2017 .
Cohen , J. ( 1977 ). Statistical power analysis for the behavioral sciences (revised ed .). New York: Academic Press.
de Leon , F. L. L. , & McQuillin , B. ( 2014 ). The role of conferences on the pathway to academic impact: Evidence from a natural experiment .
Dixon , A. , Fitzgerald , R. T. , & Gaillard , F. ( 2015 ). Letter by Dixon et al regarding article, ''A randomized trial of social media from circulation'' . Circulation, 131 , e393. doi: 10 .1161/CIRCULATIONAHA.114. 014460.
Ebrahimy , S. , Mehrad , J. , Setareh , F. , & Hosseinchari , M. ( 2016 ). Path analysis of the relationship between visibility and citation: the mediating roles of save, discussion, and recommendation metrics . Scientometrics. doi:10 .1007/s11192-016-2130-z.
Eysenbach , G. ( 2011 ). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact . Journal of Medical Internet Research , 13 , e123. doi: 10 .2196/jmir. 2012 .
Fox , C. S. , Bonaca , M. A. , Ryan , J. J. , Massaro , J. M. , Barry , K. , & Loscalzo , J. ( 2014 ). A randomized trial of social media from circulation . Circulation , 131 ( 1 ), 28 - 33 . doi: 10 .1161/CIRCULATIONAHA.114. 013509.
Zahedi , Z. , Costas , R. , & Wouters , P. ( 2014 ). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ''alternative metrics'' in scientific publications . Scientometrics , 101 , 1491 - 1513 . doi: 10 .1007/s11192-014-1264-0.
Zahedi , Z. , Costas , R. , & Wouters , P. ( 2015 ). Do Mendeley readership counts help to filter highly cited WoS publications better than average citation impact of journals (JCS)? arXiv: 1507 . 02093 .