Evolution of international collaborative research efforts to develop non-Cochrane systematic reviews
Evolution of international collaborative research efforts to develop non-Cochrane systematic reviews
Isabel Viguera-GuerraID 0 2 3
Juan RuanoID 2 3
Macarena Aguilar-Luque 2 3
Jes u?s Gay- MimbreraID 2 3
Ana Montilla 1 2 3
Jose Luis Fern a?ndez-Rueda 2 3
Jos e? Fern a?ndez-Chaichio 2 3
Juan Luis Sanz-Cabanillas 2 3
Pedro Jes u?s G o?mez-Arias 2 3
Antonio V e?lez Garc??a-Nieto 2 3
Francisco G o?mez-Garcia 2 3
Beatriz Isla-Tejera 2 3
0 Agencia de Evaluaci o ?n de Tecnolog ??as Sanitarias de Andaluc ??a (AETSA) , Sevilla , Spain , 2 Instituto Maimonides de Investigacio ?n Biom e ?dica de C o ?rdoba (IMIBIC)/Reina Sof ??a University Hospital/University of C o ?rdoba , Cordoba , Spain , 3 Department of Dermatology, Reina Sof ??a University, Hospital , C o ?rdoba , Spain
1 School of Medicine, University of Cordoba , C o ?rdoba, Spain , 5 Department of Pharmacy, Reina Sof ??a University Hospital , C o ?rdoba , Spain
2 Editor: Bright Nwaru, University of Gothenburg , SWEDEN
3 involved in overall protocol production were the UK , the U.S. , Australia , Brazil, China, Can-
This research-on-research study describes efforts to develop non-Cochrane systematic reviews (SRs) by analyzing demographical and time-course collaborations between international institutions using protocols registered in the International Prospective Register of Systematic Reviews (PROSPERO) or published in scientific journals. We have published an a priori protocol to develop this study. Protocols published in scientific journals were searched using the MEDLINE and Embase databases; the query terms ?Systematic review? [Title] AND ?protocol? [Title] were searched from February 2011 to December 2017. Protocols registered at PROSPERO during the same period were obtained by web scraping all nonCochrane records with a Python script. After excluding protocols that had a fulfillment or duplication rate of less than 90%, they were classified as published ?only in PROSPERO?, ?only in journals?, or in ?journals and PROSPERO?. Results of data and metadata extraction using text mining processes were curated by two reviewers. These Datasets and R scripts are freely available to facilitate reproducibility. We obtained 20,814 protocols of nonCochrane SRs. While ?unique protocols? by reviewers' institutions from 60 countries were the most frequent, a median of 6 (2-150) institutions from 130 different countries were involved in the preparation of ?collaborative protocols?. The highest Ranked countries ada, the Netherlands, Germany, and Italy. Most protocols were registered only in PROSPERO. However, the number of protocols published in scientific journals (924) or in both PROSPERO and journals (807) has increased over the last three years. Syst Rev and BMJ Open published more than half of the total protocols. While the more productive countries were involved in ?unique? and ?collaborative protocols?, less productive countries only partic-
Data Availability Statement: The data and code
underlying this study have been deposited to
GitHub and may be accessed via https://github.
Funding: This work was supported, in part, by
project ICI1400136 to JR, integrated into the
National Plan of R+D+I 2008-2011 and co-financed
by the ISCIII-Subdireccion General de Evaluacion
and European Regional Development Fund (ERDF),
ipated in ?collaborative protocols? that were mainly published in PROSPERO. Our results
suggest that, although most countries were involved in solitary production of protocols for
by project PIN-0316-2017 of the Consejeria de
Salud, Junta de Andalucia (Spain) to JR, and by
grants PP13/009 (JR) and PP17/001 (JG-M) of
Plan Propio de movilidad para investigadores del
Instituto Maimonides de Investigacion Biomedica
de Cordoba (IMIBIC). No funding was received
from any pharmaceutical company.
Competing interests: The authors have declared
that no competing interests exist.
non-Cochrane SRs during the study period, it would be useful to develop new strategies to
promote international collaborations, especially with less productive countries.
Systematic reviews (SRs) and meta-analyses (MAs), the standards for evidence synthesis of
primary studies, are extremely useful for supporting decision making processes in the context of
Health Systems [
]. However, it is necessary to ensure that these decisions are supported by
reviews of the highest methodological quality and that they show the lowest risk of bias [
The Cochrane Handbook for Systematic Reviews of Interventions states that a protocol should
be prepared before publishing an SR [
]. The 2010 Preferred Reporting Items for Systematic
Reviews and Meta-Analyses (PRISMA) statement called for the registration of SR protocols [
]. Preparing an a priori protocol reduces the potential for bias in the review process and
increases the transparency of the analysis and the results . Furthermore, making the content
of a protocol available for public access could also reduce duplication [
], chances of peer
review before starting the review process, and audit discrepancies between the protocol and
the finally produced SR [
Bias limits the possibility of finding or identifying the actual outcomes of a given research
activity. Rigour adherence to the scientific community?s established methodological
agreements minimizes bias and reduces uncertainty about the estimates we make. One of the
proposed methods to reduce bias in SRs is to develop a comprehensive protocol that contains 1)
the sources of primary data; 2) procedures for searching, extracting, filtering, selecting, and
analyzing data; and 3) analytical and methodological tools that will be used to conduct the
research. For now, it is only recommended that such protocols should be prepared before
making the first analysis and that they can be consulted at any time, leaving the trace that were
elaborated much before the final results were published [
Therefore, a protocol should have two main features: first, it should contain all necessary
instructions to reproduce the same results from the respective SR; second, it should be
prepared before the SR is conducted. Currently, a protocol can be freely viewed by the scientific
community, and this allows for the functionality of tracking dates to ensure that the protocol
has been created prior to the review: public repositories and scientific journals provide these
functionalities. PROSPERO (International prospective register of systematic reviews) is an
international database for prospectively registered systematic reviews; it is funded by the
National Institute for Health Research (NIHR). Scientists worldwide use the database, and
within less than a decade, it has reached a landmark achievement of 30,000 registrations.
The database is free and open for all researchers who are planning to conduct an SR and
also for those who are searching for registered, ongoing, or completed reviews in order to
develop meta-epidemiology studies. The PROSPERO Advisory Group Statement of Founding
Principles centers on free access to both registration and searches on the database; it is as
inclusive as possible and achieves the key aims of avoiding duplication and minimizing bias
The second option is to publish the protocol in a scientific journal. This option allows peer
reviewers and editors to assess the quality of the protocol based on scientific criteria. Through
their comments and suggestions, reviewers may suggest changes to improve methodological
quality. However, there is still no empirical evidence from meta-epidemiological studies that
these changes in protocol will be crucial for improving the quality of SRs. There are many
2 / 14
journals that accept protocols of SR for publication. Some of them are BMJ Open (https://
bmjopen.bmj.com/) and Systematic Reviews (https://goo.gl/mFShxv).
No matter where protocols are published, a descriptive analysis of these documents can
give us a glimpse of the efforts made by researchers to carry out SRs. Allers et al. have recently
found that one third of these protocols remained unpublished after 3-5 years, and this could
be an unknown source of bias not studied so far, similar to those meant for many randomized
clinical trials (RCTs) that were registered in public repositories such as ClinicalTrials.gov but
never been published [
]. In a first approximation to this problem, we aimed to describe
comprehensively how different reviewers represented by the institutions and countries to which
they belong, make efforts to design these protocols and their strategies to conduct reviews.
To date, no study has formally assessed the relationships between reviewers, represented by
their respective institution or country, when elaborating an a priori protocol to develop a
nonCochrane systematic review, analysing co-working patterns, and evolution of strategies to
make these protocols publicly accessible.
Materials and methods
A priori published protocol
We published an a priori protocol in Systematic Reviews [
]. This protocol describes source of
data, methods to perform document searches (PROSPERO web scraping and literature
databases queries), eligibility and screening, data extraction, analysis, and the reporting of results.
Web scraping and literature search strategies
Records stored in PROSPERO (https://www.crd.york.ac.uk/prospero/) were obtained by using
a custom Python 3.0 script for web scraping, and Chrome?s Web Scraper website data
extraction tool (http://webscraper.io/) was used for automatic and iterative extraction of raw data
from all the completed non-Cochrane registration records stored from February 2011 to
December 2017. More detailed information about scraping methods is available in Supporting
Information file (S1 File). Protocols published in scientific journals were obtained by querying
MEDLINE and Embase using the RISmed R package and the Boolean terms combination
?[?Systematic Reviews? [Title] AND ?protocol? [Title]]?.
Data filtering and eligibility criteria
Registers or protocols with a less than 90% section fulfilment rate and those that were
duplicated (i.e., those sharing titles and reviewers) were dropped from the dataset. An R script
automatically performed the screening process. Subsequently, the results were subjected to human
verification by two reviewers (JG-M and MA-L) to check for inconsistencies between machine
and human tasks. In that case, human decision was adopted.
Dataset and variables
A working .csv file, which included only the variables that we wished to subject to further
analysis, was obtained (Supplementary methods). Protocols with different reviewer?s affiliation
countries were considered to be the result of international collaboration, and their respective
countries were analyzed because they co-appeared in the protocol as ?unlisted? and tagged as
contributing to ?Collaborative protocols?. Protocols with unique reviewers? affiliation country
were considered to be produced by a unique country and were tagged as ?Unique protocols?.
More detailed information about dataset and variables is available in Supporting Information
file (S1 File).
3 / 14
Demographics and the evolution of protocol production
Considering the source (journal vs. PROSPERO), type of journal, and country, some panels
from the plots were considered to represent changes in the number of protocols published
from 2011 (the year PROSPERO was launched) to 2017 (the year PROSPERO web scraping
was performed. Considering the entire list of affiliation-association countries for all
co-reviewers per protocol, we displayed a world map that used different colors to represent the number
of times any country was involved in any protocol.
Data visualization and statistical analysis
Qualitative variables were summarized by frequency level (number, percentage) or displayed
using several types of graphs (mosaic plots, density/histogram plots, etc.). Quantitative
variables were summarized using the mean (standard deviation) or median (interquartile range)
for non-normally distributed variables. Graphs were produced, and statistics were analyzed
using several R 3.4.4 language packages [R Development Core Team(http://www.R-project.
org]; the Venn diagram was obtained using the eulerr shinny app (http://eulerr.co), and the
workflow figure was created using Review Manager 5 (RevMan 5) software (https://
community.cochrane.org/help/tools-and-software/revman-5). Our analysis can be fully
reproduced by using several source files containing raw data, and R scripts are available in the form
of an R notebook (https://github.com/info4cure/PROSPERO_protocols_demographics). It is
shared as an open source software under the MIT license. A Python script for PROSPERO
web scraping will be publicly released by the end of 2018, once our team completes all the
analyses related to the main project.
Protocol vs. research-on-research study
Our planned search strategy was published in Systematic Reviews journal and compared with
the final reported review methods from this study. The methods for web scraping, filtering,
and selection did not change. However, this is the first article that we have produced for this
research project, and the project constitutes only a partial descriptive analysis compared to the
main goals described in the above-mentioned protocol.
Since our study did not collect primary data, formal ethical assessment and informed consent
were not required.
After scraping 30,000 PROSPERO records, 5,362 documents were excluded because they were
118 not fulfilled, and 903 were duplicated versions of other protocols (Fig 1). After text mining
and manually supervising the obtained dataset, 4,364 protocols were excluded because they
were lacking crucial information required for the analyses. By searching bibliographic
databases, we obtained 1,732 protocols of SRs published in scientific journals. Only 807 protocols
were shared by both PROSPERO and the journal datasets (Fig 2b).
Thus, 20,814 protocols published from 2011 to 2017 were finally included for further analyses.
These protocols comprised 10,888 reviewers affiliated to institutions across 130 countries. The
4 / 14
Fig 1. PRISMA workflow for searching for PROSPERO records and protocols published in scientific journals
with regard to non-Cochrane systematic reviews.
median number of reviewers and institutions per protocol was five in the ranges 1?57 and
1?42, respectively. The total number of produced protocols increased?following an
exponential pattern?from 2011 to 2017, mainly due to those registered ?only in PROSPERO? (Fig 2a);
these were distantly followed by protocols published ?only in journals? or published in
?journal and PROSPERO? (Fig 2b).
Scientific journals vs PROSPERO repository
There were 124 journals where 1,758 protocols of SRs were published (Fig 3a). Some of these
journals (ranked by frequency order) were ?BMC Systematic Reviews? (Syst Rev), ?British
Medical Journal Open? (BMJ Open), ?JBI Database of Systematic Reviews and
Implementation Reports?, ?Medicine (Baltimore)?, ?JMIR Research Protocols?, ?Clinical and Translational
Allergy?, and ?TRIALS? (Figs 2f, 2i and 3a). More than half of all the protocols published in
scientific journals are available in ?Syst Rev? (33%) or ?BMJ Open? (27%) (Fig 3b). The ?Syst
Rev? published the maximum number of protocols during 2011-2017, with the majority (80%)
also being registered at PROSPERO (Fig 3b). This publishing strategy has increased
considerably since 2011, while the number of protocols without PROSPERO registration remained
consistently low during the study period (Fig 3c). This trend seems to be associated with the
major numbers of protocols authored by reviewers affiliated with institutions from the UK
and Canada and, to a lesser extent, Australia (Fig 4c). On the contrary, ?BMJ Open?, the
second journal with a higher number of published protocols seems to follow a different pattern of
protocol publication. First, the same number of published protocols was identified for with vs.
without PROSPERO registration (45%/55%) for the entire study period (Fig 3b). Second, there
was a shift in publication patterns during 2015: published protocols that were also registered in
5 / 14
Fig 2. This panel represents the main features of included protocols. (a) The frequency of protocols published from 2011 to 2017 compared to those
published ?only in journal?, ?only in PROSPERO?, and ?journal and PROSPERO?. (b) Venn diagram of the number of protocols published ?only in
journal? (coral), ?only in PROSPERO? (green), and ?journal and PROSPERO? (blue)?their intersection. (c) Map representation of the number of
protocols produced categorized by country (as a proxy for the reviewer?s affiliation country). Colors represent the levels of productivity, as defined by
the quartiles of a newly recoded variable [abs(log2(country.count / all.countries.count))] (red, very high; yellow, high; green, medium; blue, low). (d),
(e), and (f) represent the word clouds of ?unique countries? (d), ?collaborative countries? (e), and journals (f). Text size and centering is proportional to
the associated number of protocols. Colors have been randomly assigned. (g), (h), and (i) represent column plots of the ?unique countries? (g),
?collaborative countries? (h), and journals (i), which are ranked based on the total number of protocols.
PROSPERO formed the majority for the first period (2011-2015), with a peak in protocols
from China in 2015 (Fig 4c). However, since 2016, published protocols without PROSPERO
registration have become more frequent, and such protocols have been mainly produced by
institutions in the UK (Fig 3c).
Protocols by reviewers? affiliation countries
In the case of most of the countries, reviewers belonging to institutions had participated in at
least one protocol (Fig 1c). The distribution levels of each country?s participation are depicted
in Fig 3.
Despite regional differences (because of the involvement of institutions from around the
world), the generation process of protocols has been very similar. However, we show that
6 / 14
Fig 3. This panel represents the frequency and time-course changes of SR protocol publication by journals. (a) Frequency of protocols published
from 2011 to 2017 by journals compared the protocols published ?only in journal? with protocols published in ?journal and PROSPERO?. (b)
Magnified version of the plot (a) centered on the top 10 most published journals. (c) Evolution of ?only in journal? vs. in ?journal and PROSPERO?
protocol publications from 2011 to 2017 compared to ?BMJ Open? and ?Systematic Reviews? journals. SR: Systematic Review. SR: Systematic Review.
African countries-in comparison with other countries-produced the lowest numbers of
protocols. From a total of 90 countries, we found that most of the protocols (17,431; 90%) were
authored by reviewers from institutions from a single country (31.5% of countries worldwide).
In contrast, a few protocols (1,938) were elaborated by researchers who collaborated with
institutions from two or more countries (from a list of 130) (45.62% of the 285 countries selected)
(Fig 5). The word clouds (Fig 1d?1f) and bar diagrams (Fig 1g?1i) show the predominant
participation of institutions from countries such as the UK, the U.S., Australia, Brazil, Canada,
China, and the Netherlands with regard to producing protocols either in isolation or in
collaboration with others countries (Figs 1e, 1h and 4b).
The production of collaborative protocols has increased in recent years. These protocols
have been published in PROSPERO since 2011, and later, these protocols were also published
in journals (2013); some collaborative protocols were published only in journals (2014).
Protocols from a greater number of countries (more than 30) have appeared since 2016, with more
than 150 different countries participating in producing protocols in 2017. The list of countries
7 / 14
Fig 4. This panel represents the frequency and time-course changes of SR protocol publications categorized by countries. (a) The frequency of
protocols published from 2011 to 2017 categorized by country comparing protocols published ?only in a journal? with protocols published in ?journal
and PROSPERO?. (b) Magnified version of the plot (a) centered on the top 10 most productive countries comparing protocols published ?only in a
journal?, ?only at PROSPERO?, and in both ?journal and PROSPERO?. (c) Evolution of ?only journal? vs. ?journal and PROSPERO? protocols
publications from 2011 to 2017 comparing the top 10 most productive countries.
ranked by participation in collaborative protocols is characterized by the following attributes:
a) larger than the list of countries producing unique protocols; b) the top 10 countries are
repeated in both lists (except China and Brazil, which were replaced by the Netherlands and
France in the collaborative list); c) almost all the countries involved in producing ?unique
protocols? participated in the creation of ?collaborative protocols?, as this is more productive than
working in isolation; d) countries that only participated in collaborative protocols were the
least productive (Fig 5).
Analysis of time-course patterns
Fig 6 displays ?the year of the first protocol published? by ?source of publication? (?only in
PROSPERO?, ?PROSPERO and journal?, ?only in journal?) and by ?country?. To simplify the
analysis, only protocols without collaborations between countries were considered for this
plot. There seem to be four different patterns. The first pattern shows that the most productive
8 / 14
Fig 5. Rank discrepancies between two ordered lists of reviewers? affiliation countries. The ?Unique protocols?
column displays a descending list of reviewers? affiliation countries, which produced protocols in which all reviewers?
institutions belonged to a unique country. The ?Collaborative protocols? column displays a ranked list of reviewers?
affiliation countries that collaborated with other reviewers? affiliation countries to produce protocols for SRs. Arrows
connect the same country from the first list to the second list. Countries represented only in one of the lists are not
connected to/by any arrow. The countries were sub-grouped (Q1:Q4) by cutting through 25%, 50%, and 75% of the total
number of countries in each list. When comparing the ?Unique protocols? and ?Collaborative protocols? lists, the
country position was considered as being modified if the edge connected two different subgroups (i.e., Q1!Q3).
Direction of the change defines 0upgrading0 (Q2 ! Q1, Q3 ! Q1, Q4 ! Q1, Q3 ! Q2, Q4 ! Q3) or 0downgrading0 the
rank position of any country (Q1 ! Q2, Q1 ! Q3, Q1 ! Q4, Q2 ! Q3, Q2 ! Q4, Q3 ! Q4).
PLOS ONE | https://doi.org/10.1371/journal.pone.0211919
9 / 14
Fig 6. Analysis of protocol publication patterns categorized based on the most productive countries. Countries are
listed in a descendent order based on their ?Unique protocol? productivity. The points represent a hallmark in every
country?s history of protocol publication: first publication of a protocol in ?only in journal? (red dot), ?only in
PROSPERO? (green triangle), and in ?journal and PROSPERO? (blue square). Arrows connect two (by a dotted line)
or more (by a full line) hallmarks to emphasize how much time a country requires to adopt a new publication process.
and collaborative countries (the UK, Canada, Australia, China, Germany, Italy, the U.S., and
the Netherlands) started producing protocols very early and submitted them to PROSPERO or
published them in journals; however, it was only after 1-3 years that they started publishing
protocols in both PROSPERO and journals. The second pattern is as follows: some countries
(Denmark, Brazil, Ireland, South Africa, Spain, India, Taiwan, Iran, New Zealand, Japan,
Switzerland, Sweden, South Korea, and France) started publishing their protocols in PROSPERO
and, after 1-3 years in PROSPERO and journals, and finally, after another 1-4 years, they
finally started publishing their protocols only in journals. The third pattern defines countries
(from Belgium to United Arab Emirates) that started using only PROSPERO, not too early,
and most of them, after a longer period of 3-5 years, published only in journals without
submitting their protocols to PROSPERO anytime. In the final pattern, least productive countries
covering from Hong Kong to Namibia, and representing more than half of the world?s
countries, submitted their protocols only to PROSPERO.
This is the first research-on-research study that describes the diversity of collaborative
strategies and time-course preference changes adopted by countries whose institutions are involved
10 / 14
in producing and communicating protocols for SRs. John P. A. Ioannidis defines
research-onresearch or meta-research studies as the study of research itself: its methods, reporting,
reproducibility, evaluation, and incentives [
]. Overall, our findings suggest three observations:
first, most countries are involved in producing protocols for SRs, although the majority of
protocols are produced without international collaboration; second, although most protocols
were earlier registered only in PROSPERO, this tendency seems to have changed since
20132014, and most productive countries have now begun to publish protocols mainly in ?Syst
Rev? or ?BMJ Open? journals; third, less productive countries participate in collaborative
international protocol production, and such protocols are predominantly submitted to
PROSPERO and not to scientific journals.
Our findings in context
Our results show that most protocols for non-Cochrane SRs were authored by reviewers from
institutions belonging to a single country. As most of the topics are not restricted by local,
ethnic, or geographic factors, this approach may become a challenge for transforming evidence
into practice worldwide. Collaborations between countries, especially more vs. less productive
institutions, may enhance technical expertise by providing training to the latter regions and
extending collaboration beyond SRs, thus improving the adoption of evidence-based health
policies, selecting the best evidence for the right audience, and increasing focus on relevant
issues through appropriate methodologies [
]. This will be possible with the growing
innovation in tools and platforms that could enable more efficient SR production through
The registration forms of protocols submitted to PROSPERO are only checked against the
scope for inclusion in the repository and for clarity of content. Once accepted, an audit trail
detailing the major changes to planned methods may be checked at any time, even after the SR
is published. Ideally, registration should take place before the researchers start formal
screening against inclusion criteria, but reviews are eligible as long as they have not progressed
beyond completing data extraction. However, accomplishing these goals is still determined by
the reviewers? integrity. Dawid Pieper and Katharina Allers recently suggested that an a priori
design of an SR may not have the same advantages and potential to reduce the risk of bias in
SRs compared to those of RCTs. They argue that ?living SRs?, involving new workflows and
collaboration tools, text mining and machine learning technologies, emerging reusable data
repositories, shared ontologies, and harmonized data transfer protocols can be expected to
decrease any potential manipulations in the future.
Since 2011, there have been few meta-epidemiological studies about the content and use of
PROSPERO. A descriptive analysis of the number of PROSPERO registrations and the website
usage from 2011 to 2017 has recently been published, and it explores the epidemiological
characteristics and completeness of pre-specification primary outcomes in a small sample of
PROSPERO records [
]. They highlight the exponential increase in registered protocols at
PROSPERO between 2011 and 2017. However, these authors recognize that there are still
many caveats regarding the real utility of making a document, which has not been reviewed
methodologically, available for public access. These authors?one is a member of
PROSPERO?s international advisory group?raise three issues that will certainly generate future
debate about new strategies for future improvements for PROSPERO: how closely do
published SRs adhere to the planned methods of PROSPERO registrations?; can the specification
of greater outcomes in PROSPERO registrations prevent inclusion and reporting biases?; and
do registered SRs address the necessary question with regards to developing the review?
11 / 14
Limitations and strengths
Our study analyzed the largest sample of SR protocols produced in the last seven years. We did
not perform PROSPERO registration record sampling [
]. Rather, our objective was to obtain
the entire universe of registers (from the first document to the last one registered just before
the date selected for web scraping), not representative samples of them. The search specificity
for non-Cochrane PROSPERO registration records was based on a Python script that was
designed to recognize only the format of these records; this differs from registration records
for Cochrane and non-human studies. These cannot be scraped using our script due to the
structural differences in PROSPERO forms. Thus, the sensitivity and specificity for the web
scraping is 100%.
The present study, however, also had several limitations. First, we used countries as proxies
for reviewers or reviewers? institutions. Better size of information granularity would have
enabled deeper analysis regarding the reviewers? and institutions? productivity and
collaborative networks. However, based on technical issues related to the variety of formats used by
reviewers when fulfiling PROSPERO forms and time limitations on the project, we decided to
use countries as the unit of analysis. We have used this approach previously to analyze how the
author-paper affiliation network architecture influences the methodological quality of SRs and
MAs for psoriasis [
]. Second, our study is limited to protocols in non-Cochrane SRs.
Cochrane Reviews are demonstrated to have better methodological quality and lower bias risk
than non-Cochrane SRs [
]. The Cochrane Database of Systematic Reviews (CDSR) is the
leading resource for Cochrane SRs and protocols. Protocols for Cochrane Reviews have also
been published in PROSPERO since 1 October 2013. This fact introduces a publishing gap at
PROSPERO, with only non-Cochrane SRs being published from 2011 to 2013. Another area of
particular concern in relation to non-Cochrane reviews is the failure to register reviews at the
outset. Registration of Cochrane reviews is mandatory with publication of a protocol a priori.
To avoid an unbalanced sample of protocols with different proportions of quality and rates of
publication, we decided to select only non-Cochrane SRs protocols for our study.
Implications of results
Future work should focus on analyzing co-authorship networks. This would help to identify
academic talent, integrate the research interest of experienced researchers and local
investigators who can recognize regional resources and health-care necessities that change over time,
thus increasing opportunities for improving international collaborations. A recent article that
mapped 115,000 RCTs has demonstrated the mismatch between research efforts and health
needs in non-high-income regions [
]. Similarly, implementing strategies is necessary for the
efficieny coordinatioe of collaborations between countries to perform non-Cochrane SRs, and
this is especially so when most human resources and stakeholders are not coordinated by a
consolidated international organization such as the Cochrane.
Our results demonstrate that most protocols are registered only in PROSPERO. However,
interested reviewers should consider these facts: PROSPERO registrations cannot be
methodologically curated, critically peer-reviewed, or freely commented upon by anonymous readers.
Whether protocols should be submitted a priori to a public repository and/or to a journal is
debatable. Indeed, our data show that this tendency has been changing since 2013-2014, and
most productive countries have begun publishing protocols in ?Syst Rev? or ?BMJ Open?.
Future studies comparing the methodological quality of protocols registered ?only in
PROSPERO?, ?only in journals?, and in ?PROSPERO and journals? should provide empirical
evidence for an a priori peer-reviewing process of protocols before authors start SR
development. Furthermore, it would be interesting to explore whether the modifications suggested by
12 / 14
peer-reviewers after submitting an SR protocol to a journal significantly improves the quality
of protocols (i.e., assessment by using PRISMA for Protocols extension), and even to increase
the methodological quality and reduce the RoB in the final SR would be of great interest.
Conclusions and future research
Although most countries worldwide were involved in producing protocols for non-Cochrane
SRs, it is important to develop new strategies to boost international collaborations, especially
between more productive and less productive countries. While most protocols of SRs are
submitted to PROSPERO, the potential advantage of the new trend of publishing protocols in
scientific journals and not only in PROSPERO should be evaluated further.
We wish to acknowledge the assistance we received from Dr. Dawid Pieper (Head of
Department, Evidence-based health services research, Institute for Research in Operative Medicine,
Faculty of Health, School of Medicine at Witten/Herdecke University, Cologne, Germany); he
helped us in interpreting the patterns of collaborative relationships between countries during
the analysis phase of our study. We are grateful to all the contributors and to the advisory
group of the PROSPERO registry. We are also grateful to the reviewers for their helpful
comments and suggestions. We would also like to thank Editage (www.editage.com) for English
Conceptualization: Isabel Viguera-Guerra, Juan Ruano.
Data curation: Macarena Aguilar-Luque, Jesu?s Gay-Mimbrera, Ana Montilla.
Formal analysis: Isabel Viguera-Guerra, Juan Ruano.
Funding acquisition: Juan Ruano.
Methodology: Isabel Viguera-Guerra, Juan Luis Sanz-Cabanillas, Pedro Jesu?s Go?mez-Arias.
Project administration: Isabel Viguera-Guerra, Juan Ruano.
Resources: Isabel Viguera-Guerra, Ana Montilla.
Software: Jose Luis Ferna?ndez-Rueda, Jose? Ferna?ndez-Chaichio.
Supervision: Antonio Ve?lez Garc??a-Nieto, Francisco Go?mez-Garcia, Beatriz Isla-Tejera.
Visualization: Isabel Viguera-Guerra, Juan Ruano.
Writing ? original draft: Isabel Viguera-Guerra, Juan Ruano.
Writing ? review & editing: Juan Ruano, Antonio Ve?lez Garc??a-Nieto, Francisco
Go?mez-Garcia, Beatriz Isla-Tejera.
13 / 14
1. Abuabara K , Freeman EE , Dellavalle R . The role of systematic reviews and meta-analysis in dermatology . J Invest Dermatol 2012 ; 132 :e2. https://doi.org/10.1038/jid. 2012 .392 PMID: 23069910
2. Gomez-Garcia F. , Ruano J. , Gay-Mimbrera J. , Aguilar-Luque M. , Sanz-Cabanillas J.L. , Alcalde-Mellado P. , et al. Most systematic reviews of high methodological quality on psoriasis interventions are classified as high risk of bias using ROBIS tool . J Clin Epidemiol 2017 ; 92 : 79 - 88 . https://doi.org/10.1016/j. jclinepi. 2017 . 08 .015
3. Higgins J.P.T. , Green S . (Eds.), Cochrane handbook for systematic reviews of interventions , Version 5.1.0 [updated March 2011 ], The Cochrane Collaboration 2011 .
4. Moher D. , Shamseer L. , Clarke M. , Ghersi D. , Liberati A. , Petticrew M. , et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement . Syst Rev 2015 ; 4:1 . https://doi.org/10.1186/2046-4053-4-1
5. Siontis K.C. , Hernandez-Boussard T. , Ioannidis J.P.A. Overlapping meta-analyses on the same topic: survey of published studies . BMJ 2013 ; 347 :f4501. https://doi.org/10.1136/bmj.f4501
6. Ioannidis JP . The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses . Milbank Q 2016 ; 94 : 485 - 514 . https://doi.org/10.1111/ 1468 - 0009 .12210 PMID: 27620683
7. Kirkham JJ , Altman DG , Williamson PR . Bias due to changes in specified outcomes during the systematic review process . PLoS One 2010 ; 5:e9810 . https://doi.org/10.1371/journal.pone. 0009810 PMID: 20339557
8. Pandis N , Fleming PS , Worthington H , Dwan K , Salanti G. Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews . PLoS One 2015 ; 10 : e0137667. https://doi.org/10.1371/journal.pone. 0137667 PMID: 26368938
9. Tricco AC , Cogo E , Page MJ , Polisena J , Booth A , Dwan K , MacDonald H , Clifford TJ , Stewart LA , Straus SE , Moher D. A third of systematic reviews changed or did not specify the primary outcome: a PROSPERO register study . Journal of Clin Epidemiol 2016 ; 79 : 46 - 54 . https://doi.org/10.1016/j.jclinepi. 2016 . 03 .025
10. Moher D , Shamseer L , Clarke M , Ghersi D , Liberati A , Petticrew M , Shekelle P , Stewart LA . Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P ) 2015 statement. Syst Rev 2015 ; 4:1 . https://doi.org/10.1186/2046-4053-4-1 PMID: 25554246
11. Allers K , Hoffmann F , Mathes T , Pieper D. Systematic reviews with published protocols compared to those without: more effort, older search . J Clin Epidemiol 2018 ; 95 : 102 - 10 . https://doi.org/10.1016/j. jclinepi. 2017 . 12 .005 PMID: 29258907
12. Ruano J , Go? mez-Garc??a F , Gay-Mimbrera J , et al. Evaluating characteristics of PROSPERO records as predictors of eventual publication of non-Cochrane systematic reviews: a meta-epidemiological study protocol . Syst Rev 2018 ; 7 : 43 . https://doi.org/10.1186/s13643-018 -0709-6 PMID: 29523200
13. Ioannidis JPA . Meta-research: Why research on research matters. PLoS Biol 2018 ; 16 ( 3 ): e2005468. https://doi.org/10.1371/journal.pbio. 2005468 PMID: 29534060
14. Su TT , Bulgiba AM , Sampatanukul P , Sastroasmoro S , Chang P , Tharyan P , et al. Clinical Epidemiology (CE) and Evidence-Based Medicine (EBM) in the Asia Pacific region (Round Table Forum) . Prev Med 2013 ; 57 Suppl: S5 - 7 . https://doi.org/10.1016/j.ypmed. 2013 . 04 .009 PMID: 23624253
15. Page MJ , Shamseer L , and Tricco AC . Registration of systematic reviews in PROSPERO: 30,000 records and counting . Syst Rev 2018 ; 7 : 32 . https://doi.org/10.1186/s13643-018 -0699-4 PMID: 29463298
16. Sanz-Cabanillas JL , Ruano J , Gomez-Garcia F , Alcalde-Mellado P , Gay-Mimbrera J , Aguilar-Luque M , et al. Author-paper affiliation network architecture influences the methodological quality of systematic reviews and meta-analyses of psoriasis PLoS One 2017; 4: e0175419 . https://doi.org/10.1371/journal. pone.0175419
17. Fleming PS , Seehra J , Polychronopoulou A , Fedorowicz Z , Pandis N. Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm? European Journal of Orthodontics 2013 ; 2 : 244 - 248 . https://doi.org/10.1093/ejo/cjs016
18. Goldkuhle M , Narayan VM , Weigl A , et al. A systematic assessment of Cochrane reviews and systematic reviews published in high-impact medical journals related to cancer . BMJ Open 2018 ; 8:e020869 . https://doi.org/10.1136/bmjopen-2017 -020869 PMID: 29581210
19. Atal I , Trinquart L , Ravaud P , and Porcher R. A mapping of 115,000 randomized trials revealed a mismatch between research effort and health needs in non-high-income regions . Journal of Clinical Epidemiology 2018 ; 98 ; 123 - 32 . https://doi.org/10.1016/j.jclinepi. 2018 . 01 .006 PMID: 29360559