Worldwide trends in volume and quality of published protocols of randomized controlled trials
Worldwide trends in volume and quality of published protocols of randomized controlled trials
Belle V. van Rosmalen 1 2
Ingo Alldinger 2
Kasia P. Cieslak 1 2
Roos Wennink 0 2
Mike Clarke 2
Usama Ahmed Ali 0 2
Marc G. H. Besselink 1 2
0 Department of Surgery, University Medical Centre Utrecht , Utrecht , The Netherlands , 4 Northern Ireland Network for Trials Methodology Research, Queen's University Belfast , Belfast , Northern Ireland
1 Department of Surgery, Academic Medical Centre , Amsterdam , The Netherlands , 2 Department of General, Visceral and Transplantation Surgery, University of Heidelberg , Heidelberg , Germany
2 Editor: Matthias Briel , Universitatsspital Basel , SWITZERLAND
Data Availability Statement: All relevant data are
within the paper and its Supporting Information
Funding: The authors received no specific funding
for this work.
Competing interests: The authors have declared
that no competing interests exist.
A systematic search was performed in PubMed and EMBASE, identifying RCT protocols
published over a decade from 1 September 2001. Data were extracted on quality
characteristics of RCT protocols. The primary outcome, methodological quality, was assessed by
individual methodological characteristics (adequate generation of allocation, concealment of
allocation and intention-to-treat analysis). A comparison was made by publication period
(First, September 2001- December 2004; Second, January 2005-May 2008; Third, June
2008-September 2011), geographical region and medical specialty.
The number of published RCT protocols increased from 69 in the first, to 390 in the third
period (p<0.0001). Internal medicine and paediatrics were the most common specialty
topics. Whereas most published RCT protocols in the first period originated from North America
(n = 30, 44%), in the second and third period this was Europe (respectively, n = 65, 47% and
n = 190, 48%, p = 0.02). Quality of RCT protocols was higher in Europe and Australasia,
compared to North America (OR = 0.63, CI = 0.40±0.99, p = 0.04). Adequate generation of
allocation improved with time (44%, 58%, 67%, p = 0.001), as did concealment of allocation
(38%, 53%, 55%, p = 0.03). Surgical protocols had the highest quality among the three
specialty topics used in this study (OR = 1.94, CI = 1.09±3.45, p = 0.02).
Publishing RCT protocols has become popular, with a five-fold increase in the past decade.
The quality of published RCT protocols also improved, although variation between geographical regions and across medical specialties was seen. This emphasizes the importance of international standards of comprehensive training in RCT methodology.
In 2004, the International Committee of Medical Journals Editors (ICMJE) announced that
randomized controlled trials (RCT) should be registered in a public trials registry before the
recruitment of the first participant. This registration is now a condition for publication of the
final trial results.[1±3]
Although trial registries have many benefits, some authors have suggested that they do not
provide full and transparent information about RCT methodology.[4±6] Furthermore, a
systematic review highlighted changes between the information in trial registries and the full
] Publishing RCT protocols gives authors the opportunity to fully explain
the rationale and proposed methods for their trial as well as related ethical and safety issues.[
] Although publishing RCT protocols is not a common practice yet, it would potentially
benefit trial users and complement the information in trial registries.[
] Moreover, some
experts have suggested that it should be mandatory to publish a protocol in order to minimise
publication bias, false sample size reporting, switching of endpoints and increase transparency.
In recent years, several studies have addressed trends in the number and methodological
quality of RCTs, [7, 11±13] and some addressed the discrepancies between RCTs and their
initially published protocols.[14±16] However, no previous study has analysed trends in the
publication of study protocols. Thus, little is known about trends in the practice of publishing trial
protocols and their methodological quality. Since clinical medicine depends heavily on RCTs,
transparency and high quality of RCT protocols is crucial. Therefore, the aim of this study was
to assess worldwide trends in the volume and methodological quality of published protocols of
RCTs through the first decade of the 21st century.
This study aimed to analyse trends in the publication of RCT protocols by assessing their
volume and methodological quality across specialties and geographic regions.
Search strategy and selection process
PubMed and EMBASE were searched for trial protocols published in a ten-year period (1
September 2001 to 1 September 2011). In order to interpret the current status of methodological
protocol quality, the time interval of ten years was chosen to minimize sampling error, and to
ensure sustainability of the results. Also it provides an interesting insight in the development
of methodological quality over time. The search syntax was as follows: (design rationale trial
AND (randomised OR randomized)) OR (protocol trial AND (randomised or randomized)).
2 / 13
All retrieved abstracts were screened according to the inclusion and exclusion criteria by two
reviewers (KC, IA). If the relevance was uncertain, the full text of the article was obtained and
reviewed. All disagreements were resolved through discussion and reaching consensus by
including a third reviewer (MGB).[
] Protocols were included if they described a RCTs,
defined as any prospective study assessing the effect of health care interventions in humans,
randomly allocated to one of at least two study groups. Studies were excluded when (1) trial
results were listed rather than protocols, (2) the study was not a RCT, (3) the study was not a
study in humans, (4) the publication was not written in the English language, (5) no abstract
was present, (6) no full text was present, and (7) the protocol was published after the study had
The primary study outcome was methodological quality, with as secondary outcome the
volume of published protocols. Methodological quality was assessed on two parameters:
1. Individual methodological characteristics: All protocols were appraised according to a list
adapted from the Cochrane risk of bias assessment tool and Chan and Altman's review
including the following characteristics [
· Specification of primary outcome: adequate if primary outcome was explicitly specified in
· Sample size calculation: adequate if performed and reported.
· Generation of allocation sequence: adequate if method of generation was reported and
considered adequate (i.e. computer, random table, shuffle of cards).
· Concealment of treatment allocation: adequate if method of concealment was reported
and considered adequate (i.e. envelopes, central unit for randomization, pharmacy, and
· Any blinding: adequate if any type of blinding was performed.
· Double blinding: adequate if both patient and one of the following were blinded: physician,
observer, adjudication / consensus committee.
· Type of analysis: adequate if intention-to-treat analysis was explicitly mentioned.
2. High vs. low quality designs: a trial was designated as `high quality' if all three of the
following methodological items were adequately reported: generation of allocation, concealment
of allocation and intention-to-treat analysis. Blinding was not included as an item. Some
have claimed that the role of blinding is overstated [
]. Blinding may be impossible in
some surgical trials.[21±24] Estimating correct implementation (or legitimate
non-implementation) of blinding will not be possible, considering the great variety of possibilities to
implement blinding among medical specialties. Therefore concealment of sequence
generation was chosen instead, since it is a more generalizable parameter.
Data extraction and definitions
The following geographical, publishing and epidemiological characteristics were extracted:
geographical region, specialty (based on the corresponding author and divided into the
following (arbitrary) categories: Internal medicine and paediatrics, primary care, surgery (including
subspecialties) and other), number of study centres, study arms (two arms, or three and more),
3 / 13
number of randomized patients, trial design, funding (any kind of involvement of the industry
was stated as commercial), presence of written informed consent, presence of a data safety
monitoring board and plan for dealing with adverse events.
Characteristics and outcomes of included protocols were compared for three approximately
equal periods: September 2001 to December 2004, January 2005 to May 2008 and June 2008 to
Because of the search strategy used, only study protocols published in the last 4 months of
the year 2001 and the first 8 months of the year 2011 were included. A random sample of
publications from those years was added to the database as substitute for the missing months in
2001 and 2011. This was conducted according to the following manner: the number of
publications per month of included protocols that were scored on quality in 2001 and 2011
respectively was calculated. This mean was multiplied by the number of missing months (8 in 2001
and 4 months in 2011 respectively). This resulted in 3 protocols being added to 2001 and 40 to
2011. The added protocols were randomly selected from period 1 and period 3, respectively.
Subgroup analyses were based on geographical region and medical specialty. The rational
for examining geographical variation as well as medical specialties was that previous research
demonstrated differences in methodological quality of surgical trials between continents.[
Dichotomous outcomes were presented as the number (percentage) of events, whereas
medians and interquartile ranges were used for continuous data. Study groups were compared by
Fisher exact, χ2 and Mann-Whitney U tests, as appropriate. A p-value of <0.05 was considered
significant. The odds ratio (OR) with the corresponding 95% confidence intervals (95% CIs)
was calculated for comparison of methodological quality between subgroups by means of
univariate and multivariate logistic regression. All variables were included in the univariate
analysis. Variables showing potential association (p<0.2) in the univariate analysis were
subsequently included in the multivariate analysis.[
] IBM SPSS Statistics for Windows
Version 20.0 (IBM Corp., Armonk, NY, USA) was used for all statistical analysis.
Our search identified a total of 11 782 records. The selection process is depicted in figure A
of the supporting information. The screening of the titles resulted in the selection of 6074
potentially relevant publications, and after screening by title and abstract, 615 publications
remained. After final selection of full-text, 553 eligible protocols were identified. A random
sample of 43 protocols was added to the database, resulting in a total of 596 protocols.
General characteristics and volume
was observed, although the numbers remained relatively low in the surgical category (n = 6,
19, 41). The proportion of non-industry-funded trials almost doubled between the first and
last time period from 27 (39%) to 254 (65%), (p<0.0001). There was a decrease in the reporting
of the use of a data safety committee: from 54% in the first period to about 36% in the third
period, (p = 0.02).
Methodological quality of the included protocols is presented in Table 2.
5 / 13
Legend: This table describes all characteristics concerning methodological quality, subdivided by period of publication. Study groups were compared by
Fisher exact, χ2 and Mann-Whitney U tests, as appropriate. A p-value of <0.05 was considered signi®cant.
* de®ned as presence of the following 3 criteria: adequate generation of allocation, concealment of allocation and intention-to-treat analysis.
The proportion of high quality protocols increased non-significantly across the study
periods: 18 (26%), 43 (31%) and 143 (37%), (p = 0.17). Adequate methods for generation and
concealment of allocation improved significantly over time (p = 0.03). Blinding was applied
relatively frequent (about 70%) throughout the study periods, while the use of a blinded
observer increased from 48% in the first period to 67 (60%) in the third (p = 0.02). The number
of studies attempting to blind patients decreased significantly over time: from 39% and 37% in
the first two periods, respectively, to 26% in the third (p = 0.02). The rate of double-blinding
also decreased accordingly, from 39% and 36% to 25%, respectively (p<0.008). There was a
non-significant increase in explicit intention-to-treat analysis, from 62% in the first period to
75% in the third (p = 0.08).
Subgroup and regression analysis
Subgroup analysis by geographic region is presented in Table 3.
6 / 13
Legend: This table describes the subgroup analysis for methodological quality characteristics subdivided by specialty or region. Study groups were
compared by Fisher exact, χ2 and Mann-Whitney U tests, as appropriate. A p-value of <0.05 was considered signi®cant.
* Presence of the following three criteria: adequate generation of allocation, concealment of allocation and intention-to-treat analysis.
Adequate generation and concealment of allocation were equally frequent in RCT protocols
from Europe and Australasia (around 60%), while less often so in protocols from North
America (52% and 43%, respectively, p 0.01). A similar trend was observed for adequate type of
planned analyses (i.e. explicitly intention to treat): 79% and 76% for European and
Australasian, respectively, compared to 60% for North American protocols (p<0.0001). However, for
blinding, North America achieved the highest percentages on practically all parameters. This is
reflected in a double-blinding proportion of 36% compared to 24% and 25% for European and
Australasian protocols, respectively (p = 0.02).
Subgroup analyses comparing the three most common specialties (Internal medicine and
paediatrics, primary care and surgery) were performed (Table 3). A significant difference in
adequate generation of allocation and adequate concealment allocation was observed with the
highest percentage achieved by surgery protocols (p = 0.02 and p = 0.002, respectively). On the
7 / 13
other hand internal medicine and paediatrics consistently scored the highest percentage for
blinding. The highest percentage of high quality protocols was in surgery (44%), (p = 0.018).
Univariate regression analysis shows origin from North America to be negatively associated
with methodological quality, while origin from Europe, primary care or surgery as specialty,
presence of informed consent and presence of a plan for adverse event were predictors for
high methodological quality. In multi-variate analysis, all these factors were confirmed as
independent predictors for methodological quality (Table 4).
This first systematic empirical literature-based study on volume and quality of RCT protocols
found a five-fold increase in the number of RCT protocols published over a ten-year period.
Although the overall quality of published protocols improved, there were differences between
continents, with protocols from Australasia and Europe being of higher quality than those
8 / 13
from North America. This empirical literature-based study also found medical specialty to be
correlated with the quality of published RCT protocols. Both primary care and surgical trials
were associated with significantly higher quality compared to internal medicine and paediatric
protocols. However, the confidence intervals of these parameters were relatively broad.
Therefore it cannot be excluded that confounders that were not accounted for in the multivariate
analysis contributed to the overall significance.
Previous studies have identified similar trends for the volume of published RCTs, as for
published protocols.[26, 28±32] Whether protocol publication is increasing in popularity, or
whether the augmentation in volume can be subscribed as a direct consequence of the
increased amount of RCTs remains uncertain. In contrast with the current study, previous
studies found published RCTs from Australasia to have the lowest rates of adequate reporting.
[26, 33±36] The higher quality of surgical trials is remarkable especially since surgery used to
have a reputation of being based on tradition rather than scientific research.[
] A possible
explanation for this phenomenon could be that due to the increasing rate of technological
innovation more (e.g. minimally invasive) techniques have become available which allow for
randomized comparisons. Moreover, increased awareness of the importance of surgical trials
and enhanced training in trial methodology may have attributed to this improvement, but
data are lacking. The recent IDEAL (Idea, Development, Exploration, Assessment, Long-term
Follow-up) framework for surgical innovation may provide guidance for further improvement
of trials on surgical interventions.[
A troubling, and yet unexplained, finding is the apparent decrease in the use of a data safety
committee from 54% in the first period to 36% in the third period. Close monitoring of this
development is imperative.[
] In fact, the presence of a plan to handle adverse events seemed
to be the strongest indicator for high quality RCT protocols. This might be explained by the
importance of having such a plan is especially important in trials with a high degree of trial
complexity; such trials will have been designed more carefully.[
] The intention of gaining
written informed consent was also found to be a marker for high protocol quality. It seems
that evidence-based guidance on how to design and perform RCTs would be welcomed. The
Trial Forge platform and the SPIRIT guidelines (Standard Protocol Items: Recommendations
for Interventional Trials) could be instrumental in this aspect as it strives to provide a
systematic approach to improving trials and their protocols.[
A shortcoming of our study is that the quality of the protocol does not automatically
translate into the quality of the RCT. Although previous studies have compared the quality of RCTs
with the quality of their protocols, such studies are scarce. Furthermore, they have used small
samples and some of their results are contradictory.[
9, 15, 45
] Whether the trials described in
the protocols in our study will be performed and published as designed, should be investigated
further. This might reveal important insights in the life cycle of RCTs, and would allow
prospective evaluation of factors that might be related to early termination of RCTs and
non-publication. Another shortcoming of this study is that instead of the SPIRIT guidelines, the Cochrane
risk of bias tool was used. The use of the SPIRIT item check list would have expanded the
analysis. The drawback of the SPIRIT checklist, however, is that it covers over 50 items, including
recommendations on version identifiers and statements regarding who obtained informed
consent and who have access to the final data. These data are often not available in the published
protocols. Therefore, a more selective approach was chosen in which a selected list of items was
evaluated with empirical evidence showing their importance in that they affect final outcomes
of RCTs. Additionally, the fact that these items have been used previously in several studies
allows comparison between studies.
Medical specialties were subdivided into three fairly broad and subjective groups, in order
to compare and contrast our findings across this range of subspecialties. This might have
9 / 13
resulted in a loss of detail.[
] Furthermore, only protocols published in English were
included, which may have led to an underestimation of the number of published protocols,
assuming that some are published in other languages. The inclusion of protocols was not
limited to the top listed medical journals, which is a strength of this study. Also our review covers
all medical specialties, which makes our study results generalizable.
In conclusion, this systematic review found a five-fold increase in the number of published
study protocols in the past decade. The methodological quality of the protocols improved
during the same period but varies greatly between regions and medical specialties, which suggests
that different regions and medical specialties may face different challenges when seeking to
improve the quality of RCTs. Nevertheless, it is important to strive for such improvements,
given the importance of RCTs and systematic reviews of them as a source of reliable and robust
evidence on the effects of healthcare interventions. Comprehensive training in RCT
methodology, as for example is already offered in a master programme at the University of Oxford,
amongst others, could benefit responsible conduct and reporting of RCTs greatly. The
involvement of international medical societies in developing standards for training could enhance
RCT quality improvement world-wide.
S1 File. Figure A in: Flow chart of systematic search strategy in PubMed and EMBASE.
Figure B in S1 File: Volume of published protocols in number, per region; This figure
describes the number of published protocols (listed on the x-axis) for different regions, sub
divided in three time periods (listed on the y-axis).
S2 File. Figure C in S2 File: Filled in Prisma checklist.
Conceptualization: MB UAA IA BvR MC.
Data curation: BvR UAA.
Formal analysis: BvR UAA.
Investigation: BvR RW KC.
Methodology: BvR UAA MB.
Project administration: MB BvR UAA KC.
Supervision: MB UAA KC BvR.
Writing ± original draft: BvR MB UAA KC.
Writing ± review & editing: BvR IA UAA KC RW MB MC.
10 / 13
11 / 13
12 / 13
1. De Angelis CD , Drazen JM , Frizelle FA , Haug C , Hoey J , Horton R , et al. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. Lancet . 2005 ; 365 ( 9474 ): 1827 ± 9 . Epub 2005/06/01. https://doi.org/10.1016/S0140- 6736 ( 05 ) 66588 - 9 PMID: 15924965
2. De Angelis C , Drazen JM , Frizelle FA , Haug C , Hoey J , Horton R , et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Lancet . 2004 ; 364 ( 9438 ): 911 ± 2 . Epub 2004/09/15. https://doi.org/10.1016/S0140- 6736 ( 04 ) 17034 - 7 PMID: 15364170
3. Ghersi D , Pang T. From Mexico to Mali: four years in the history of clinical trial registration . Journal of evidence-based medicine . 2009 ; 2(1):1±7 . Epub 2009/02/01. https://doi.org/10.1111/j.1756- 5391 . 2009 . 01014 . x PMID : 21348976
4. Zarin DA , Ide NC , Tse T , Harlan WR , West JC , Lindberg DA . Issues in the registration of clinical trials . Jama . 2007 ; 297 ( 19 ): 2112 ± 20 . Epub 2007/05/18. https://doi.org/10.1001/jama.297.19.2112 PMID: 17507347
5. Zarin DA , Tse T , Williams RJ , Califf RM , Ide NC . The ClinicalTrials.gov results databaseÐupdate and key issues . The New England journal of medicine . 2011 ; 364 ( 9 ): 852 ± 60 . Epub 2011/03/04. PubMed Central PMCID: PMCPmc3066456. https://doi.org/10.1056/NEJMsa1012065 PMID: 21366476
6. Faure H , Hrynaszkiewicz I. The ISRCTN Register: achievements and challenges 8 years on . Journal of evidence-based medicine . 2011 ; 4 ( 3 ): 188 ± 92 . Epub 2011/09/10. PubMed Central PMCID: PMCPmc3206222. https://doi.org/10.1111/j.1756- 5391 . 2011 . 01138 . x PMID : 21894613
7. Dwan K , Altman DG , Cresswell L , Blundell M , Gamble CL , Williamson PR . Comparison of protocols and registry entries to published reports for randomised controlled trials . The Cochrane database of systematic reviews . 2011 ; (1):Mr000031 . Epub 2011/01/21.
8. Besselink MG , Gooszen HG , Buskens E. Clinical trial registration and the ICMJE . Jama . 2005 ; 293 ( 2 ): 157 ± 8; author reply 8 . Epub 2005/01/13.
9. Chan AW , Hrobjartsson A , Haahr MT , Gotzsche PC , Altman DG . Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles . Jama . 2004 ; 291 ( 20 ): 2457 ± 65 . Epub 2004/05/27. https://doi.org/10.1001/jama.291.20.2457 PMID: 15161896
10. Hopewell S , Dutton S , Yu LM , Chan AW , Altman DG . The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed . BMJ (Clinical research ed) . 2010 ; 340 : c723 . Epub 2010 /03/25. PubMed Central PMCID: PMCPmc2844941 .
11. Chan AW , Altman DG . Epidemiology and reporting of randomised trials published in PubMed journals . Lancet . 2005 ; 365 ( 9465 ): 1159 ± 62 . Epub 2005/03/30. https://doi.org/10.1016/S0140- 6736 ( 05 ) 71879 - 1 PMID: 15794971
12. Siegfried N , Clarke M , Volmink J . Randomised controlled trials in Africa of HIV and AIDS: descriptive study and spatial distribution . BMJ (Clinical research ed) . 2005 ; 331 ( 7519 ): 742 . Epub 2005/10/01. PubMed Central PMCID: PMCPmc1239977 .
13. Bastian H , Glasziou P , Chalmers I . Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS medicine . 2010 ; 7 ( 9 ): e1000326 . Epub 2010 /09/30. PubMed Central PMCID: PMCPmc2943439. https://doi.org/10.1371/journal.pmed. 1000326 PMID: 20877712
14. Blumle A , Meerpohl JJ , Schumacher M , von Elm E. Fate of clinical research studies after ethical approvalÐfollow-up of study protocols until publication . PloS one . 2014 ; 9 ( 2 ): e87184 . Epub 2014 /03/ 04. PubMed Central PMCID: PMCPmc3929354. https://doi.org/10.1371/journal.pone. 0087184 PMID: 24586265
15. Chan AW , Hrobjartsson A , Jorgensen KJ , Gotzsche PC , Altman DG . Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols . BMJ (Clinical research ed) . 2008 ; 337 : a2299 . Epub 2008 /12/06. PubMed Central PMCID: PMCPmc2600604 .
16. Decullier E , Lheritier V , Chapuis F . Fate of biomedical research protocols and publication bias in France: retrospective cohort study . BMJ (Clinical research ed) . 2005 ; 331 ( 7507 ): 19 . Epub 2005/06/22. PubMed Central PMCID: PMCPmc558532 .
17. Edwards P , Clarke M , DiGuiseppi C , Pratap S , Roberts I , Wentz R . Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records . Statistics in medicine. 2002 ; 21 ( 11 ): 1635 ± 40 . Epub 2002/07/12. https://doi.org/10.1002/sim.1190 PMID: 12111924
18. Higgins JP , Altman DG , Gøtzsche PC , JuÈni P , Moher D , Oxman AD , et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials . BMJ (Clinical research ed) . 2011 ; 343 : d5928 .
19. Devereaux PJ , Manns BJ , Ghali WA , Quan H , Lacchetti C , Montori VM , et al. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials . Jama . 2001 ; 285 ( 15 ): 2000 ± 3 . Epub 2001/04/20. PMID: 11308438
20. Schulz KF , Grimes DA . Blinding in randomised trials: hiding who got what . Lancet (London, England) . 2002 ; 359 ( 9307 ): 696 ± 700 . Epub 2002/03/07.
21. Kao LS , Aaron BC , Dellinger EP . Trials and tribulations: current challenges in conducting clinical trials . Archives of surgery (Chicago , Ill: 1960 ). 2003 ; 138 ( 1 ): 59 ± 62 . Epub 2003/01/07.
22. Lilford R , Braunholtz D , Harris J , Gill T. Trials in surgery . The British journal of surgery . 2004 ; 91 ( 1 ):6± 16 . Epub 2004/01/13. https://doi.org/10.1002/bjs.4418 PMID: 14716788
23. McCulloch P , Taylor I , Sasako M , Lovett B , Griffin D. Randomised trials in surgery: problems and possible solutions . BMJ (Clinical research ed) . 2002 ; 324 ( 7351 ): 1448 ± 51 . Epub 2002/06/18. PubMed Central PMCID: PMCPMC1123389 .
24. Karanicolas PJ , Farrokhyar F , Bhandari M. Practical tips for surgical research: blinding: who, what, when , why, how? Canadian journal of surgery Journal canadien de chirurgie . 2010 ; 53 ( 5 ): 345 ± 8 . Epub 2010/09/23. PubMed Central PMCID: PMCPMC2947122 . PMID: 20858381
25. Schulz KF , Grimes DA . Allocation concealment in randomised trials: defending against deciphering . Lancet (London, England) . 2002 ; 359 ( 9306 ): 614 ± 8 . Epub 2002/02/28.
26. Ahmed Ali U, van der Sluis PC , Issa Y , Habaga IA , Gooszen HG , Flum DR , et al. Trends in worldwide volume and methodological quality of surgical randomized controlled trials . Ann Surg . 2013 ; 258 ( 2 ): 199 ± 207 . Epub 2013/06/19. https://doi.org/10.1097/SLA.0b013e31829c7795 PMID: 23774315
27. Mickey RM , Greenland S. The impact of confounder selection criteria on effect estimation . American journal of epidemiology . 1989 ; 129 ( 1 ): 125 ± 37 . Epub 1989/01/01. PMID: 2910056
28. Agha R , Cooper D , Muir G. The reporting quality of randomised controlled trials in surgery: a systematic review . International journal of surgery (London, England) . 2007 ; 5 ( 6 ): 413 ± 22 . Epub 2007/11/22.
29. Scales CD Jr., Norris RD , Keitz SA , Peterson BL , Preminger GM , Vieweg J , et al. A critical assessment of the quality of reporting of randomized, controlled trials in the urology literature . The Journal of urology . 2007 ; 177 ( 3 ): 1090 ± 4; discussion 4±5 . Epub 2007/02/14. https://doi.org/10.1016/j.juro. 2006 . 10 .027 PMID: 17296417
30. Solomon MJ , McLeod RS . Clinical studies in surgical journalsÐhave we improved? Diseases of the colon and rectum . 1993 ; 36 ( 1 ): 43 ± 8 . Epub 1993/01/01. PMID: 8416778
31. Schumm LP , Fisher JS , Thisted RA , Olak J. Clinical trials in general surgical journals: are methods better reported? Surgery . 1999 ; 125 ( 1 ): 41 ± 5 . Epub 1999/01/16. PMID: 9889796
32. Juni P , Altman DG , Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials . BMJ (Clinical research ed) . 2001 ; 323 ( 7303 ): 42 ± 6 . Epub 2001/07/07. PubMed Central PMCID: PMCPmc1120670 .
33. Chen Y , Li J , Ai C , Duan Y , Wang L , Zhang M , et al. Assessment of the quality of reporting in abstracts of randomized controlled trials published in five leading Chinese medical journals . PloS one . 2010 ; 5 ( 8 ): e11926 . Epub 2010 /08/07. PubMed Central PMCID: PMCPmc2914031. https://doi.org/10.1371/ journal.pone. 0011926 PMID: 20689853
34. Wang G , Mao B , Xiong ZY , Fan T , Chen XD , Wang L , et al. The quality of reporting of randomized controlled trials of traditional Chinese medicine: a survey of 13 randomly selected journals from mainland China . Clinical therapeutics . 2007 ; 29 ( 7 ): 1456 ± 67 . Epub 2007/09/11. https://doi.org/10.1016/j.clinthera. 2007 . 07 .023 PMID: 17825697
35. Xu L , Li J , Zhang M , Ai C , Wang L . Chinese authors do need CONSORT: reporting quality assessment for five leading Chinese medical journals . Contemporary clinical trials . 2008 ; 29 ( 5 ): 727 ± 31 . Epub 2008/ 06/27. https://doi.org/10.1016/j.cct. 2008 . 05 .003 PMID: 18579449
36. Hall JC , Hill D , Watts JM . Misuse of statistical methods in the Australasian surgical literature . The Australian and New Zealand journal of surgery . 1982 ; 52 ( 5 ): 541 ± 3 . Epub 1982/10/01. PMID: 6959608
37. Horton R . Surgical research or comic opera: questions, but few answers . Lancet . 1996 ; 347 ( 9007 ): 984 ± 5 . Epub 1996/04/13. PMID: 8606606
38. Neugebauer EA , Morino M , Habermalz B . Surgical research or comic opera? Let's give answers! Surgical endoscopy . 2008 ; 22 ( 6 ): 1411 ± 2 . Epub 2008/04/10. https://doi.org/10.1007/s00464-008-9906-z PMID: 18398640
39. McCulloch P , Cook JA , Altman DG , Heneghan C , Diener MK . IDEAL framework for surgical innovation 1: the idea and development stages . BMJ (Clinical research ed) . 2013 ; 346 : f3012 . Epub 2013 /06/20. PubMed Central PMCID: PMCPmc3685515 .
40. Cook JA , McCulloch P , Blazeby JM , Beard DJ , Marinac-Dabic D , Sedrakyan A . IDEAL framework for surgical innovation 3: randomised controlled trials in the assessment stage and evaluations in the long term study stage . BMJ (Clinical research ed) . 2013 ; 346 : f2820 . Epub 2013 /06/20. PubMed Central PMCID: PMCPmc3685513 .
41. Administration FaD. Guidance for Clinical Trial Sponsors Establishment and Operation of Clinical Trial Data Monitoring Committees . 2006 .
42. Noah BA . Adverse Drug Reactions: Harnessing Experiential Data to Promote Patient Welfare . University of Florida College of Law 2000: 449 : 55 .
43. Chan AW , Tetzlaff JM , Gotzsche PC , Altman DG , Mann H , Berlin JA , et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials . BMJ (Clinical research ed) . 2013 ; 346 : e7586 . Epub 2013 /01/11. PubMed Central PMCID: PMCPmc3541470 .
44. Treweek S , Altman DG , Bower P , Campbell M , Chalmers I , Cotton S , et al. Making randomised trials more efficient: report of the first meeting to discuss the Trial Forge platform . Trials . 2015 ; 16 : 261 . Epub 2015/06/06. PubMed Central PMCID: PMCPmc4475334. https://doi.org/10.1186/s13063-015 -0776-0 PMID: 26044814
45. Pildal J , Chan AW , Hrobjartsson A , Forfang E , Altman DG , Gotzsche PC . Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study . BMJ (Clinical research ed) . 2005 ; 330 ( 7499 ): 1049 . Epub 2005/04/09. PubMed Central PMCID: PMCPmc557221 .