Expediting citation screening using PICo-based title-only screening for identifying studies in scoping searches and rapid reviews
Rathbone et al. Systematic Reviews
Expediting citation screening using PICo- based title-only screening for identifying studies in scoping searches and rapid reviews
John Rathbone 0
Loai Albarqouni 0
Mina Bakhit 0
Elaine Beller 0
Oyungerel Byambasuren 0
Tammy Hoffmann 0
Anna Mae Scott 0
Paul Glasziou 0
0 Centre for Research in Evidence Based Practice, Bond University , Gold Coast , Australia
Background: Citation screening for scoping searches and rapid review is time-consuming and inefficient, often requiring days or sometimes months to complete. We examined the reliability of PICo-based title-only screening using keyword searches based on the PICo elements-Participants, Interventions, and Comparators, but not the Outcomes. Methods: A convenience sample of 10 datasets, derived from the literature searches of completed systematic reviews, was used to test PICo-based title-only screening. Search terms for screening were generated from the inclusion criteria of each review, specifically the PICo elements-Participants, Interventions and Comparators. Synonyms for the PICo terms were sought, including alternatives for clinical conditions, trade names of generic drugs and abbreviations for clinical conditions, interventions and comparators. The MeSH database, Wikipedia, Google searches and online thesauri were used to assist generating terms. Title-only screening was performed by five reviewers independently in Endnote X7 reference management software using OR Boolean operator. Outcome measures were recall of included studies and the reduction in screening effort. Recall is the proportion of included studies retrieved using PICo title-only screening out of the total number of included studies in the original reviews. The percentage reduction in screening effort is the proportion of records not needing screening because the method eliminates them from the screen set. Results: Across the 10 reviews, the reduction in screening effort ranged from 11 to 78% with a median reduction of 53%. In nine systematic reviews, the recall of included studies was 100%. In one review (oxygen therapy), four of five reviewers missed the same included study (median recall 67%). A post hoc analysis was performed on the dataset with the lowest reduction in screening effort (11%), and it was rescreened using only the intervention and comparator keywords and omitting keywords for participants. The reduction in screening effort increased to 57%, and the recall of included studies was maintained (100%). Conclusions: In this sample of datasets, PICo-based title-only screening was able to expedite citation screening for scoping searches and rapid reviews by reducing the number of citations needed to screen but requires a thorough workup of the potential synonyms and alternative terms. Further research which evaluates the feasibility of this technique with heterogeneous datasets in different fields would be useful to inform the generalisability of this technique.
Systematic review; Rapid review; Scoping search; Expediting citation screening; Title screening; Semi-automation; PICo
There is no universal definition of what constitutes a
scoping search although various criteria have been
]. In general, scoping searches are useful to
attain a preliminary assessment of the size and scope of
research literature and to help assess the feasibility of
conducting research, including determining whether
clinical questions have previously been evaluated, or are
up-to-date, and for estimating time frames and
budgetary considerations. Similarly, rapid reviews have no
universally agreed-upon definition but typically are a form
of knowledge synthesis where some components of the
systematic review process are simplified or omitted to
produce information in a timely manner [
Scoping searches and rapid reviews both seek knowledge
using a less formalised and rigorous methodology compared
with systematic reviews. Rapid reviews attempt to expedite
the process by modifying tasks that traditional systematic
reviews eschew due to the concerns over data loss [
tasks that are modified include literature searching, which
may be expedited by restricting the number of databases
], restricting by date range and language types
] or omitting grey literature searches. Other strategies
include restricting the number of personnel who screen
studies, abstract data and assess risk of bias [
Citation screening of the title and abstract is
timeconsuming because of the large number of citations typically
retrieved (the average retrieval from a PubMed search
produces 17,284 citations [
]) and is imprecise with often over
98% of citations from systematic searches excluded after
title/abstract and full text screening [
]. Titles of
published studies typically incorporate the main components of
a study design which can be categorised into the PICo
components (Participants, Intervention, and Comparator, but
not the Outcome). Therefore, screening restricted to the title
field using the PICo components and the associated
synonyms should identify the corpus of relevant studies whilst
also being more precise, due to the constrained screening
method. The aim of this study was to evaluate the feasibility
of PICo-based title-only screening for scoping searches and
rapid reviews by measuring the reduction in screening effort
and the maintenance of recall of relevant records.
A convenience sample of 10 datasets derived from the
literature searches of completed systematic reviews were
used to test the PICo-based title-only screening. Seven
7, 9–12, 14, 16
] available to the authors were
used, and an additional three datasets [
8, 13, 15
created by replicating the search strategy from the published
reviews. These three reviews were selected prima facie
based on being intervention studies that contained
adequately reported search strategies and study inclusion
details. Duplicate records were removed from the datasets to
provide a more homogeneous corpus so that any benefits
from reduced screening effort would not be artificially
inflated by the presence of duplicate records. We used a
convenience sample of five reviewers, (three clinicians and
two non-clinicians) based at the Centre for Research in
Evidence-Based Practice, Bond University, to assess the
reliability and reproducibility of PICo-based title-only
screening. All reviewers were familiar with evidence-based
practice and systematic review methodology. Four
reviewers had no prior knowledge of the datasets used and
were unaware of the titles of included studies. One
reviewer was a co-author on three of the review datasets;
however, the PICo screening terms were derived from the
inclusion criteria of the reviews and were not based upon
potential recollection of titles of included studies.
Each reviewer independently compiled a list of search
terms derived from the inclusion criteria of the reviews,
specifically the (P) Participants, (I) Interventions and (C)
Comparators but not the Outcomes. PICo synonyms
including drug trade names and alternate names for clinical
conditions were sought in the MeSH database, Wikipedia,
online thesauri and Google searches. Typically, three to
four synonyms were generated for each term (see
Appendix 1), but there was no restriction on the number of
terms used. For further examples, see Additional file 1.
Keywords with both British and American spellings were
used, and keywords with different suffixes were truncated
using an asterisk. PICo-based title-only screening was
performed in Endnote X7 reference management software
using ‘OR’ Boolean operator (see Appendix 2).
Outcome measures (Table 1) were the recall of included
studies and the reduction in screening effort (RSE). Recall
is the proportion of included studies retrieved using PICo
title-only screening out of the total number of included
studies in the original reviews. The percentage reduction
in screening effort is the proportion of records that do not
need to be screened i.e. those records which are
eliminated from the set needing screening by the PICo
screening method. This was reported individually for each
reviewer and as the median value across the five scores. A
post hoc analysis was performed with one of the datasets
(Parkinson’s) to examine the impact of screening using
only keywords for the (I) Intervention and (C)
Comparator and omitting keywords for (P) Participants.
Ten systematic reviews were evaluated with a total of
31,359 records. Reduction in screening effort across the
reviews (Fig. 1) ranged from 11% (Parkinson’s review) to
78% (Phenytoin review) with a median reduction in
screening effort of 53%. The recall of reports of included
studies was 100% in 9 of the 10 reviews (Table 2). In the
*One reviewer identified all three studies
†Review included 11 studies, with 2 not retrieved from bibliographic database
searches and assumed to be from hand searches and therefore not included
in the analysis but would have been retrieved with PICo screening
oxygen therapy review, 4 of 5 reviewers missed the same
included study (median recall 67%). Differences in the
selection of PICo keywords between reviewers affected
the reduction in screening effort, the most notable
example being the Phenytoin dataset. Four reviewers
obtained similar results with a reduction in screening effort
of 78%, whereas one reviewer (reviewer 3) achieved only
a 42% reduction in screening effort due to the inclusion
of the PICo screening keyword ‘epilepsy’. The focus of
treatment was seizure prophylaxis after brain injury.
Post hoc analysis
The minimal reduction in screening effort in the
Parkinson’s dataset was principally due to the keyword
‘Parkinson’ retrieving 80% of all records. A post hoc analysis
was performed to determine if complete recall could be
maintained and reduction in screening effort improved
when relying only on keywords for the intervention(s)
and comparator(s) but not the participants. Screening
without type of participants improved the median
reduction in screening effort from 11 to 57%, and the recall of
included studies was 100% (Fig. 2).
Our results indicate that PICo-based title-only screening
considerably reduces the workload of citation screening,
maintains high recall of relevant studies and can be used
to expedite scoping searches and rapid reviews.
Reduction in screening effort
The reduction in screening effort ranged from 11 to 78%
with seven of the datasets having a reduction in
screening effort above 50%. The two prognostic review datasets
(Prostate and Thyroid cancer) had a median reduction
in screening effort of 12 and 19%; however, these reviews
used a more focused search and are atypical of most
search strategies. The post hoc analysis was undertaken
because the reduction in screening effort was minimal in
the Parkinson’s dataset due to 80% of the citations
containing the keyword ‘parkinson’ or variations e.g.
‘parkinsonian’ in the title field, and therefore, the median
reduction in screening was only 11%; the post hoc
analysis found that restricting the PICo search terms to only
the intervention and comparator maintained 100% recall
and improved the reduction in screening effort to 57%.
This could be a useful strategy to ensure a reduction in
screening effort is maintained when a particular PICo
term is ubiquitous within a dataset.
The median reduction in screening effort was 53% but
varied considerably from 11 to 78%. PICo-based title-only
screening may confer minimal time-saving advantages
when the reduction in screening is only small (such as
10–20%) as found in the relatively small prostate and
thyroid cancer datasets, although knowledge of such
scenarios would be difficult to foresee. However, for
searches that are not highly focused, even a small
reduction in screening effort may be advantageous, especially
when datasets are large. In addition, general searches
conducted in MEDLINE typically produce over 17,000
], suggesting that most searches are not highly
focused and these would also benefit by applying
PICobased title-only screening. Care must be taken to ensure
British and American spellings and suffix variations are
incorporated into the keyword screening and that
compound terms e.g. ‘transoesophageal echocardiography’ are
entered as separate search terms to allow for variations in
word order; otherwise, relevant citations could potentially
be missed when using PICo-based title-only screening.
The recall was 100% in 9 of 10 systematic reviews. One
reviewer, a clinician, identified all included studies across
the 10 reviews including the oxygen therapy review;
however, four reviewers missed the same included study in the
oxygen therapy review. ‘Ventilation’ was used in the title
as an alternative term for oxygen therapy, and this was
not listed in the MeSH database nor found whilst
searching other resources, and therefore, subject knowledge was
needed to identify the study. Nonetheless, for other
datasets, PICo-based title-only screening was reliable.
Strengths and limitation of the research
The strengths in this study were that 10 datasets were
used to test the hypothesis that using PICo-based
titleonly screening could retrieve all studies that should have
been found and reduce the number of citations to
screen. Also, the results were reproducible for recall in 9
of 10 datasets, and the methodology is simple and easily
implemented by reviewers or information specialists
with knowledge of screening and Endnote software. The
datasets used were a convenience sample, and reduction
in screening effort may differ with different clinical
specialities and study designs. Applying PICo screening to
datasets with less structured vocabulary, such as reviews
of non-drug interventions and in fields such as health
services research and the social sciences, is likely to be
more challenging. Nonetheless, in this study, the sample
of reviews tested included a variety of clinical specialties,
different types of interventions and different study
designs, such as diagnostic accuracy, prognostic and
The limiting factor for the applicability for screening is
the presence or absence of either controlled or
consistent vocabulary. The high recall and improvement in the
reduction in screening effort was due to the sample
datasets using clearly defined terms for (P) clinical
conditions, (I) interventions and (C) comparators, but using
PICo-based title-only screening where the ontology is
less clearly defined (e.g. where there are no MeSH terms
indexed) could potentially affect recall; in such scenarios,
PICo-based title-only screening may be unsuited for
rapid review but would remain useful for scoping
searches since identifying all studies is not the objective.
This potential for error, however, could be allayed by
including topic experts to help compile search terms.
However, it has been shown that the retrieval of relevant
studies for inclusion can be impaired in rapid reviews
when the number of databases searched or the number
of screeners is restricted [
]. Similarly, traditional title
and abstract screening for systematic review can be
imperfect with relevant studies wrongly excluded [
screening methodology could also be applied to
systematic review screening where one reviewer examines all
records whilst a second reviewer screens the sub-set
identified from PICo-based-title screening. Also, the
PICo screening methodology could be adapted for
searching directly in biomedical bibliographic databases.
This study has examined expediting screening on the
assumption that titles of articles will include at least one
of the PICo components to enable a focused title-based
search to identify all relevant studies and minimise the
number of citations to screen. Other methods have
been developed to expedite screening using
semiautomated predictive algorithms that ‘learn’ to
distinguish relevant and irrelevant citations [
]. The recall
and reduction in screening effort from PICo-based
title-only screening are similar to those achieved with
semi-automated predictive algorithms [
However, semi-automated screening algorithms require an
initial training set (typically ~ 25% of the total citations)
to be manually screened in order to train the algorithm.
This step could be expedited by incorporating
PICobased title-only screening to generate a sub-set of
citations to automatically train the algorithm and dispense
with manual training. Further work is needed to
explore how PICo screening can be incorporated into
machine learning technologies to further accelerate the
training of datasets.
In the sample of datasets used, PICo-based title-only
screening was able to expedite citation screening for
scoping searches and rapid reviews by reducing the
number of citations to screen but requires a
thorough workup of the potential synonyms and
alternative terms. Further research which evaluates the
feasibility of this technique with heterogeneous
datasets in different fields (such as health services
research) and intervention types (such as non-drug
interventions) would be useful to inform the
generalisability of this technique.
Appendix 1. Example of PICo-based search terms used for screening
Clopidogrel and aspirin versus aspirin alone for stroke
prevention: a meta-analysis
Participants—people with stroke or transient ischaemic
Intervention—clopidogrel and aspirin
Appendix 2: Example of PICo-based title-only screening using OR Boolean operator in Endnote reference management software (oxygen therapy)
Additional file 1: Example of PICo-based search terms used for
screening. (DOCX 22 kb)
MEDLINE: Medical Literature Analysis and Retrieval System Online;
MeSH: Medical subject headings; PICo: Participant, Interventions,
Comparators, Outcomes; RSE: Reduction in screening effort
The authors would like to thank Justin Clark and David Honeyman for replicating
the search strategies and Evelyne Rathbone for graphing the data with
R statistical program.
This study received funding from the NHMRC Australia Fellowship:
Availability of data and materials
JR, EB, TH and PG contributed to the study design and concept. JR devised
the testing and analysis and drafted the initial manuscript. JR, LA, MB, OB
and AMS created the screening terms. All authors contributed to the
manuscript and revisions and approved the final manuscript.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published
maps and institutional affiliations.
1. The Joanna Briggs Institute Reviews' Manual 2015: methodology for JBI scoping reviews . http://joannabriggs.org/assets/docs/sumari/ReviewersManual_ Methodology-for- JBI-Scoping-Reviews _ 2015 _v2. pdf. Accessed 22 Nov 2016 .
2. Levac D , Colquhoun H , O'Brien K. Scoping studies: advancing the methodology . Implement Sci . 2010 ; 5 .
3. Arksey H , O'Malley L . Scoping studies: towards a methodological framework . Int J Soc Res Methodol Theory Pract . 2005 ; 8 : 19 - 32 .
4. Tricco A , Antony J , Zarin W , Strifler L , Ghassenmi M , Ivory J , Perrier L , Hutton B , Moher D , Straus S. A scoping review of rapid review methods . BMC Med . 2015 ; 13 : 224 .
5. Ganann R , Ciliska D , Thomas H. Expediting systematic reviews: methods and implications of rapid reviews . Implement Sci . 2010 ; 5 : 56 .
6. Islamaj Dogan R , Murray GC , Névéol A , Lu Z. Understanding PubMed user search behavior through log analysis . Database J Biol databases curation . 2009 ; 2009 : 1 .
7. De Sutter A , Saraswat A , van Driel M. Antihistamines for the common cold . Cochrane Database Syst Rev . 2015 ;(11): CD009345 .
8. Tan S , Xiao X , Ma H , Zhang Z , Chen J , Ding L , Yu S , Xu R , Yang S , Huang X , Hong H. Clopidogrel and aspirin versus aspirin alone for stroke prevention: a meta-analysis . PLoS One . 2015 ; 13 ; 10 ( 8 ): e0135372 .
9. Holmes M , Rathbone J , Littlewood C , Rawdin A , Stevenson M , Stevens J , Archer R , Evans P , Wang J . Routine echocardiography in the management of stroke and transient ischaemic attack: a systematic review and economic evaluation . Health Technol Assess . 2014 ; 18 ( 16 ): 1 - 176 .
10. Rathbone J , Kaltenthaler E , Richards A , Tappenden P , Bessey A , Cantrell A. A systematic review of eculizumab for atypical haemolytic uraemic syndrome (aHUS) . BMJ Open . 2013 ; 3 : 1 - 11 .
11. Oxygen therapy for pneumonia . In Press.
12. Ren S , Rathbone J , Cooper K , Gomersall T , Stevens J , Harnan S , Simpson E , Sutton A , Anderson J , Cooper J , Smith H , Shaikh S : The efficacy and safety of pharmacological therapies used for advanced Parkinson's disease: a systematic review and network meta-analysis . In Press.
13. Zafar S , Khan A , Ghauri A , Shamim M. Phenytoin versus Leviteracetam for seizure prophylaxis after brain injury-a meta analysis . BMC Neurol . 2012 ; 12 : 30 .
14. Bell K , Del Mar C , Wright G , Dickinson J , Glasziou P . Prevalence of incidental prostate cancer: a systematic review of autopsy studies . Int J Cancer . 2015 ; 137 : 1749 - 57 .
15. Asrani S , Leise M , West C , Murad M , Pedersen R , Erwin P , Tian J , Wiesner R , Kim W. Use of sirolimus in liver transplant recipients with renal insufficiency: a systematic review and meta-analysis . Hepatology . 2010 ; 52 : 1360 - 70 .
16. Furuya-Kanamori L , Bell K , Clark J , Glasziou P , Doi S. Prevalence of differentiated thyroid cancer in autopsy studies over six decades: a metaanalysis . J Clin Oncol . 2016 ;
17. Pham M , Waddell L , Rajic A , Sargeant J , Papadopoulos A , RcEven S . Implications of applying methodological shortcuts to expedite systematic reviews: three case studies using systematic reviews from agri-food public health . Res Synth Methods . 2016 ; 7 : 433 - 46 .
18. Ng L , Pitt V , Huckvale K , Clavisi O , Turner T , Gruen R , Elliott J . Title and Abstract Screening and Evaluation in Systematic Reviews (TASER): a pilot randomised controlled trial of title and abstract screening by medical students . Syst Rev . 2014 ; 3 : e1 - 8 .
19. Wallace B , Small K , Brodley C , Lau J , Schmid C , Bertram L , Lill C , Cohen J , Trikalinos T. Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining . Genet Med . 2012 ; 14 : 663 - 9 .
20. Rathbone J , Hoffmann T , Glasziou P . Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers . Syst Rev . 2015 ; 4 : 80 .
21. Wallace B , Small K , Brodley C , Lau J , Trikalinos T. Deploying an interactive machine learning system in an evidence-based practice center . Proc 2nd ACM SIGHIT Symp Int Heal informatics . 2012 .