Clinimetric properties of the electronic Pain Assessment Tool (ePAT) for aged-care residents with moderate to severe dementia
Journal of Pain Research
Clinimetric properties of the electronic Pain Assessment Tool (ePAT) for aged-care residents with moderate to severe dementia
Kreshnik Hoti 0 1
Mustafa Atee 1
Jeffer y D Hughes 1
0 Division of Pharmacy, Faculty of Medicine, University of Prishtina , Pristina, Kosovo
1 School of Pharmacy, Curtin University , Perth , Australia
8 1 0 2 - l u J - 2 1 n o 2 7 1 . 3 6 1 . 7 3 . 4 5 y b / m o c . s rsvpee l.yon PowerdbyTCPDF(ww.tcpdf.org) Purpose: Accurate pain assessment is critical to detect pain and facilitate effective pain management in dementia patients. The electronic Pain Assessment Tool (ePAT) is a point-of-care solution that uses automated facial analysis in conjunction with other clinical indicators to evaluate the presence and intensity of pain in patients with dementia. This study aimed to examine clinimetric properties (clinical utility and predictive validity) of the ePAT in this population group. Methods: Data were extracted from a prospective validation (observational) study of the ePAT in dementia patients who were ≥65 years of age, living in a facility for ≥3 months, and had Psychogeriatric Assessment Scales - cognitive scores ≥10. The study was conducted in two residential aged-care facilities in Perth, Western Australia, where residents were sampled using purposive convenience strategy. Predictive validity was measured using accuracy statistics (sensitivity, specificity, positive predictive value, and negative predictive value). Positive and negative clinical utility index (CUI) scores were calculated using Mitchell's formula. Calculations were based on comparison with the Abbey Pain Scale, which was used as a criterion reference. Results: A total of 400 paired pain assessments for 34 residents (mean age 85.5±6.3 years, range 68.0-93.2 years) with moderate-severe dementia (Psychogeriatric Assessment Scales - cognitive score 11-21) were included in the analysis. Of those, 303 episodes were classified as pain by the ePAT based on a cutoff score of 7. Unadjusted prevalence findings were sensitivity 96.1% (95% CI 93.9%-98.3%), specificity 91.4% (95% CI 85.7%-97.1%), accuracy 95.0% (95% CI 92.9%-97.1%), positive predictive value 97.4% (95% CI 95.6%-99.2%), negative predictive value 87.6% (95% CI 81.1%-94.2%), CUI+ 0.936 (95% CI 0.911-0.960), CUI- 0.801 (95% CI 0.748-0.854). Conclusion: The clinimetric properties demonstrated were excellent, thus supporting the clinical usefulness of the ePAT when identifying pain in patients with moderate-severe dementia.
with dementia and constitute a significant burden for their
families and carers.1–3 In order to improve the current
situation, the electronic Pain Assessment Tool [ePAT; also known
as PainChek] was developed, which is a point-of-care smart
device-enabled application that uses automated facial (video)
analysis in conjunction with clinical indicators to evaluate
the presence and intensity of pain in those with
communication difficulties, including dementia.4 This application was
021 designed to improve the objectivity and accuracy of assessing
l--Ju pain in patients with dementia, ultimately leading to effective
21n pain management for this disadvantaged group.
o2 In July 2017, the ePAT application was approved as a
.173 Class I medical device by the Therapeutic Goods
Administra.671 tion in Australia and received the CE mark in Europe “. . . to
.354 assess and monitor pain in people who cannot verbalise such
/yb as people with dementia or communication difficulties.”5
com To date, the application has been validated in a total of 74
.s residents aged 60–98 years across five residential aged-care
rsvpee l.yon facilities in Western Australia.4,6 In this setting, it has shown
.dow lsue strong psychometric properties, including concurrent validity,
/ww ano discriminant validity, and interrater reliability.4,6
:/tsp rspe Clinimetric properties, such as predictive validity and
th ro clinical utility (CU), are also important criteria in
evaluatfrom F ing new pain-assessment tools, because they determine
dde the usefulness and applicability of the tool in the clinical
lano setting.7 Literature data indicate that a large number of
odw psychometric evaluation studies of pain-assessment tools
rcha have neglected this aspect.2,8,9 Zwakhalen et al point out that
see further research should test validity, reliability, and CU of
inR existing pain-assessment tools with the view of improving
faP them.8 Furthermore, a more recent systematic review by
lna Lichtner et al concluded that no single pain-assessment tool
rJou can be currently recommended in patients with dementia on
the basis of lack of evidence concerning validity and CU to
assess pain in patients with cognitive impairment.2
The CU of a pain-assessment tool is an essential
clinimetric parameter that goes beyond the analytical, technical, or
even diagnostic accuracy performance of the tool.10 Testing
CU in fact provides more insight into potential health benefits
and outcomes,10 especially in comparison to existing options,
in this case the Abbey Pain Scale (APS), which is currently a
widely used (ie, silver) standard for assessing pain in patients
with dementia in Australia and other countries.11,12
Furthermore, CU provides important information in relation to how
useful the tool is in assisting the decision-making process
about the patients, eg, administering an analgesic drug when
a patient is in pain.13
submit your manuscript | www.dovepress.com
In addition to our aim of examining CU, the focus of this
paper is also to assess the predictive validity of the ePAT. This
is relevant, given the significance of data in evidencing the
potential ability of the ePAT in predicting criterion-reference
measures of the APS, which are a key reason for assessing
this important clinimetric property.13–15 Specifically, this study
aimed to analyze the predictive validity of the ePAT with a
view toward discriminating and comparing between pain on
movement and at rest with reference to the APS.
This research was approved by the Human Research Ethics
Committee of Curtin University, Western Australia (HREC:
HR10/2014) and Mercy Health, Victoria (R15/50AC) in
Australia. The study was also registered with the
Therapeutic Goods Administration under the Clinical Trial
Notification scheme (CT-2016-CTN-04886-1 v1). Proxy (third
party) written informed consent was provided by legal
representative(s) of participants involved, due to the latter’s
impaired cognitive capacity to provide their own consent.
No additional data were collected for the current study, as
all data were sourced from a previously published study.6
Therefore, there was no burden on staff or residents. Data
were deidentified to ensure confidentiality.
Study design, setting, and inclusion
Data were extracted from a previous validation (observational)
study of the ePAT, which was conducted in two residential
aged-care facilities in Perth, Western Australia over 10 weeks
between January and April 2017.6 Pain assessments were
done twice (at rest and postmovement) in pairs by ePAT rater
and APS raters (nursing staff of the residential aged-care
facilities). Raters performed assessments independently, and
were blinded in terms of one another’s assessment and pain
therapies received by the residents. All pain assessments
were performed inside the facilities between 1-4pm while the
residents were receiving their standard clinical care. This was
either done during sitting, or recumbent positions to reflect rest
(non-nociceptive) conditions, or during walking or transfer as
examples of movement (nociceptive) conditions. Residents’
cognitive level was assessed prior to the study using
Psychogeriatric Assessment Scales – cognitive (PAS-Cog), which
has been validated in people with dementia.16 PAS-Cog scores
categorize the degree of cognitive impairment according to
increments of 0–3 = minimal, 4–9 = mild, 10–15 = moderate,
and 16–21 = severe. Cognitive data were extracted from the
electronic medical records of the residents.6
Residents were selected if they met the inclusion criteria:
≥65 years of age, living in facility for ≥3 months, PAS-Cog
score ≥10, had a diagnosis of dementia by a geriatrician, and
a medical history or presenting complaint(s) linked to painful
condition(s), such as arthritis. Residents were excluded if they
were medically unwell or unable to display intact facial
expressions. Further details of the methodology employed in this
study, including inclusion and exclusion criteria of residents
and raters, have also been discussed in a previous publication.6
Descriptive statistics (eg, mean, median) were reported for
pain scores, while inferential statistics (eg, CI) were used to
determine the level of significance. The receiver-operating
characteristic (ROC) curve was used to determine cutoff
scores for the presence of pain, which were classified in
a binary format (yes/no). The ROC curve is a graphical
representation of the sensitivity versus 1 – specificity for
a dichotomous variable.17,18 The curve is a method of
estimating the best balance between sensitivity and specificity
using multiple measurements, and enables the selection of an
optimal threshold value (cutoff point). Besides cutoff points,
ROC curve analysis can be used to convert continuous or
ordinal variables into dichotomous outcomes.17,18 To derive
the maximum value for these cutoff points, Youden’s index
(sensitivity + specificity – 1) was also used.17 In our study,
ePAT was the tool under investigation against the standard
instrument, ie, the APS. Cross-tab calculations were made to
enable univariate percentage agreement between ePAT and
APS of the presence or absence of pain.
Predictive validity was measured using accuracy
statistics (sensitivity, specificity, positive predictive value,
negative predictive value, and positive and negative likelihood
ratios, respectively). These values were calculated twice:
before and after prevalence adjustment. Positive (+) and
negative (-) CU index (CUI) values were calculated using
Mitchell’s formula.19 In this study, we used the STARD
(Standards for Reporting of Diagnostic Accuracy Studies)
guidelines to report the clinimetric findings.20 Calculations
were based on comparison with the APS, which was used
as a criterion reference in our study. Presence of pain was
defined as per published scores of the APS (≥3) or ePAT
(≥7), with absence of pain defined as APS score of 0–2 or
ePAT score of 0–6.4,11
All analyses were done using Microsoft Excel 2013 for
Windows 7 Enterprise (Microsoft Corporation, Redmond,
WA, USA), Clinical Utility Index Calculator,19 and MedCalc
statistical software (version 17.4 for Windows; MedCalc
Software, Ostend, Belgium).
A total of 34 residents aged 68.0–93.2 years were included in
our sample, with a mean of 85.5±6.3 years.6 The sample
comprised 20 females (58.8%) and 14 males (41.2%), of whom
35.3% had Alzheimer’s dementia and 44.1% an
unspecified type of dementia. With the exception of one resident,
all (n=33, 97.1%) were Caucasian. A total of 27 residents
(79.4%) had a diagnosis of severe dementia based on
PASCog scores (mean 19.7±2.5, range 11–21). The vast majority
of the sample were either nonambulant (n=14, 41.2%) or
ambulant with assistance (n=19, 55.9%). A full description
of the sample data was published in a previous study.6
A total of 400 paired pain assessments for 34 residents
were included in the analysis.6 Of those, 303 episodes were
classified as pain by the ePAT based on a cutoff score of 7
(Table 1). Values ≤6 were defined as no pain. On the other
hand, 307 were deemed pain by APS based on a cutoff score
of 3, whereby scores ≤2 were considered no pain (Table 2).
Mean ePAT pain scores were significantly greater with
movement (11.44±3.54, P<0.0001) than rest (8.33±3.34).
During assessments, types of physical activities undertaken
by residents were varied, and ranged from sitting to walking.6
Predictive validity was calculated using sensitivity,
specificity, and accuracy before and after prevalence
adjustment. In the context of this validation study (n=400 paired
assessments), sensitivity represents the true positive rate
of detecting pain by the ePAT when pain actually exists, as
indicated by the APS. Specificity denotes the true-negative
rate of detecting pain (ie, absence of pain) by the ePAT when
no pain is identified according to the APS. For the ePAT, the
presence of pain was defined as a score of ≥7 on the final
scale.4 According to published scores of the APS, a total
score ≥3 refers to presence of pain, while a score ≤2 refers
to the absence of pain.11
CUI values were calculated using Mitchell’s formula.21 CUI+
is the CU of the test for case-finding (ie, confirmation), which
is calculated as a product of sensitivity and positive predictive
value, ie, CUI+ = 0.936 (95% CI 0.911–0.960).19 CUI– is the
CU of the test for screening (ie, ruling out pain), which is
calculated as a product of specificity and negative predictive
value, ie, CUI– = 0.801 (95% CI 0.748–0.854).21 The overall
ePAT value for combined screening and case finding was 95%
(ie, CUI = 0.95). This meant the ePAT has excellent utility.21
Receiver-operating characteristic (ROC) curve
When the prevalence of pain adjusted to 50% (from 76.8%
provided in Table 4), the sensitivity and specificity data were
Clinimetric parameter Formula
Sensitivity (TP/[TP + FN]) ¥100
Specificity (TN/[TN + FP]) ¥100
Positive likelihood ratio Sensitivity/100 – specificity
Negative likelihood ratio 100 – specificity/sensitivity
Positive predictive value (TP/[TP + FP]) ¥100
Negative predictive value (TN/[TN +FN)) ¥100 Pain prevalence ([TP + FN]/[TP + TN + FP + FN]) ¥100 Accuracy ([TP + TN]/[TP + TN + FP + FN]) ¥100
Note: All values approximated to closest decimal point.
Abbreviations: TP, true positive; FP, false positive; FN, false negative; TN, true negative.
slightly varied. A choice of 50% prevalence was made to
reflect the current data available in the literature.22 The
optimal cutoff score where there was a balance of sensitivity and
specificity was 7. This was confirmed by the
area-under-thecurve value of 0.98 and P<0.0001. A Youden’s index value
of close to 1 (ie, 0.88) pointed to minimal FPs and FNs in
the data set. The closeness of the curve to the left corner of
the graph illustrates ePAT’s high sensitivity and specificity
in distinguishing between pain/no pain cutoffs. Results from
the ROC curve used to analyze cutoff scores are presented
in more detail in Table 5 and Figure 1.
This study evaluated the clinimetric properties (predictive
validity and CU) of the ePAT in residents with moderate–
severe dementia. For the ePAT application, this provides more
evidence to how clinically meaningful the test results are
in relation to pain detection in the target-population group.
Other psychometric properties, such as concurrent validity,
discriminant validity, and interrater reliability, have been
As reported by Herr et al9 and later by Lichtner et al,2
currently the evidence for existing tools in regard to their validity
and CU is limited and unclear. Due to these psychometric
and clinimetric limitations, clinician and carer guidance on
effective pain assessment is currently at best compromised,
leading to gaps in relation to informed treatment options
and care plans.2 Some studies that have previously reported
predictive validity of pain-assessment tools have been marred
by failure actually to report the data, whereas others had
significant scoring differences (pre- and postintervention).2 The
situation is similar in regard to the CU of pain-assessment
tools. In addition to clinical data being completely or
substantially absent for some tools, conflicting data have been
reported for other tools.2 Furthermore, CU dimensions, such
as the cutoff scores (eg, pain vs no-pain cutoff described in
the present study) needed for clinical decision making, have
not been reported by many studies evaluating the currently
available pain-assessment tools.2 For example, the total score
of the Pain Assessment in Advanced Dementia Scale can be
0–10 points. However, there is no evidence of pain-intensity
scoring.23 The original Pain Assessment Checklist for Seniors
with Limited Ability to Communicate (PACSLAC) study did
not report a specific cutoff score to determine the presence
of pain. Rather, an increased PACSLAC score suggests that
an increase in pain is likely, while a lower score suggests
that pain has reduced.24 Later, it was found that PACSLAC
scores greater than 12 (of 60) are indicative of high pain
intensity, whereas scores of 0–5 represent usual pain.24 In
the current study, the fact that the area under the curve was
0.98 (P<0.0001) indicated the ePAT’s strong ability to
distinguish between cutoff scores for pain. Our current analysis
confirmed a cutoff score of 7 for the presence of pain, also
previously reported by Atee et al.4 This lends further support
to our previous findings.4
This is the first study to demonstrate that the ePAT is a
useful and valid instrument to assess pain in patients with
moderate–severe dementia. This is an important step in
further evidencing this instrument in the context of
better assessment and management of pain in patients with
dementia, given their compromised ability to communicate
210 pain, which results in underrecognition and undertreatment
l--Ju of their pain.1
n21 High sensitivity and specificity values for the ePAT in
2o identifying pain yield strong support to its predictive validity
.317 and responsiveness to change. Even after disease-adjusted
.671 prevalence analysis had been performed, sensitivity and
.543 specificity values changed only slightly. This validates our
/yb approach to the conceptual foundation of the tool, previously
com reported by Atee et al.4 Pain assessments were performed
.s during rest and movement, and the resultant scores reflected
rsvpee l.yon the change in timing, regardless of the order of testing.6 The
.dow lsue overall CUI value of the ePAT reported in this study was
/ww ano excellent (0.95) as indicated by Mitchell’s analysis. This
:/tsp rspe suggests that the ePAT is a useful tool to assist in the
clinith ro cal decision-making processes related to pain management,
from F including informing clinicians on specific actions, such as
ded analgesic administration, emphasized as a necessary property
loan by van Herk et al13 when observing pain-assessment tools in
dow patients with cognitive impairment. Currently, there are no
rcha published studies reporting Mitchell’s CUI of pain
assesssee ment tools in dementia. Other clinical tools where Mitchell’s
inR CUI has been used include the Patient Health Questionnaire
faP (PHQ9 and PHQ2) for depression in primary care, and the
lona Cornell Scale for Depression in Dementia.25,26 Our findings
rJou presented much better accuracy data than those tools.
Strengths and limitations
The present study has several points of strength. Statistical
analyses were methodical and covered a number of variables.
For example, Mitchell’s index was used for the first time (as far
as we know) to investigate the CU of a pain-assessment tool for
people with dementia. The data were gleaned from participants
with varying types of dementia and pain conditions, and this also
addresses recommendations by systematic reviews.2,9 Full
medical histories of residents were accessible to raters, which assisted
in informing the clinical picture when doing pain assessments.
Among the limitations of our study (which was noted
previously),6 it is important to highlight that the comparison was
made with reference to the APS. Despite its frequent use in
Australia, the APS is not a global gold standard. The original
submit your manuscript | www.dovepress.com
sample was relatively small, and contained disproportionate
sex representation. Further, data obtained were limited to
residential aged-care settings. Thus, the findings need to be
interpreted with caution before applying them to other settings, eg,
hospital. Also, there was no physical examination conducted
for the sake of the study to identify the potential source of
pain. In fact, pain data were based on medical history, where
some aspects might not be a true representation of the current
clinical status of residents. However, in the case of the ePAT,
medical history was weighed as only 1 of the 42 points of the
total scale.4 Therefore, it had a negligible effect on our data.
CU encompasses a wide range of effects that tests or
tools can have on the patient.10 In this regard, it is worth
emphasizing that a recent study conducted by van Kooten
et al suggested that improving communication of results to
physicians can lead to improvement in pain management in
long-term care patients with dementia.27 Although the present
study did not evaluate this aspect, it is worth emphasizing
that the ePAT tool is the only pain-assessment tool available
that uses automation and is able to document and manage
clinical information electronically.4 This is thus expected to
have positive effects on pain detection and management and
communication to physicians. This aspect needs to be further
researched in future as an additional potential indicator of the
ePAT’s clinical usefulness. CU also has a significant value
for implementing tools in clinical practice. More emphasis
and research on the CU of other pain-assessment tools are
needed, because these parameters provide data on
practicalities, which may improve the uptake and utilization of tools
in clinical settings. Use of an electronic pain-assessment
tool, which eases the process of documentation and pain
tracking over time, has the potential to positively impact the
tool’s CU through facilitating information on the temporality
(ie, trends and patterns) of pain. In this regard, it is worth
highlighting that the temporality of pain was regarded as
“most useful to the assessment of pain” by Lichtner et al.28
This potential strength of the ePAT application needs to be
researched further in future.
The findings reveal strong clinimetric properties of the ePAT.
We have demonstrated that both its predictive validity and
CU are excellent, giving further evidence to the quality of
pain assessment attained by the ePAT.
The authors express their gratitude to everyone involved
in the study, including aged-care staff, residents, and their
families. The authors would like to acknowledge the
contribution of an Australian Government Research Training Program
Scholarship in supporting this research. The original research
that led to the development of the PainChek tool is part of a
PhD project that was also supported by the Dementia
Australia Research Foundation (DARF) through grant funding
and a stipend scholarship. The content of the article is solely
the responsibility of the authors, and does not necessarily
represent the official views of DARF. The project has been
commercialized initially through a startup company (ePAT
Pty Ltd), and since October 2016 the Australian Share
Securities (ASX)-listed EPAT Technologies Ltd (now known as
PainChek Ltd). This research was also sponsored by EPAT
KH, MA, and JDH conceived the idea and designed the study.
KH and MA performed the literature review and drafted
the manuscript. KH, MA, and JDH revised the manuscript.
MA conducted and interpreted the statistical analyses. All
authors contributed to and approved the final version of the
KH, MA, and JDH are shareholders in PainChek Ltd
(formerly known as EPAT Technologies), which is
commercializing the ePAT instrument as PainChekTM. They also have
a patent application titled “A pain assessment method and
system; PCT/AU2015/000501”, which has been under the
national phase examination since February 2, 2017. KH is
employed as a consultant by PainChek Ltd while serving as
an assistant professor at the University of Prishtina and an
adjunct senior lecturer at the School of Pharmacy, Curtin
University. MA is a research scientist for PainChek Ltd
while serving as a research fellow and PhD candidate with
the School of Pharmacy, Curtin University. JDH is employed
as chief scientific officer of PainChek Ltd while serving as
a professor in the School of Pharmacy, Curtin University.
submit your manuscript | www.dovepress.com
Publish your work in this journal
The Journal of Pain Research is an international, peer reviewed, open
access, online journal that welcomes laboratory and clinical findings
in the fields of pain research and the prevention and management
of pain. Original research, reviews, symposium reports,
hypothesis formation and commentaries are all considered for publication.
1. Herr K , Coyne PJ , McCaffery M , Manworren R , Merkel S . Pain assessment in the patient unable to self-report: position statement with clinical practice recommendations . Pain Manag Nurs . 2011 ; 12 ( 4 ): 230 - 250 .
2. Lichtner V , Dowding D , Esterhuizen P , et al. Pain assessment for people with dementia: a systematic review of systematic reviews of pain assessment tools . BMC Geriatr . 2014 ; 14 : 138 .
3. Shega JW , Hougham GW , Stocking CB , Cox-Hayley D , Sachs GA . Pain in community-dwelling persons with dementia: frequency, intensity, and congruence between patient and caregiver report . J Pain Symptom Manage . 2004 ; 28 ( 6 ): 585 - 592 .
4. Atee M , Hoti K , Parsons R , Hughes JD . Pain assessment in dementia: evaluation of a point-of-care technological solution . J Alzheimers Dis . 2017 ; 60 ( 1 ): 137 - 150 .
5. Therapeutic Goods Administration eBusiness Services. ePAT Technologies Limited: information system software, application program . 2017 . Available from: https://www.ebs.tga.gov.au/servlet/xmlmillr6?dbid=ebs/ PublicHTML/pdfStore.nsf& docid=CD49D725FBC4131ECA2581610 042344A&agid=(PrintDetailsPublic) & actionid=1. Accessed August 31 , 2017 .
6. Atee M , Hoti K , Hughes JD . Psychometric evaluation of the electronic Pain Assessment Tool (ePAT): an innovative instrument for individuals with moderate to severe dementia . Dement Geriatr Cogn Disord . 2018 ; 44 ( 5-6 ): 256 - 267 .
7. Mokkink LB , Terwee CB , Patrick DL , et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study . Qual Life Res . 2010 ; 19 ( 4 ): 539 - 549 .
8. Zwakhalen SM , Hamers JP , Abu-Saad HH , Berger MP . Pain in elderly people with severe dementia: a systematic review of behavioural pain assessment tools . BMC Geriatr . 2006 ; 6 : 3 .
9. Herr K , Bjoro K , Decker S . Tools for assessment of pain in nonverbal older adults with dementia: a state-of-the-science review . J Pain Symptom Manage . 2006 ; 31 ( 2 ): 170 - 192 .
10. Bossuyt PM , Reitsma JB , Linnet K , Moons KG . Beyond diagnostic accuracy: the clinical utility of diagnostic tests . Clin Chem . 2012 ; 58 ( 12 ): 1636 - 1643 .
11. Abbey J , Piller N , de Bellis A , et al. The Abbey Pain Scale: a 1-minute numerical indicator for people with end-stage dementia . Int J Palliat Nurs . 2004 ; 10 ( 1 ): 6 - 13 .
12. Gregersen M , Melin AS , Nygaard IS , Nielsen CH , BeedholmEbsen M. Reliability of the Danish Abbey Pain Scale in severely demented and non-communicative older patients . Int J Palliat Nurs . 2016 ; 22 ( 10 ): 482 - 488 .
13. van Herk R , van Dijk M , Baar FP , Tibboel D , de Wit R. Observation scales for pain assessment in older adults with cognitive impairments or communication difficulties . Nurs Res . 2007 ; 56 ( 1 ): 34 - 43 .
14. Salkind NJ . Encyclopedia of Research Design. Thousand Oaks (CA): SAGE; 2010 .
15. Cronbach LJ , Meehl PE . Construct validity for psychological tests . Psychol Bull . 1955 ; 52 ( 4 ): 281 - 302 .
16. Jorm AF , Mackinnon AJ , Henderson AS , et al. The Psychogeriatric Assessment Scales: a multi-dimensional alternative to categorical diagnoses of dementia and depression in the elderly . Psychol Med . 1995 ; 25 ( 3 ): 447 - 460 .
17. Hanley JA , Mcneil BJ . The meaning and use of the area under a receiver operating characteristic (ROC) curve . Radiology. 1982 ; 143 ( 1 ): 29 - 36 .
18. Greiner M , Pfeiffer D , Smith RD . Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests . Prev Vet Med . 2000 ; 45 ( 1-2 ): 23 - 41 .
19. Mitchell AJ. Sensitivity × PPV is a recognized test called the clinical utility index (CUI+) . Eur J Epidemiol . 2011 ; 26 ( 3 ): 251 - 252 .
20. Bossuyt PM , Reitsma JB , Bruns DE , et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration . Clin Chem . 2003 ; 49 ( 1 ): 7 - 18 .
21. Mitchell AJ. How do we know when a screening test is clinically useful ? In: Mitchell AJ, Coyne JC , editors. Screening for Depression in Clinical Practice: An Evidence-Based Guide . Oxford: Oxford University Press; 2010 : 99 - 112 .
22. Achterberg WP , Pieper MJ , van Dalen-Kok AH , et al. Pain management in patients with dementia . Clin Interv Aging . 2013 ; 8 : 1471 - 1482 .
23. Warden V , Hurley AC , Volicer L . Development and psychometric evaluation of the Pain Assessment in Advanced Dementia (PAINAD) scale . J Am Med Dir Assoc . 2003 ; 4 ( 1 ): 9 - 15 .
24. Fuchs-Lacelle S , Hadjistavropoulos T , Lix L . Pain assessment as intervention: a study of older adults with severe dementia . Clin J Pain . 2008 ; 24 ( 8 ): 697 - 707 .
25. Mitchell AJ , Yadegarfar M , Gill J , Stubbs B . Case finding and screening clinical utility of the Patient Health Questionnaire (PHQ-9 and PHQ-2) for depression in primary care: a diagnostic meta-analysis of 40 studies . BJPsych Open . 2016 ; 2 ( 2 ): 127 - 138 .
26. Hancock P , Larner AJ . Cornell Scale for Depression in Dementia: clinical utility in a memory clinic . Int J Psychiatry Clin . 2015 ; 19 ( 1 ): 71 - 74 .
27. van Kooten J , Smalbrugge M , van der Wouden J , Stek ML , Hertogh C . Evaluation of a pain assessment procedure in long-term care residents with pain and dementia . J Pain Symptom Manage . 2017 ; 54 ( 5 ): 727 - 731 .
28. Lichtner V , Dowding D , Closs SJ . The relative meaning of absolute numbers: the case of pain intensity scores as decision support systems for pain management of patients with dementia . BMC Med Inform Decis Mak . 2015 ; 15 : 111 .