Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping
Powell et al. Implementation Science
Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping
Byron J. Powell 0
Cameo F. Stanick 2
Heather M. Halko 1
Caitlin N. Dorsey 6
Bryan J. Weiner 5
Melanie A. Barwick 4
Laura J. Damschroder 3
Michel Wensing 8
Luke Wolfenden 7
Cara C. Lewis 6
0 Department of Health Policy and Management, Gillings School of Global Public Health, University of North Carolina at Chapel Hill , 1105C McGavran-Greenberg Hall, 135 Dauer Drive, Campus Box 7411, Chapel Hill, NC 27599 , USA
1 Department of Psychology, University of Montana , Missoula, MT , USA
2 Hathaway-Sycamores Child and Family Services , Pasadena, CA , USA
3 VA Ann Arbor Center for Clinical Management Research and Diabetes QUERI, VA Ann Arbor Healthcare System , Ann Arbor, MI , USA
4 Hospital for Sick Children, University of Toronto , Toronto, ON , Canada
5 Department of Global Health and Department of Health Services, University of Washington , Seattle, WA , USA
6 Kaiser Permanente Washington Health Research Institute , Seattle, WA , USA
7 School of Medicine and Public Health, The University of Newcastle , Callaghan, NSW , Australia
8 Department of General Practice and Health Services Research, University Hospital Heidelberg , Im Neuenheimer Feld, Heidelberg , Germany
Background: Advancing implementation research and practice requires valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes. However, researchers and implementation stakeholders are unlikely to use measures if they are not also pragmatic. The purpose of this study was to establish a stakeholder-driven conceptualization of the domains that comprise the pragmatic measure construct. It built upon a systematic review of the literature and semi-structured stakeholder interviews that generated 47 criteria for pragmatic measures, and aimed to further refine that set of criteria by identifying conceptually distinct categories of the pragmatic measure construct and providing quantitative ratings of the criteria's clarity and importance. Methods: Twenty-four stakeholders with expertise in implementation practice completed a concept mapping activity wherein they organized the initial list of 47 criteria into conceptually distinct categories and rated their clarity and importance. Multidimensional scaling, hierarchical cluster analysis, and descriptive statistics were used to analyze the data. Findings: The 47 criteria were meaningfully grouped into four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful. Average ratings of clarity and importance at the category and individual criteria level will be presented. Conclusions: This study advances the field of implementation science and practice by providing clear and conceptually distinct domains of the pragmatic measure construct. Next steps will include a Delphi process to develop consensus on the most important criteria and the development of quantifiable pragmatic rating criteria that can be used to assess measures.
Background
Bridging the gap between research and practice by
advancing implementation science will require valid and
reliable measures of implementation determinants,
mechanisms, processes, strategies, and outcomes [
1
].
However, implementation stakeholders (i.e., researchers
and practice-based implementers) are unlikely to use
measures solely on the basis of strong psychometric
properties; they also need to be pragmatic [
2, 3
]. For
example, a measure that is psychometrically sound, but is
time-consuming or expensive to administer, is unlikely
to be used. There is currently no consensus about what
constitutes a pragmatic measure. Glasgow and Riley [2]
advanced the conceptualization of the pragmatic
measure construct by suggesting two types of criteria:
required (important to stakeholders, low burden for
respondents and staff, actionable, and sensitive to
change) and recommended (broadly applicable, used for
benchmarking, unlikely to cause harm, psychometrically
strong, and related to theory or model). However, these
recommendations may be limited as they were not
developed through a systematic literature review, were
not informed by relevant stakeholders, and focused on
clinical measures. Key aspects of the pragmatic measure
construct may have been overlooked.
The “Advancing Implementation Science through
Measure Development and Evaluation” study [
3
] aims to
(1) establish a stakeholder-driven operationalization of
pragmatic measures and develop reliable, valid rating
criteria for assessing this construct; (2) develop reliable,
valid, and pragmatic measures of three different
implementation outcomes [
4
] (acceptability, appropriateness,
and feasibility) [
5
]; and (3) identify measures that
demonstrate both psychometric and pragmatic strength. This
article details our Aim 1 efforts to establish a
stakeholder-driven conceptualization of (...truncated)