Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping

Implementation Science, Oct 2017

Advancing implementation research and practice requires valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes. However, researchers and implementation stakeholders are unlikely to use measures if they are not also pragmatic. The purpose of this study was to establish a stakeholder-driven conceptualization of the domains that comprise the pragmatic measure construct. It built upon a systematic review of the literature and semi-structured stakeholder interviews that generated 47 criteria for pragmatic measures, and aimed to further refine that set of criteria by identifying conceptually distinct categories of the pragmatic measure construct and providing quantitative ratings of the criteria’s clarity and importance. Twenty-four stakeholders with expertise in implementation practice completed a concept mapping activity wherein they organized the initial list of 47 criteria into conceptually distinct categories and rated their clarity and importance. Multidimensional scaling, hierarchical cluster analysis, and descriptive statistics were used to analyze the data. The 47 criteria were meaningfully grouped into four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful. Average ratings of clarity and importance at the category and individual criteria level will be presented. This study advances the field of implementation science and practice by providing clear and conceptually distinct domains of the pragmatic measure construct. Next steps will include a Delphi process to develop consensus on the most important criteria and the development of quantifiable pragmatic rating criteria that can be used to assess measures.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://implementationscience.biomedcentral.com/track/pdf/10.1186/s13012-017-0649-x?site=implementationscience.biomedcentral.com

Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping

Powell et al. Implementation Science Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping Byron J. Powell 0 Cameo F. Stanick 2 Heather M. Halko 1 Caitlin N. Dorsey 6 Bryan J. Weiner 5 Melanie A. Barwick 4 Laura J. Damschroder 3 Michel Wensing 8 Luke Wolfenden 7 Cara C. Lewis 6 0 Department of Health Policy and Management, Gillings School of Global Public Health, University of North Carolina at Chapel Hill , 1105C McGavran-Greenberg Hall, 135 Dauer Drive, Campus Box 7411, Chapel Hill, NC 27599 , USA 1 Department of Psychology, University of Montana , Missoula, MT , USA 2 Hathaway-Sycamores Child and Family Services , Pasadena, CA , USA 3 VA Ann Arbor Center for Clinical Management Research and Diabetes QUERI, VA Ann Arbor Healthcare System , Ann Arbor, MI , USA 4 Hospital for Sick Children, University of Toronto , Toronto, ON , Canada 5 Department of Global Health and Department of Health Services, University of Washington , Seattle, WA , USA 6 Kaiser Permanente Washington Health Research Institute , Seattle, WA , USA 7 School of Medicine and Public Health, The University of Newcastle , Callaghan, NSW , Australia 8 Department of General Practice and Health Services Research, University Hospital Heidelberg , Im Neuenheimer Feld, Heidelberg , Germany Background: Advancing implementation research and practice requires valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes. However, researchers and implementation stakeholders are unlikely to use measures if they are not also pragmatic. The purpose of this study was to establish a stakeholder-driven conceptualization of the domains that comprise the pragmatic measure construct. It built upon a systematic review of the literature and semi-structured stakeholder interviews that generated 47 criteria for pragmatic measures, and aimed to further refine that set of criteria by identifying conceptually distinct categories of the pragmatic measure construct and providing quantitative ratings of the criteria's clarity and importance. Methods: Twenty-four stakeholders with expertise in implementation practice completed a concept mapping activity wherein they organized the initial list of 47 criteria into conceptually distinct categories and rated their clarity and importance. Multidimensional scaling, hierarchical cluster analysis, and descriptive statistics were used to analyze the data. Findings: The 47 criteria were meaningfully grouped into four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful. Average ratings of clarity and importance at the category and individual criteria level will be presented. Conclusions: This study advances the field of implementation science and practice by providing clear and conceptually distinct domains of the pragmatic measure construct. Next steps will include a Delphi process to develop consensus on the most important criteria and the development of quantifiable pragmatic rating criteria that can be used to assess measures. Background Bridging the gap between research and practice by advancing implementation science will require valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes [ 1 ]. However, implementation stakeholders (i.e., researchers and practice-based implementers) are unlikely to use measures solely on the basis of strong psychometric properties; they also need to be pragmatic [ 2, 3 ]. For example, a measure that is psychometrically sound, but is time-consuming or expensive to administer, is unlikely to be used. There is currently no consensus about what constitutes a pragmatic measure. Glasgow and Riley [2] advanced the conceptualization of the pragmatic measure construct by suggesting two types of criteria: required (important to stakeholders, low burden for respondents and staff, actionable, and sensitive to change) and recommended (broadly applicable, used for benchmarking, unlikely to cause harm, psychometrically strong, and related to theory or model). However, these recommendations may be limited as they were not developed through a systematic literature review, were not informed by relevant stakeholders, and focused on clinical measures. Key aspects of the pragmatic measure construct may have been overlooked. The “Advancing Implementation Science through Measure Development and Evaluation” study [ 3 ] aims to (1) establish a stakeholder-driven operationalization of pragmatic measures and develop reliable, valid rating criteria for assessing this construct; (2) develop reliable, valid, and pragmatic measures of three different implementation outcomes [ 4 ] (acceptability, appropriateness, and feasibility) [ 5 ]; and (3) identify measures that demonstrate both psychometric and pragmatic strength. This article details our Aim 1 efforts to establish a stakeholder-driven conceptualization of (...truncated)


This is a preview of a remote PDF: https://implementationscience.biomedcentral.com/track/pdf/10.1186/s13012-017-0649-x?site=implementationscience.biomedcentral.com

Byron J. Powell, Cameo F. Stanick, Heather M. Halko, Caitlin N. Dorsey, Bryan J. Weiner, Melanie A. Barwick, Laura J. Damschroder, Michel Wensing, Luke Wolfenden, Cara C. Lewis. Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping, Implementation Science, 2017, pp. 118, DOI: 10.1186/s13012-017-0649-x