Sustainment of diverse evidence-informed practices disseminated in the Veterans Health Administration (VHA): initial development and piloting of a pragmatic survey tool

Reardon et al. Implementation Science Communications (2023) 4:6 https://doi.org/10.1186/s43058-022-00386-z Implementation Science Communications Open Access RESEARCH Sustainment of diverse evidence‑informed practices disseminated in the Veterans Health Administration (VHA): initial development and piloting of a pragmatic survey tool Caitlin M. Reardon1* , Laura Damschroder1, Marilla A. Opra Widerquist1, Maria Arasim1, George L. Jackson2,3,4,5, Brandolyn White2, Sarah L. Cutrona6,7,8, Gemmae M. Fix6,9, Allen L. Gifford6,9,10, Kathryn DeLaughter6,7, Heather A. King2,3,4, Blake Henderson11, Ryan Vega11 and Andrea L. Nevedal1 Abstract Background There are challenges associated with measuring sustainment of evidence-informed practices (EIPs). First, the terms sustainability and sustainment are often falsely conflated: sustainability assesses the likelihood of an EIP being in use in the future while sustainment assesses the extent to which an EIP is (or is not) in use. Second, grant funding often ends before sustainment can be assessed. The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) program is one of few large-scale models of diffusion; it seeks to identify and disseminate practices across the VHA system. The DoE sponsors “Shark Tank” competitions, in which leaders bid on the opportunity to implement a practice with approximately 6 months of implementation support. As part of an ongoing evaluation of the DoE, we sought to develop and pilot a pragmatic survey tool to assess sustainment of DoE practices. Methods In June 2020, surveys were sent to 64 facilities that were part of the DoE evaluation. We began analysis by comparing alignment of quantitative and qualitative responses; some facility representatives reported in the open-text box of the survey that their practice was on a temporary hold due to COVID-19 but answered the primary outcome question differently. As a result, the team reclassified the primary outcome of these facilities to Sustained: Temporary COVID-Hold. Following this reclassification, the number and percent of facilities in each category was calculated. We used directed content analysis, guided by the Consolidated Framework for Implementation Research (CFIR), to analyze open-text box responses. Results A representative from forty-one facilities (64%) completed the survey. Among responding facilities, 29/41 sustained their practice, 1/41 partially sustained their practice, 8/41 had not sustained their practice, and 3/41 had never implemented their practice. Sustainment rates increased between Cohorts 1–4. Conclusions The initial development and piloting of our pragmatic survey allowed us to assess sustainment of DoE practices. Planned updates to the survey will enable flexibility in assessing sustainment and its determinants at any phase after adoption. This assessment approach can flex with the longitudinal and dynamic nature of sustainment, *Correspondence: Caitlin M. Reardon Full list of author information is available at the end of the article This is a U.S. Government work and not under copyright protection in the US; foreign copyright protection may apply 2023. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativeco mmons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/ zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Reardon et al. Implementation Science Communications (2023) 4:6 Page 2 of 14 including capturing nuances in outcomes when practices are on a temporary hold. If additional piloting illustrates the survey is useful, we plan to assess the reliability and validity of this measure for broader use in the field. Keywords Sustainability, Sustainment, Measurement, Outcomes, Model of diffusion, Consolidated Framework for Implementation Research (CFIR) Contributions to the literature • The terms sustainability and sustainment are used interchangeably in the literature; this paper provides clarity in defining and differentiating these terms. • Sustainment determinants and outcomes are often conflated in the literature; this paper illustrates that many sustainment determinants are inaccurately described as outcomes. • Sustainment is dynamic; this paper provides an approach to better capture nuance in sustainment outcomes when practices are on a temporary hold. • A high rate of practice sustainment among responding facilities suggests that the VHA DoE is a promising large-scale model of diffusion. Background Evaluating sustainment of evidence‑informed practices is challenging There is growing interest in sustainment of evidenceinformed practices (EIPs) [1, 2]; however, the literature on how to best measure sustainment over time is still developing [3]. Understanding sustainment of EIPs is challenging, which Birken et al. suggest is due to a lack of conceptual clarity and methodological challenges [4]. First, the terms sustainability and sustainment are often used interchangeably [4]. While these terms are related, there are important distinctions. Sustainability assesses the likelihood of an EIP being in use at a future point in time; it is measured by assessing contextual determinants (i.e., factors which decisively affect the nature or outcome of something) [5]. For example, the EIP is perceived to have low sustainability due to inadequate funding or lack of priority. Operationally, the goal is to determine whether the conditions indicative of sustaining EIPs are in place, and if not, to guide efforts to put such conditions into place [6, 7]. In contrast, sustainment assesses the extent to which an EIP is (or is not) in use after a specific period of time after initial implementation; for example, the RE-AIM Framework specifies that the sustainment period begins at least 6 months after initial implementation is completed [8]. Sustainment is measured by assessing outcomes (i.e., the way a thing turns out; a consequence), e.g., the EIP is in use/not in use. Operationally, the goal is to determine if EIPs are still in place following the end of implementation support [9]. Distinguishing between sustainability and sustainment w (...truncated)