Testing the 4Rs and 2Ss Multiple Family Group intervention: study protocol for a randomized controlled trial
Acri et al. Trials
Testing the 4Rs and 2Ss Multiple Family Group intervention: study protocol for a randomized controlled trial
Mary Acri 0
Emily Hamovitch 0
Maria Mini 0
Elene Garay 0
Claire Connolly 0
Mary McKay 1
0 McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work , 41 East 11th Street, 7th floor, New York, NY 10003 , USA
1 George Warren Brown School of Social Work, Washington University , 1 Brookings Drive, St. Louis, MO 63130 , USA
Background: Oppositional defiant disorder (ODD) is a major mental health concern and highly prevalent among children living in poverty-impacted communities. Despite that treatments for ODD are among the most effective, few children living in poverty receive these services due to substantial barriers to access, as well as difficulties in the uptake and sustained adoption of evidence-based practices (EBPs) in community settings. The purpose of this study is to examine implementation processes that impact uptake of an evidence-based practice for childhood ODD, and the impact of a Clinic Implementation Team (CIT)-driven structured adaptation to enhance its fit within the public mental health clinic setting. Methods/design: This study, a Hybrid Type II effectiveness-implementation research trial, blends clinical effectiveness and implementation research methods to examine the impact of the 4Rs and 2Ss Multiple Family Group (MFG) intervention, family level mediators of child outcomes, clinic/provider-level mediators of implementation, and the impact of CITs on uptake and long-term utilization of this model. All New York City public outpatient mental health clinics have been invited to participate. A sampling procedure that included randomization at the agency level and a sub-study to examine the impact of clinic choice upon outcomes yielded a distribution of clinics across three study conditions. Quantitative data measuring child outcomes, organizational factors and implementation fidelity will be collected from caregivers and providers at baseline, 8, and 16 weeks from baseline, and 6 months from treatment completion. The expected participation is 134 clinics, 268 providers, and 2688 caregiver/child dyads. We will use mediation analysis with a multi-level Structural Equation Modeling (SEM) (MSEM including family level variables, provider variables, and clinic variables), as well as mediation tests to examine study hypotheses. Discussion: The aim of the study is to generate knowledge about effectiveness and mediating factors in the treatment of ODDs in children in the context of family functioning, and to propose an innovative approach to the adaptation and implementation of new treatment interventions within clinic settings. The proposed CIT adaptation and implementation model has the potential to enhance implementation and sustainability, and ultimately increase the extent to which effective interventions are available and can impact children and families in need of services for serious behavior problems. Trial registration: ClinicalTrials.gov, ID: NCT02715414. Registered on 3 March 2016.
Implementation and sustainability; Child mental health; Oppositional defiant disorder; Family functioning
Background
Oppositional defiant disorder (ODD) is a chronic and
impairing mental health disorder that afflicts between 1
and 21% of the population, and primarily elementary
school-age children, with the median age of onset being
11.6 years [
1
]. Along with conduct disorder and
attention deficit hyperactivity disorder, ODD is part of the
disruptive behavior disorder (DBD) diagnostic group,
which is characterized by severe behavioral problems
including impulsivity, aggression, violence, delinquency,
and criminal acts [
1
].
Family poverty is one of the greatest risk factors for
the onset and perpetuation of DBDs [
2–4
]: studies
indicate that the accumulation of risk factors associated with
poverty, including increased exposure to violence and
traumatic events, devastating impact of substance abuse,
criminal activity, and chronic disease, as well as family
strain have an additive effect, in that the more risk
factors a child is exposed to increases their likelihood of
exhibiting serious behavioral challenges [
1, 5
].
Accordingly, children living in poverty experience DBDs at rates
of up to four times higher than children reared in
families with more financial resources [6].
Treatments for DBDs are among the most well-studied
and efficacious [
7
], yet as many as 80% of children in
poverty-impacted communities do not receive services.
And for those who do manage to initially engage in
behavioral health care, premature attrition from services
approaches 50% [
1, 8
]. Further, the uptake of evidence-based
practices (EBPs) in community-based settings is slow and
beset with challenges. An oft-cited statistic suggests that it
can take as long as two decades before treatments that are
studied in research trials become integrated as part of
standard practice in real-world settings [9] if at all. Thus,
even though children who live in poor communities are at
the highest risk for DBDs, they are the least likely to
receive effective mental health services.
The 4Rs and 2Ss Multiple Family Group (MFG) intervention
The 4Rs and 2Ss Multiple Family Group (4Rs 2Ss MFG)
for Strengthening Families Program is an
evidencebased, manualized, family based treatment for families of
children with disruptive behaviors [
10–15
]. Recently
listed on the Substance Abuse and Mental Health
Services Administration (SAMHS) National Registry of
Evidence-based Programs and Practices [16], this model is
associated with high attendance rates, improvement in
behavior problems, caregiver depression and stress among
predominantly low-income, minority families [
14
].
behavioral parent training [
12
]. It was developed using a
common elements approach [
10
], which entails
identifying techniques and procedures that are common to
already existing evidence-based protocols for specific
problem areas [
17
]. As such, the 4Rs and 2Ss MFG
integrates family processes and parenting skills linked to
conduct problems from the empirical academic
literature [
10, 14
]. The targeted skills and processes are
referred to in the curriculum as the 4Rs (Rules,
Responsibility, Relationships, and Respectful Communication)
and 2Ss (Stress and Social support) [14]. Additionally,
families of youth with DBDs and mental health providers
assisted in the model’s development specific to
enhancing cultural and contextual relevance of the
intervention content, as well as engagement [
10
].
Implementation processes
Although positive outcomes from the 4Rs and 2Ss MFG
were found in prior studies, significant implementation
challenges emerged [
18
]. Ongoing leadership support
varied, and provider motivation waned in the presence
of high levels of family engagement and intensity of
family need. These findings suggested that the
implementation of the 4Rs and 2Ss MFG required attention to
multi-level implementation processes to modify service
delivery practices on a sustainable level.
There has been a growing interest within the mental
health field in implementation processes that facilitate the
uptake and adoption of evidence-based practices in
community-based settings [
19, 20
]. The adoption of EBPs
is predicated upon both inner contextual factors
pertaining to the organization, including the agency’s climate,
leadership support, and provider perceptions about EBPs
[
21–26
], and the larger external context in which the
organization is embedded [
23
], including the political and
legislative landscape that may mandate or enforce specific
practices, and reimbursement mechanisms [
21
].
Hybrid effectiveness-implementation research designs
were developed to focus both on an intervention’s
effectiveness and implementation processes, either
sequentially or simultaneously, which is proposed to accelerate
the process to integrating EBPs in real-world practice
settings [
27
]. As noted by Curran [
27
] there are three
types of hybrid designs; examination of the intervention’s
effects while observing implementation processes,
examination of implementation processes while observing
effectiveness, and examining both effectiveness and
implementation simultaneously. This study will employ the
latter, a Hybrid Type II design, which tests both clinical
and implementation areas simultaneously [
27
].
Conceptual model
The 4Rs and 2Ss MFG intervention draws from group
therapy techniques, systemic family therapy, and
Purpose of this study
This study reflects a combination of clinical effectiveness
and implementation research methods, with aims
focused on replicating child outcomes associated with
the 4Rs and 2Ss MFG, and gathering information about
family and clinic/provider-level moderators of
implementation. A secondary objective is to test an innovative
method of adaptation of the 4Rs 2Ss MFG through the
formation of change teams that are hypothesized to
enhance outcomes and inner contextual factors. Within
the academic literature on implementation processes,
there is a growing emphasis upon staff serving as
champions or change agents to implement EBPs and
address potential barriers to adoption [
25, 28, 29, 35
]. In
this study, providers and leadership will form a Clinic
Implementation Team (CIT) and will be guided through
a standardized process by which they adapt the
intervention’s content, format, and structure. As shown in Fig. 1,
a main premise of this study is that adaptation of 4Rs
and 2Ss MFG by the CIT will impact clinic-level
mediators of readiness, leadership support, and climate
(path m) which, in turn, is proposed to affect
providerlevel variables (path n). These differential effects then
carry through to the core family variables (path o) and,
in turn, child outcomes (path p).
Methods/design
Study aims
In 2014, the National Institute of Mental Health
(NIMH)-funded “Family Groups for Urban Youth with
Disruptive Behavior.” The grant began with a planning
stage in 2015 and will be completed by 2019. The four
aims of the study are to examine:
1. The short-term and longitudinal impact of the 4Rs
and 2Ss MFG model on child ODD and functioning
2. Family level mediators, such as parenting and family
processes, that impact child outcomes
3. Clinic- and provider-level factors (organizational
culture and climate, leadership support, provider
attitudes towards EBPs) upon implementation
of the 4Rs and 2Ss MFG implementation, and
4. The impact of Clinic Implementation Teams on
clinic- and provider-level moderators of 4Rs and 2Ss
MFG implementation and integration
Participants
All NYS Office of Mental Health (OMH) child-serving
public outpatient mental health clinics across the five
boroughs of New York City (NYC) are eligible to
participate. To secure the sample of clinics, the study team
obtained a master list of 168 clinics from OMH. Based on
prior statewide studies conducted by the investigative
team, it was estimated that approximately 80% of the
clinics would participate; thus, we estimated that
approximately 134 clinics will enroll in the study.
At each clinic, we anticipate enrolling 20 caregiver/
child dyads (a minimum of n = 896 per arm, or 2688
total), and two providers per arm, yielding a total of 268
providers. Researchers will consecutively consent
families to participate in the study until a sample of 20
caregiver/child dyads per clinic is achieved. Eligible
caregivers are 18 years of age or older, English or Spanish
speaking, and the primary caregiver of a 7- to 11-year-old
child with ODD. Eligible providers are staff members who
work in a participating clinic in a clinical capacity, and are
English speaking.
Sampling procedure
A multi-stage sampling procedure was undertaken, with
randomization occurring at the agency level in order to
prevent an unbalanced representation of clinics across
the treatment arms and to conduct a sub-study in which
we will examine the impact of clinic choice upon
outcomes. From a master list of OMH-licensed agencies
and corresponding clinics within the five boroughs of
NYC that was received from The Office of Mental
Health in 2016, the investigative team created three lists:
(1) single-site (agencies that had only one outpatient
mental health clinic, (2) multi-site agencies with between
two to four clinics and, (3) mega-agencies that had five
or more clinics. Each list was organized alphabetically.
A separate randomization process was employed for
each list. Clinics within single-site agencies were
stratified by borough, and a random number generator was
employed to randomly assign clinics to one of three
conditions: (1) condition 0, which was services as usual
(SAU), (2) condition 1, the 4Rs and 2Ss MFG, and (3)
condition 2, the 4Rs and 2Ss MFG + CIT (all three
conditions are described more thoroughly on page 8). Once
the initial number was generated, subsequent clinics
were assigned the next number in the sequence (e.g., if
the first clinic received condition 0, the next clinic would
be enrolled in condition 1, and so forth). Of note, the
research staff member was unaware what numbers
corresponded with which condition (e.g., that a clinic in
condition 0 would be randomized to SAU). This
procedure was undertaken in order to minimize allocation bias.
Multi-site agencies with between two and four clinics
were also organized alphabetically within borough, and a
random number generator was used to assign the initial
clinic to a particular condition. Moreover, additional
clinics that were part of the agency in which the initial
clinic was randomized received the same condition. For
example, if one clinic was assigned condition 0, the
remaining clinics, regardless of which borough they were
located in, were also assigned to condition 0.
Finally, agencies with five or more clinics were given the
choice to select the study condition in which they would
participate with an even distribution of these mega
agencies occurring across the three study conditions.
Description of the conditions
The study consists of three arms: two active treatment
and one comparison condition:
1. The 4Rs and 2Ss Multiple Family Group intervention
The 4Rs and 2Ss MFG is a 16-week group for urban
children of between 7 and 11 years of age with ODD
and their caregiver(s). In this model, six to eight
families meet in weekly sessions focused on
addressing family and parenting factors associated
with behavior problems [
30–33
]. Session content
also addresses lack of social support and high stress,
which are two factors known to hinder treatment
attendance [
34–36
]. Families are provided with free
childcare, transportation expenses, and a meal to
address common logistical barriers to attendance
[8]. Moreover, perceptual barriers, such as stigma
and fears of being blamed for their child’s difficulties,
are targeted through group processes which
normalize family difficulties, validate their expertise
in solving problems, and promote mutual aid [
37
].
Providers who participate in this condition will
attend a 5-h training to learn the 4Rs and 2Ss MFG
model; thereafter, they will participate in bimonthly
supervision over the course of implementation
2. The 4Rs and 2Ss MFG + CIT
In addition to implementing the model and
receiving supervision, a CIT consisting of providers,
a clinic director and supervisor will participate in
ongoing meetings with the research team’s
adaptation specialist for a guided analysis of the
structure, content and processes of the treatment
model, including those that are amenable to
modification and those that must remain unchanged
in order to maintain fidelity
3. Standard care
Standard care includes the services that the clinic
normally provides to children with behavioral
problems, including outpatient mental health
therapy and pharmacology
Fidelity and supervision
Group facilitators will provide supervision by phone on
a bimonthly basis, and three face-to-face supervision
meetings after each fidelity check. Supervision will be
conducted by clinical staff who have extensive
knowledge of the 4Rs and 2Ss model, and who have
supervised clinicians in other initiatives using this model.
Additionally, two independent raters will conduct fidelity
checks for a minimum of three sessions to ensure
adherence to the treatment model. As fidelity checks will be
conducted by two personnel, inter-rater reliability will
be computed. Providers falling below 80% in fidelity will
be given additional supports, such as additional
supervisory sessions, to improve adherence.
Clinic involvement
Of the approximately 134 clinics that will be invited to
participate; 37 clinics have been enrolled in the study
thus far. Data collection has begun at 14 clinics in the
SAU clinic, eight clinics in condition 1 and seven clinics
in condition 2. Others have agreed to participate and
data collection will begin shortly.
Measures
Data collection will occur at baseline, 8, and 16 weeks from
baseline, and at 6-month follow-up. Demographic
characteristics about familial factors (e.g., age, race/ethnicity,
family income) will be collected via a questionnaire used in
prior studies. A separate demographic questionnaire will
be administered to providers to gather information
including their age, race/ethnicity, education, and credentials.
and above considered clinically significant depressive
symptoms [
45
]. The CESD short form evidences high
internal consistency, with a Cronbach α of above 0.8 [
45
]
Child factors
Child oppositional defiant disorder
will be measured by the Disruptive Behavior Disorders
Rating Scale Oppositional Defiant Subscale [
38
].
Completed by the child’s caregiver, this subscale consists of
eight items that are ranked using a 4-point Likert scale
ranging from “not at all” (0) to “very much” (3). Total
scores range from 0 to 24, and a total of four or more
items endorsed as “pretty much” or “very much” are
needed to meet criteria for ODD. Two studies that used
this scale across several outpatient mental health clinics
within NYC found good internal consistency with a
Cronbach’s α at baseline of .70 and .89, respectively [
39
].
Child ODD will also be measured by the Iowa Conners
Rating Scale-Oppositional/Defiant Subscale (IOWA CRS
OD) [
40
], a widely used brief measure of oppositional
defiant behavior in children. Completed by the child’s
caregiver, the IOWA CRS OD includes items ranked
using a 4-point Likert ranging from “not at all” (0) to
“very much” (3). Total scores range from 9 to 30, with
higher scores indicating greeter severity of symptoms. A
previous randomized effectiveness study of the MFG
intervention found that this scale had good internal
consistency with a Cronbach’s α at baseline of .86 [
39
].
Functioning
Child functioning will be measured via the Impairment
Rating Scale (IRS), [
41
] a six-item instrument that asks
caregivers to rate the severity of their child’s problems
and need for treatment across functional domains,
including their relationship with peers, parent(s), and
sibling(s); academic progress, and family functioning.
Caregivers place an “X” on a 7-point scale to signify
their child’s functioning along a continuum of
impairment that ranges from 0 (no need for treatment) to 6
(extreme need for treatment). Scores of three or greater
indicates clinical impairment [
42
]. The IRS has shown
evidence for concurrent, discriminant, and convergent
validity, and acceptable levels of temporal stability [
43
].
Caregiver factors
Depression
Caregiver depression will be measured using the short
form of the Center for Epidemiologic Studies Depression
Scale (CESD) [
44-46
], a free, publically available
screening tool consisting of seven items that assesses the
frequency of depressive symptoms within the past week.
Items are anchored on a 4-point Likert scale ranging
from 0 (rarely/none of the time) to 3 (most or all of the
time). CESD scores range from 0 to 21, with a score of 8
Stress
The Parenting Stress Index (PSI) short form will be
administered to measure stress [
47
]. This 36-item scale
uses a 5-point Likert scale, ranging from “strongly
disagree (1)” to “strongly agree (5).” Total scores range
from 36 to 180, with higher scores indicating increased
levels of parent stress [
48
]. In a prior study of the 4Rs
and 2Ss MFG, Cronbach’s α values reported as baseline,
mid test, post test, and 6-month follow-up, were 0.91,
0.92, 0.94, and 0.94, respectively [
39
].
Family factors
Parenting quality
The Alabama Parenting Questionnaire (APQ-9) short
form will measure parenting factors, and specifically
positive parenting, inconsistent discipline, and poor
supervision [
49
]. This scale consists of nine items that
are measured in terms of frequency, ranging from never
(1), to always (5). Items are summed to determine
positive parenting, inconsistent discipline, and poor
supervision. The APQ-9 short form was shown to be reliable
and valid in two prior studies, and has a Cronbach’s α
ranging from 0.59 to 0.84 in the first study and 0.80 to
0.92 in the second study [
49
].
Uptake of the 4Rs and 2Ss MFG
Two subscales of the McMaster Family Assessment Device
(FAD) [
50
] will measure the extent to which the 4Rs are
used within the family. The two subscales, focusing on roles
and communication, use a 4-point Likert scale ranging
from strongly disagree (1) to strongly agree (4). Higher
numbers indicate more problematic functioning. The FAD
was shown to have strong validity in multiple studies, and a
test of this scale using a sample of 503 individuals found
acceptable reliability of the subscales, with a Cronbach’s α of
0.72 (Roles) and 0.75 (Communication) [
50
].
Social support
To measure social support, the family subscale of the
Multidimensional Scale of Perceived Social Support
(MSPSS) [
51
] will be administered. This four-item
subscale, uses a 7-point Likert scale ranging from very strongly
disagree (1) to very strongly agree (7). Scores are summed,
with higher scores reflecting greater perceived support.
The MSPSS has strong factorial validity as well as good
internal and test-retest reliability (Cronbach α = 0.85 for
the Family subscale) [
51
].
Engagement in treatment
Attendance will be collected, and a total number of
sessions attended will be tallied for each participant.
Barriers to treatment
These will be assessed using the Kazdin Barriers to
Treatment (KBT) scale [
52
], which is comprised of three
subscales: perceived relevance of treatment, relationship
with the therapist, and critical events. The first two
subscales consist of 14 items combined, and are rated on a
5-point scale, ranging from (1) never a problem to (5)
very often a problem. The third subscale, delivered in a
yes-no format, includes 14 critical events that may lead
to treatment termination. Items on the 5-point scale are
summed to provide the total barriers score, and items
on the binary scale (1 = no, 2 = yes) are summed to
reflect the absence or presence of critical events [
52, 53
].
Higher scores reflect greater severity regarding barriers
to treatment [52].
Implementation processes factors
Attitudes about evidence-based practices
The Evidence-Based Practice Attitude Scale [
21
] will
assess provider attitudes toward evidence-based practices.
The scale is comprised of four subscales: the intuitive
appeal of EBP, the likelihood of adopting EBP in light of
such requirements, openness to new practices, and the
degree to which divergence is perceived between usual
practice and research-based/academically developed
interventions. Five possible responses are provided on
Likert scale, from 0 (not at all) to 4 (to a very great
extent). For each subscale, a mean score is calculated and
the fourth (divergence) is reverse coded. A higher mean
score indicates a greater degree of the construct
reflected by that subscale. A study using this scale to
assess service provider attitudes from 51 mental health
service programs found good internal consistency
reliability, with an overall Cronbach’s of 0.77 [
21
].
Organizational processes and readiness to change
This will be assessed via the Texas Christian University
(TCU) Organizational Readiness for Change (TCU
ORC) instrument [
54
]. Responses to this 66-item
instrument range from strongly disagree (1) to strongly
agree (5). Subscales focus on motivation for change,
resources, and staff. In addition, the Training Exposure
and Utilization subscale measures training exposure and
utilization within an agency. It contains seven items that
are assessed on a scale that ranges from never (1) to
almost always (5). Mean scores for each subscale are
multiplied by 10, so that final scores range from 10 to 50.
Scale scores above 30 indicate some average agreement on
the concept measured by the scale, and scale scores below
30 indicate some disagreement on average [
55
]. Reliability
and validity of the ORC were examined in a study using a
national sample of over 500 staff members from more
than 100 programs, with the majority of subscales
showing a Cronbach’s α of above .70 [
54
].
Beliefs about treatment
The Measure of Beliefs about Participation in Family
Centered Service Delivery [
56
] will assess providers’
beliefs about participating in family centered services. This
28-item instrument consists of five scales that measure
beliefs about family centered philosophy and principles,
positive and negative outcomes, personal competencies,
and barriers [
56
]. Responses are provided on a 7-point
scale ranging from strongly disagree (1) to strongly agree
(7). The total scale score is calculated as the average of
all means of the scale scores. Higher scores on scales
represent stronger beliefs regarding family centered
services. A study examining the development and validity
of this instrument using 818 respondents showed good
evidence of construct validity for this measure as well as
moderate to excellent reliability, with coefficient α values
ranging from .61 to .83 [
56
].
Treatment fidelity and supervision
Fidelity monitoring tools will assess facilitator adherence
to curriculum content (e.g., session topic, information
conveyed, competence in guiding discussions) and
clinical skills (i.e., active listening). These fidelity
assessments were standardized during fidelity observations of
prior 4Rs and 2Ss MFGs studies [
1, 18
].
Organizational characteristics
These include size, fiscal health and funding streams,
and populations served will be analyzed from two
preexisting datasets collected by New York State Office of
Mental Health (NYSOMH) about organizational
composition and service delivery.
Hypotheses and data analytic strategy
The purpose of this study is to examine outcomes
associated with the 4Rs and 2Ss MFG, implementation
processes that impact uptake and sustained use of the
intervention over time, and the impact of an innovative
change agent team upon outcomes. The following
hypotheses will be tested:
Aim #1: to examine the short-term and longitudinal
impact of 4Rs and 2Ss MFG on child ODD and
functioning
Hypothesis #1: youth who participate in 4Rs and 2Ss
MFG will display significantly reduced conduct
difficulties and increased functioning over time
compared to those involved in standard care: youth in
the 4Rs and 2Ss MFG + CIT condition will evidence the
greatest magnitude of change in outcomes over time
Aim #2: to examine family level mediators (e.g.,
parenting, family process) of child outcomes
Hypothesis #1: based on existing 4Rs and 2Ss MFG
findings, specific family level variables (caregiver stress,
the parent/child relationship and established rules) will
be associated with significantly greater impact on child
outcomes over time relative to social support,
established responsibilities, and respectful
communication
Aim #3: to examine clinic- (readiness to adopt an
innovation, leadership support, and climate) and
provider-level moderators (engagement, preparedness,
motivation) of 4Rs and 2Ss MFG implementation and
integration
Hypothesis #1: leadership support will evidence
significantly greater impact on 4Rs and 2Ss MFG
implementation/integration in comparison to general
readiness and clinic climate
Hypothesis #2: provider motivation will evidence
significantly greater impact on 4Rs and 2Ss MFG
implementation/integration relative to other
provider-level variables
Aim #4: to examine the impact of CITs on clinic- and
provider-level moderators of 4Rs and 2Ss MFG
implementation and integration
Hypothesis #1: clinic directors assigned to MFG + CIT
will evidence significantly higher readiness for
innovation and support, and providers assigned to
MFG + CIT will evidence significantly higher
preparedness and motivation to implement MFG
relative to MFG alone
Hypothesis #2: preparedness, motivation of providers,
and leadership support assigned to MFG + CIT will be
maintained over time and be significantly higher
relative to MFG alone
Hypothesis #3: providers assigned to MFG + CIT will
evidence significantly enhanced fidelity to MFG relative
to MFG only
Hypothesis #4: clinics assigned to MFG + CIT will have
more MFGs initiated and completed (with high fidelity
and attendance) relative to MFG alone
Overall, the two primary outcomes being tested in
this study are changes in child behavior and child
functioning. These outcomes are being tested specifically in
aims 1 and 2, as measured by the Disruptive Behavior
Disorders Rating Scale Oppositional Defiant Subscale
[
38
], Iowa Conners Rating Scale-Oppositional/Defiant
Subscale [
40
] and the Impairment Rating Scale (IRS)
[
41
]. Aims 3 and 4 intend to measure mediators of the
primary outcome, by testing clinic readiness and
motivation to implement the 4Rs and 2Ss intervention
initially and over time, and testing whether fidelity has
been adhered to.
Analytic strategy
The overall analytic structure uses a three-level,
multilevel Structural Equation Modeling (MSEM) framework
with family level variables at level 1, provider variables
at level 2, and clinic variables at level 3. The analytic
structure is complex, so we highlight our approach using
Fig. 2 as our primary reference. The MSEM framework
is useful because it ultimately allows a test of these aims
within an integrated path modeling structure.
We will employ mediation analysis with two-level
MSEM models as explicated by Preacher, Zhang and
Zyphur [
57
] and extended to three-level models by
Preacher [
57
]. We will use a combination of limited
information estimation frameworks (focused on
subportions of the full model) and full information
estimation frameworks (focused on the entire three-level
model) to strategically answer questions. At level 1, child
outcomes at the immediate post test are modeled as a
function of the family predictors (see Fig. 2) using
baseline measures of outcomes and family mediators as
covariates, taking into account the clustering at the
higher levels of the model. The follow-up (6-month)
measures of the mediators and the outcomes are included
within the model in a classic SEM panel model with
autoregressive effects, thereby linking the follow-up data to the
immediate post-test data. This is an advantage of using
MSEM over traditional multi-level modeling. Different
centering strategies (e.g., grand mean versus group mean)
can be explored to garner various perspectives on the
data. This feature of the model provides perspectives on
aim 2 and the relative importance of paths g–l in Fig. 1.
To ascertain perspectives of the effects of clinic-level
variables on provider variables, random intercept MSEM with
the provider-level variables can be estimated as a function
of the clinic variables that include dummy coded
treatment variables (4Rs and 2Ss MFG + CIT versus 4Rs and
2Ss MFG) impacting the clinic mediators of readiness,
leadership support, and climate, which in turn, affect the
provider-level intercepts. This modeling, or variants of it,
address aims 3 and 4.
The overall analytic structure requires strategic use of
different dummy contrasts and limited information
estimation of various MSEM models that map onto the
questions dictated by the aims, coupled with judicious use of
mediational tests linked to joint significance testing. An
alternative analytic structure is to work with all four time
periods simultaneously in a classic growth curve model
and then to model the linear growth in variables as a
function of the provider and clinic variables in strategically
defined limited information estimation frameworks dictated
by the overall model structure in Fig. 2. We also will
explore this approach. Analyses of variable importance
(Fig. 1) have been summarized by Tonidandel et al. [
58
].
Of the two major tools to address the issue (dominance
analysis and relative weight analysis), we will use relative
weight analysis to gain perspectives because of the less
intense computational demands that it makes and its overall
flexibility. Missing data will be treated using Full
Information Maximum Likelihood (FIML) methods.
A Data Monitoring and Review Board is responsible
for reviewing this study’s data to ensure quality and
integrity, the safety of participants and study progress, and
to make any determinations regarding potential changes
to the study. This is comprised of the investigative team
with ongoing support from New York University’s
Institutional Review Board (IRB). The DMRB meets on a
quarterly basis under normal circumstances, or within
two business days should an adverse event occur. This
process is independent from the sponsors, although they
will be notified in the case of a significant change to
study protocol or in the case of an adverse event.
Ethical issues
There are potential but minor risks for providers
associated with participating in the study, specifically, feeling
coerced into participating, loss of confidentiality, and
experiencing discomfort. These risks are addressed in
several ways: potential participants are assured that
participation is voluntary and will not affect their
employment status, all study materials are coded with ID
numbers and do not include identifying information,
and all data will be maintained in password-protected
computer files.
Among caregivers, there is a risk that they may feel
uncomfortable with some of the questions in the
assessment measures. However, research staff clarify that they
do not need to answer any question that makes them
feel uncomfortable. The risk to loss of confidentiality is
also present for caregivers and is addressed following the
same precautions and guidelines used for protecting
provider confidentiality. Finally, there is a potential risk
that symptoms may worsen as a consequence of the
progression of a serious mental health issue or response to
a service provided. In the event that this happens, study
participation will be halted and appropriate care
provided immediately in accordance with clinic procedure.
Discussion
This study represents an innovative approach to examine
the uptake and sustainment of a National Registry of
Evidence-based Programs and Practices (NREPP)-approved,
evidence-based treatment for child DBDs, and the
organizational factors that impede or facilitate adoption of
EBPs in public mental health settings. This study will also
examine a novel approach to adapt an EBP in a structured
way, in order to enhance its fit with the needs of the
populations being served and the organizational context. One
of the strengths of this study is that it will test uptake
across the entire child-serving, public mental health
service system in the five boroughs of NYC.
The study aims to generate knowledge needed to address
seemingly intractable urban service delivery challenges,
including lack of engagement of low-income youth with
serious DBDs and their families, too few clinics offering family
based, evidence-informed services; lack of scalable,
empirically supported interventions designed for resource-strapped
child settings; and few empirically supported options for
public policy-makers to support the uptake and integration
of service innovations within behavioral health care.
Trial status
Clinic recruitment for standard care has been
completed. Data collection has been initiated, and one third
(n = 792) of the sample of providers and caregivers have
been enrolled. Clinic recruitment for MFG and MFG +
CIT have been initiated, and some clinics have already
completed training and have participated in adaptation
meetings. Recruitment of clinics for the experimental
arms will be ongoing until enrollment targets are met.
Groups for both MFG and MFG + CIT are expected to
begin in the first trimester of 2017. A populated
Standard Protocol Items: Recommendations for Interventional
Trials (SPIRIT) Checklist (see Additional file 1) and
Figure ( see Fig. 3) for all study protocols are included as
an additional file (see Additional file 1 and Fig. 3). A
composition of the data monitoring commitee has also
been included (see Additional file 2).
Additional files
Additional file 1: SPIRIT Checklist. List of where to locate recommended
items to address in a clinical trial protocol and related documents.
(DOC 122 kb)
Additional file 2: Data Monitoring Committee Information. Information
regarding the Data Monitoring Committee for this study. (DOC 15 kb)
Abbreviations
APQ-9: Alabama Parenting Questionnaire nine-item short form; CESD: Center
for Epidemiologic Studies Depression Scale; CIT: Clinic Implementation Team;
DBD: Disruptive behavior disorder; EBP: Evidence-based practice;
FAD: McMaster Family Assessment Device; IOWA CRS: Iowa Conners Rating
Scale; IRS: Impairment Rating Scale; KBT: Kazdin Barriers to Treatment;
MFG: Multiple Family Group; MSEM: Multi-level Structural Equation Modeling;
MSPSS: Multidimensional Scale of Perceived Social Support; NREPP: National
Registry of Evidence-based Programs and Practices; NYC: New York City;
NYSOMH: New York State Office of Mental Health; OD: Oppositional defiant;
ODD: Oppositional defiant disorder; PSI: Parenting Stress Index;
SAMHSA: Substance Abuse and Mental Health Services Administration;
SEM: Structural Equation Modeling; TCU ORC: Texas Christian University
Organizational Readiness for Change
Acknowledgements
We would like to thank Anthony Salerno for his help in the development of the
CIT model, Lindsay Bornheimer for her assistance with data management, and
Geetha Gopalan for her reviews and consultations on the project.
Funding
The research has been funded by the NIMH grant no. RFA-MH-15-320. The
study sponsor/funders’ role was to evaluate the suitability of the study
design, data collection and management, as part of approval for funding.
Availability of data and materials
The data for this study has not been collected in full and, therefore, is not
yet available to the public. Once complete, the dataset will be sent to the
NIMH, which will make it publicly available.
Authors’ contributions
MA provided oversight for this study and wrote the “Background” section, the
analytic strategy, and the “Discussion” sections. EH wrote the “Measures”
section, the hypotheses, and contributed to the “Discussion” section. EG wrote
the randomization procedure in the “Methods” section, and contributed to
the “Discussion” section. MM (Mini) wrote the description of the conditions in the
“Methods” section, and contributed to the “Discussion” section. CC wrote the “4Rs
and 2Ss MFG intervention” section in the “Background” section. MM (McKay)
contributed to the “Background” section. All authors read and approved the
final manuscript.
Ethics approval and consent to participate
This study was approved by New York University’s Institutional Review Board
(IRB). The study was also approved by agencies’ own IRBs when applicable
(a list of all ethics committees is attached). An explanation of the study
procedures and its risks and benefits is provided to all caregivers and
providers via an Informed Consent Form. Participants have an opportunity to
read and to receive clarification from the research staff on the content of the
Informed Consent Form as well as answers to any additional questions or
concerns they might have. Participants receive a copy of the Consent Form,
which contains the principal investigator’s contact information and the contact
information for the IRB that approved the research project (New York University
IRB, number 14-10423). Children are required to give verbal assent in order to
participate in the group and to have their caregivers answer questions about
them. Research personnel explain the study to the children, using age-appropriate
language to ensure their understanding. Children are given the option not to
participate, and are assured that refusal to participate will not affect the services
that they receive.
All letters of approval from IRBs are attached as a separate document. The
New York University IRB was used when agencies did not have their own
ethics review board. In the circumstance that agencies had their own IRB,
additional approval was sought from their agency as well. The protocol was
reviewed by the Services Research and Clinical Epidemiology Branch at the
National Institute of Mental Health.
Consent for publication
Any results that will be published in academic journals will adhere to
New York University’s IRB guidelines.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.
Submit your next manuscript to BioMed Central
and we will help you at every step:
1. Acri M , Gopalan G , Chacko A , McKay M . (in press). Engaging families into treatment for child behavior disorders: a synthesis of the literature . In: Lochman JE , Matthys W , editors. The Wiley handbook of disruptive and impulse-control disorders . New York: Wiley.
2. Linver MR , Brooks-Gunn J , Kohen DE . Family processes as pathways from income to young children's development . Dev Psychol . 2002 ; 38 ( 5 ): 719 - 34 .
3. Mistry RS , et al. Economic well-being and children's social adjustment: the role of family process in an ethnically diverse low-income sample . Child Dev . 2002 ; 73 ( 3 ): 935 - 51 .
4. National Institute of Child, Health and Human Development Early Child Care Research Network. Duration and developmental timing of poverty and children's cognitive and social development from birth through third grade . Child Dev . 2005 ; 76 ( 4 ): 795 - 810 .
5. Herrenkohl TI , Maguin E , Hill KG , Hawkins JD , et al. Developmental risk factors for youth violence . J Adolesc Health . 2000 ; 26 ( 3 ): 176 - 86 .
6. Ghandour RM , Kogan MD , Blumberg SJ , Jones JR , Perrin JM . Mental health conditions among school-aged children: geographic and sociodemographic patterns in prevalence and treatment . J Dev Behav Pediatr . 2012 ; 33 ( 1 ): 42 - 54 .
7. Brestan EV , Eyberg SM . Effective psychosocial treatments of conduct-disordered children and adolescents: 29 years, 82 studies, and 5,272 kids . J Clin Child Psychol . 1998 ; 27 ( 2 ): 180 - 9 .
8. McKay MM , Bannon Jr WM . Engaging families in child mental health services . Child Adolesc Psychiatr Clin N Am . 2004 ; 13 ( 4 ): 905 - 21 .
9. Hoagwood KE , et al. Scaling up evidence-based practices for children and families in New York State: toward evidence-based policies on implementation for state mental health systems . J Clin Child Adolesc Psychol . 2014 ; 43 ( 2 ): 145 - 57 .
10. Chacko A , Gopalan G , Franco L , Dean-Assael K , et al. Multiple Family Group service model for children with disruptive behavior disorders: child outcomes at post-treatment . J Emot Behav Disord . 2015 ; 23 ( 2 ): 67 - 77 .
11. Chacko A , Gopalan G , Jackson J , Franco L. , Dean-Assael K , McKay M . (under review). Effectiveness of Multiple Family Groups: Post-treatment outcomes for youth with disruptive behavior disorders .
12. Gopalan GA , Fuss A , Wisdom JP . Multiple family groups for child behavior difficulties: retention among child welfare-involved caregivers . Res Soc Work Pract . 2015 ; 25 ( 5 ): 564 - 77 .
13. McKay MM , Harrison ME , Gonzales J , Kim L , Quintana E . Multiple-family groups for urban children with conduct difficulties and their families . Psychiatr Serv . 2002 ; 53 ( 11 ): 1467 - 8 .
14. McKay MM , Gopalan G , Franco L , Dean-Assael K , Chacko A , Jackson JM , Fuss A. A collaboratively designed child mental health service model: multiple family groups for urban children with conduct difficulties . Res Soc Work Pract . 2011 ; 21 ( 6 ): 664 - 74 .
15. McKay MM , Gonzales J , Quintana E , Kim L , Abdul-Adil J . Multiple family groups: an alternative for reducing disruptive behavioral difficulties of urban children . Res Soc Work Pract . 1999 ; 9 ( 5 ): 593 - 607 .
16. S. A.M.H.S. A. The 4 Rs and 2 Ss for strengthening families program . 2015 . http://nrepp.samhsa.gov/ProgramProfile.aspx? id=41. Accessed 1 Feb 2017 .
17. Chorpita BF , Becker KD , Daleiden EL . Understanding the common elements of evidence-based practice: misconceptions and clinical examples . J Am Acad Child Adolesc Psychiatry . 2007 ; 46 ( 5 ): 647 - 52 .
18. Gopalan G , Franco LM , Dean-Assael K , McGuire-Schwartz M , Chacko A , McKay MM . Statewide implementation of the 4Rs and 2Ss for strengthening families . J Evid Based Soc Work . 2014 ; 11 ( 1-2 ): 84 - 96 .
19. Cabassa LJ . Implementation science: why it matters for the future of social work . J Soc Work Educ . 2016 ; 52 : 1 - 13 .
20. Marriott BR , Rodriguez AL , Landes SJ , Lewis CC , Comtois KA . A methodology for enhancing implementation science proposals: comparison of face-to-face versus virtual workshops . Implement Sci . 2016 ; 11 : 62 .
21. Aarons GA . Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-based Practice Attitude Scale (EBPAS) . Ment Health Serv Res . 2004 ; 6 : 61 - 74 .
22. Beidas RS , Adams DR , Kratz HE , Jackson K , Berkowitz S , Zinny A , Evans A , et al. Lessons learned while building a trauma-informed public behavioral health system in the City of Philadelphia . Eval Program Plann . 2016 ; 59 : 21 - 32 .
23. Willging CE , Green AE , Gunderson L , Chaffin M , Aarons GA . From a “perfect storm” to “smooth sailing”: policymaker perspectives on implementation and sustainment of an evidence-based practice in two states . Child Maltreat . 2015 ; 20 ( 1 ): 24 - 36 .
24. Mitchell RE , Florin P , Stevenson JF . Supporting community-based prevention and health promotion initiatives: developing effective technical assistance systems . Health Educ Behav . 2002 ; 29 ( 5 ): 620 - 39 .
25. Chaple M , Sacks S. The impact of technical assistance and implementation support on program capacity to deliver integrated services . J Behav Health Serv Res . 2014 ; 43 ( 1 ): 3 - 17 .
26. Mitchell PF. Evidence-based practice in real-world services for young people with complex needs: new opportunities suggested by recent implementation science . Child Youth Serv Rev . 2011 ; 33 ( 2 ): 207 - 16 .
27. Curran GM , Bauer M , Mittman B , Pyne JM , Stetler C. Effectivenessimplementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact . Med Care . 2012 ; 50 ( 3 ): 217 - 26 .
28. Glisson C , Schoenwald SK . The ARC organizational and community intervention strategy for implementing evidence-based children's mental health treatments . Ment Health Serv Res . 2005 ; 7 ( 4 ): 243 - 59 .
29. Glisson C , Williams NJ , Hemmelgarn A , Proctor E , Green P . Increasing clinicians' EBT exploration and preparation behavior in youth mental health services by changing organizational culture with ARC . Behav Res Ther . 2016 ; 76 : 40 - 6 .
30. Lanza ST , Rhoades BL , Nix RL , Greenberg MT . Modeling the interplay of multilevel risk factors for future academic and behavior problems: a person-centered approach . Dev Psychopathol . 2010 ; 22 ( 2 ): 313 - 35 .
31. Montague M , Cavendish W , Enders C , Dietz S . Interpersonal relationships and the development of behavior problems in adolescents in urban schools: a longitudinal study . J Youth Adolesc . 2010 ; 39 ( 6 ): 646 - 57 .
32. Patterson GR , Crosby L , Vuchinich S. Predicting risk for early police arrest . J Quant Criminol . 1992 ; 8 ( 4 ): 335 - 55 .
33. Patterson GR , Reid JB , Dishion TJ . Antisocial boys: a social interactional approach . Eugene: Castalia; 1992 .
34. Kazdin AE , Whitley MK . Treatment of parental stress to enhance therapeutic change among children referred for aggressive and antisocial behavior . J Consult Clin Psychol . 2003 ; 71 ( 3 ): 504 - 15 .
35. Kazdin A . Conduct disorders in childhood and adolescence . Vol 9 . Thousand Oaks : Sage; 1998 .
36. Wahler RG , Dumas JE . Attentional problems in dysfunctional mother-child interactions: an interbehavioral model . Psychol Bull . 1989 ; 105 ( 1 ): 116 - 30 .
37. Gopalan G , Franco L . Multiple family groups to reduce disruptive behaviors . In: Gitterman A, editor. Encyclopedia of social work with groups . New York: Routledge; 2009 .
38. Pelham Jr WE , et al. Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders . J Am Acad Child Adolesc Psychiatry . 1992 ; 31 ( 2 ): 210 - 8 .
39. Acri M , Bornheimer L , Jessell L , Chomancuzuk H , Adler JG , Gopalan G , McKay MM . The intersection of extreme poverty and familial mental health in the United States . Social Work in Mental Health . 2017 ; 15 ( 6 ): 677 - 89 .
40. Waschbusch DA , Willoughby MT . Attention-deficit/hyperactivity disorder and callous-unemotional traits as moderators of conduct problems when examining impairment and aggression in elementary school children . Aggress Behav . 2008 ; 34 ( 2 ): 139 - 53 .
41. Fabiano GA , et al. A practical measure of impairment: psychometric properties of the impairment rating scale in samples of children with attention deficit hyperactivity disorder and two school-based samples . J Clin Child Adolesc Psychol . 2006 ; 35 ( 3 ): 369 - 85 .
42. Fabiano GA & Pelham WE Jr. Impairment in children . In: Goldstein S, Naglieri J , editors. Assessing impairment: from theory to practice . Thousand Oaks: Springer Science; 2009 . p. 105 - 119 .
43. Pelham Jr WE , Fabiano GA , Massetti GM . Evidence-based assessment of attention deficit hyperactivity disorder in children and adolescents . J Clin Child Adolesc Psychol . 2005 ; 34 ( 3 ): 449 - 76 .
44. Radloff LS. The CES-D Scale : A self-report depression scale for research in the general population . Appl Psychol Meas . 1977 ; 1 ( 3 ): 385 - 401 .
45. Levine SZ . Evaluating the seven-item Center for Epidemiologic Studies depression scale short-form: a longitudinal U.S. community study . Soc Psychiatry Psychiatr Epidemiol . 2013 ; 48 ( 9 ): 1519 - 26 .
46. Zich JM , Attkisson CC , Greenfield TK . Screening for depression in primary care clinics: the CES-D and the BDI . Int J Psychiatry Med . 1990 ; 20 ( 3 ): 259 - 77 .
47. Barroso N , Hungerford GM , Garcia D , Grazano PA , Bagner DM . Psychometric properties of the parenting stress index-short form (PSF) in a high-risk sample of mothers and their infants . Psychol Assess . 2006 ; 29 ( 10 ): 1331 - 5 .
48. Abidin RR . Parenting Stress Index, third edition: professional manual . Odessa: Psychological Assessment Resources , Inc; 1995 .
49. Elgar F , et al. Development and validation of a short form of the Alabama Parenting Questionnaire . J Child Fam Stud . 2007 ; 16 ( 2 ): 243 - 59 .
50. Epstein NB , Baldwin LM , Bishop DS . The McMaster Family Assessment Device . J Marital Fam Ther . 1983 ; 9 ( 2 ): 171 .
51. Dahlem NW , Zimet GD , Walker RR . The Multidimensional Scale of Perceived Social Support: a confirmation study . J Clin Psychol . 1991 ; 47 ( 6 ): 756 - 61 .
52. Kazdin AE , et al. Barriers to Treatment Participation Scale: evaluation and validation in the context of child outpatient treatment . J Child Psychol Psychiatry . 1997 ; 38 ( 8 ): 1051 - 62 .
53. Kazdin AE , Holland L , Crowley M. Family experience of barriers to treatment and premature termination from child therapy . J Consult Clin Psychol . 1997 ; 65 ( 3 ): 453 - 63 .
54. Lehman WE , Greener JM , Simpson DD . Assessing organizational readiness for change . J Subst Abus Treat . 2002 ; 22 : 197 - 209 .
55. Lehman WE , Greener JM , Rowan-Szal GA , Flynn PM . Organizational readiness for change in correctional and community substance abuse programs . J Offender Rehabil . 2012 ; 51 ( 1-2 ): 96 - 114 .
56. King G , et al. A measure of parents' and service providers' beliefs about participation in family-centered services . Child Health Care . 2003 ; 32 ( 3 ): 191 .
57. Preacher KJ , Zyphur MJ , Zhang Z. A general multilevel SEM framework for assessing multilevel mediation . Psychol Methods . 2010 ; 15 ( 3 ): 209 - 33 .
58. Tonidandel S , LeBreton JM . Relative Importance Analysis: a useful supplement to regression analysis . J Bus Psychol . 2011 ; 26 ( 1 ): 1 - 9 .