Use of the estimated intraclass correlation for correcting differences in effect size by level

Soyeon Ahn Nicolas D. Myers Ying Jin In a meta-analysis of intervention or group comparison studies, researchers often encounter the circumstance in which the standardized mean differences (d-effect sizes) are computed at multiple levels (e.g., individual vs. cluster). Cluster-level d-effect sizes may be inflated and, thus, may need to be corrected using the intraclass correlation (ICC) before being combined with individual-level d-effect sizes. The ICC value, however, is seldom reported in primary studies and, thus, may need to be computed from other sources. This article proposes a method for estimating the ICC value from the reported standard deviations within a particular meta-analysis (i.e., estimated ICC) when an appropriate default ICC value (Hedges, 2009b) is unavailable. A series of simulations provided evidence that the proposed method yields an accurate and precise estimated ICC value, which can then be used for correct estimation of a d-effect size. The effects of other pertinent factors (e.g., number of studies) were also examined, followed by discussion of related limitations and future research in this area. - treatment effect or group mean difference (Cooper, Hedges, & Valentine, 2009). Thus, many researchers have combined d-effect sizes from multiple studies and have drawn a statistical inference about the overall intervention/treatment effect (e.g., Mol, Bus, & de Jong, 2009; Slavin, Lake, Chambers, Chueng, & Davis, 2009) or group mean difference (e.g., Swanson & Hsieh, 2009). In a meta-analysis of studies in the social sciences and education, researchers often encounter the circumstance in which d-effect sizes are computed using summary statistics (i.e., means and standard deviations) originating from multiple levels (e.g., student vs. classroom or patient vs. clinic) across studies. For example, in a meta-analysis examining the effect of teachers professional development programs on student mathematics achievement by Salinas (2010), two studies used aggregated data at the classroom level, while the rest of the studies were based on data at the student level. Because individuals (e.g., students, clients) within the same cluster (e.g., classroom, counselors) are likely to be nonindependent, resulting in underestimated standard errors (Raudenbush & Bryk, 2002), d-effect sizes computed at the cluster level tend to be inflated, as compared with d-effect sizes computed at the individual level (Hedges, 2007, 2009a, b; What Works Clearinghouse [WWC], 2008). Thus, d-effect sizes from different levels should not be combined in a meta-analysis prior to the estimation of an overall effect size that accounts for the magnitude of nonindependence by level. Such an issue often arises when studies included in a meta-analysis are based on either individual- or clusterlevel data. Studies reporting findings from both cluster- and individual-level data could provide sufficient statistics to take account of dependency among samples within the same cluster when computing d-effect size. However, there are some cases in education and/or the social sciences where providing findings from both cluster- and individuallevel data is not feasible. For instance, a researcher using data gathered from a state may be restricted only to classroom data, rather than individual student data. In such a case, no information is provided at the individual level, which prevents the researcher from taking account of sample dependency when computing effect size. Clearly, in cases where only individual-level data are available, it would be incorrect for the researcher to rely on negatively biased standard errors that result from ignoring the nested nature of the data. For handling such an issue, researchers have suggested correcting the d-effect size computed at the cluster-level (dclusters) and its associated variance vdclusters from the clusters using an intraclass correlation (ICC), which can ultimately be combined with the d-effect size computed at the individual level (dindividuals). The formulas for correcting dclusters that can be compatible with dindividuals, which is described in Hedges (2009b), are dadjusted dclusterspffiffiffiffiffiCffiffiffi; IC Consequently, the cluster-level d-effect size (dadjusted) and its variance vdadjusted adjusted by the ICC value are no longer biased and, thus, are compatible with the individual-level d-effect sizes. The critical correcting factor in Eqs. 1 and 2 is the ICC value, which represents the degree to which observations in the same cluster are dependent due to shared variances (Hox, 2002; Kreft & de Leeuw, 1998). Simply, the ICC value () is the proportion of cluster-level variance sc2lusters to total variance st2otal and is given by where si2ndividualsis the individual-level variance. However, the correcting factor given in Eqs. 1 and 2, the ICC value, is seldom reported in studies (Hedges, 2009b), which makes it difficult to correct for differences in the cluster-level d-ef (...truncated)