Quality Indicator Analysis

Quality Indicator: 1.0. Context and Setting

Quality Indicator: 2.0. Participants

“2.2. The study describes disability or risk status of the participants (e.g., specific learning disability, autism spectrum disorder, behavior problem, at risk for reading failure) and method for determining status (e.g., identified by school using state IDEA criteria, teacher nomination, standardized intelligence test, curriculum-based measurement probes, rating scale).”

Quality Indicator: 3.0. Intervention Agent

“3.2. The study describes any specific training (e.g., amount of training, training to a criterion) or qualifications (e.g., professional credential) required to implement the intervention, and indicates that the interventionist has achieved them.”

“4.2. When relevant, the study describes materials (e.g., manipulatives, worksheets, timers, cues, toys), or cites one or more accessible sources providing this information.”

“5.2. The study assesses and reports implementation fidelity related to dosage or exposure using direct, reliable measures (e.g., observations or self-report of the duration, frequency, curriculum coverage of implementation).”

“5.3. As appropriate, the study assesses and reports implementation fidelity (a) regularly throughout implementation of the intervention (e.g., beginning, middle, end of the intervention period), and (b) for each interventionist, each setting, and each participant or other unit of analysis. If either adherence or dosage is assessed and reported, this item applies to the type of fidelity assessed. If neither adherence nor dosage is assessed and reported, this item is not applicable.”

“6.2. The study describes baseline (single-subject studies) or control/comparison (group comparison studies) conditions, such as the curriculum, instruction, and interventions (e.g., definition, duration, length, frequency, and learner: instructor ratio).”

“6.3. Control/comparison-condition or baseline-condition participants have no or extremely limited access to the treatment intervention.”

“6.4. The study clearly describes assignment to groups, which involves participants (or classrooms, schools, or other unit of analysis) being assigned to groups in one of the following ways:

(a) randomly;

(b) nonrandomly, but the comparison groups are matched very closely to the intervention group (e.g., matched on prior test scores, demographics, a propensity score; see Song & Herman, 2010);

(c) nonrandomly, but techniques are used to measure differences and, if meaningful differences are identified—for example, statistically significant difference, difference greater than 5% of a standard deviation (What Works Clearinghouse, 2011)—to statistically control for any differences between groups on relevant pretest scores or demographic characteristics (e.g., statistically adjust for confounding variable through techniques such as ANCOVA or propensity score analysis); or

(d) nonrandomly on the basis of a reasonable cutoff point (regression discontinuity design).””

“6.5. The design provides at least three demonstrations of experimental effects at three different times.”

“6.6. For single-subject research designs with a baseline phase (alternating treatment designs do not require a baseline), all baseline phases include at least three data points (except when fewer are justified by study author due to reasons such as measuring severe or dangerous problem behaviors and zero baseline behaviors with no likelihood of improvement without intervention) and establish a pattern that predicts undesirable future performance (e.g., increasing trend in problem behavior, consistently infrequent exhibition of appropriate behavior, highly variable behavior).”

“6.7. The design controls for common threats to internal validity (e.g., ambiguous temporal precedence, history, maturation, diffusion) so plausible, alternative explanations for findings can be reasonably ruled out. Commonly accepted designs such as reversal (ABAB), multiple-baseline, changing criterion, and alternating treatment address this quality indicator when properly designed and executed, although other approaches can be accepted if study authors justify how they ruled out alternative explanations for findings or control for common threats to internal validity.”

“6.8. Overall attrition is low across groups (e.g., <30% in a 1-year study).”

“6.9. Differential attrition (between groups) is low (e.g., ≤10%) or is controlled for by adjusting for noncompleters (e.g., conducting intent-to-treat analysis).”

“7.2. The study clearly defines and describes measurement of the dependent variables.”

“7.3. The study reports the effects of the intervention on all measures of the outcome targeted by the review (p levels and effect sizes or data from which effect sizes can be calculated for group comparison studies; graphed data for single-subject studies), not just those for which a positive effect is found.”

“7.4. Frequency and timing of outcome measures are appropriate. For most single-subject studies, a minimum of three data points per phase is necessary if a given phase is to be considered as part of a possible demonstration of experimental effect (except when fewer are justified by study author due to reasons such as measuring severe or dangerous problem behaviors and zero baseline behaviors with no likelihood of improvement without intervention). For alternating treatment designs, at least four repetitions of the alternating sequence are required (e.g., ABABABAB; see Kratochwill et al., 2013).”

“7.5. The study provides evidence of adequate internal reliability, interobserver reliability, test-retest reliability, or parallel-form reliability, as relevant (e.g., score reliability coefficient ≥ .80, interobserver agreement ≥ 80%, kappa ≥ 60%).”

“7.6. The study provides adequate evidence of validity, such as content, construct, criterion (concurrent or predictive), or social validity.”

“8.2. The study provides a single-subject graph clearly representing outcome data across all study phases for each unit of analysis (e.g., individual, classroom, other group of individuals) to enable determination of the effects of the practice. Regardless of whether the study report includes visual or other analyses of data, graphs depicting all relevant dependent variables targeted by the review should be clear enough for reviewers to draw basic conclusions about experimental control using traditional visual analysis techniques (i.e., analysis of mean, level, trend, overlap, consistency of data patterns across phases).”

“8.3. The study reports one or more appropriate effect size statistic (e.g., Cohen’s d, Hedge’s G, Glass’s ∆, η^2) for all outcomes relevant to the review being conducted, even if the outcome is not statistically significant, or provides data from which appropriate effect sizes can be calculated.”