Results from the Stop Now and Plan (SNAP) Program

Results from the Stop Now and Plan (SNAP) Program PDF Version (227 KB)

Table of contents

• 1. Introduction
• 2. Program Description
• 3. Evaluation Objectives
• 4. Evaluation Methodology
• 5. Evaluation Research Limitations
• 6. Outcomes to be Measured
• 7. Measurement Tools and Data Collection
• 8. Interim Results
• 9. Reporting
• End Notes

Introduction

Stop Now and Plan (SNAP^®) is a community-based program for children (under the age of twelve) who have come into contact, or are at risk of coming into contact, with the criminal justice system, and who display early signs of anti-social or aggressive behaviour.

The program uses a cognitive-behavioural, multi-component approach to decrease the risks of children engaging in future delinquent behaviour. The SNAP^® model is based on a comprehensive framework for effectively teaching children with serious behavioural problems, emotional regulation, self-control and problem-solving skills. Parents also learn SNAP^® skills, as well as cognitive-behavioural parenting techniques. Children learn how to stop and think in order to find solutions to resolve their problems. Although there is evidence regarding the effectiveness of SNAP^® in Canadian contexts (within accredited mental health centres), further evaluation is needed to assess the impact the program has in a variety of community-based organizations across Canada. This summary provides an overview of the multi-site impact evaluation of SNAP^® that is being funded by the National Crime Prevention Centre (NCPC).^{Note 1}

Program Description

SNAP^® targets boys and girls between the ages of 6 and 12, an age when cognitive and behavioural functioning are most impressionable. One of the criteria for admission to the program is involvement in antisocial and/or delinquent behaviour. These children typically fall within the top two percent of aggressive and delinquent behaviour, such as bullying, rule-breaking, and conduct problems. This empirically based, multi-component model matches levels of risk/need to levels of treatment intensity.

There are two core program components. The first component, the SNAP^® Boy's Group or the SNAP^® Girl's Club, is a 12-week gender-specific program that teaches impulse/self-control and problem-solving skills.

The second component is the concurrent SNAP^® Parent Group that teaches parents effective child management strategies. Other program components based on level of risk/need include individual counselling/mentoring, family counselling, academic tutoring, youth leadership and a gender-specific component called “Girls Growing Up Healthy.”

The NCPC contributed approximately $10 million to fund nine SNAP^® programs across Canada (including process evaluation costs).

Target Group

Table 1 provides a summary of the risk levels of children from the pooled results for the Toronto and Edmonton sites. The table indicates that slightly less than three quarters (70%) of the children enter the program at a moderate or high risk level. This suggests that the projects are reaching children that are within clinical range, have prior police contact and have an increased likelihood of offending if they do not participate in effective treatment.

Evaluation Objectives

An independent firm was contracted to conduct a multi-site impact evaluation of SNAP^®.^{Note 2} The study, valued at $875,000, started in 2010 and will end in 2014. The objectives of the evaluation are as follows:

• Incorporate process evaluation information and assess the extent to which the project is being implemented as intended;
• Determine whether the intended outcomes were achieved and identify any unintended outcomes;
• Provide a descriptive cost analysis for each project and determine feasibility in order to conduct a cost effectiveness analysis;
• Identify lessons learned and recommendations; and
• Assess the extent to which each project has been adapted to meet the needs of the youth and the community.

This evaluation is assessing the efficacy of SNAP^® in three project locations: Toronto, Edmonton and a Cree Nation community in Quebec. Staff at all sites were trained to implement the full SNAP^® model (including core and additional components). These programs were delivered in both urban and rural settings, with Aboriginal and other populations, thus offering an opportunity to evaluate SNAP^®'s external validity. The sites also provide a large sample size, offering the potential for a rigorous evaluation design.

A cost effectiveness analysis will be conducted for the Edmonton and Toronto sites. The results will respond to the question, “What does it cost to have an effect on scales measuring change in the SNAP^® program?”

Evaluation Methodology

The initial evaluation design proposed a unique methodology for each site:

• The Cree Nation community site is using a matched comparison group with an estimated 63 participants per year;
• The Edmonton SNAP^® site is using a matched comparison group involving a delayed treatment group with 63 estimated participants per year;
• The Toronto site is using a repeated measures design with an estimated 60 participants per year.

Due to challenges in finding a comparison group, a delayed treatment group (DTG) was used for all three sites. Once the pre and post measures were administered on the waiting list sample, the participants were then eligible to start the SNAP^® program. This compromise means that the NCPC is unable to compare long term differences between the SNAP^® children and the DTG group at the 6-month and 12-month follow-up measures. All staff at the SNAP^® sites confirmed that the children did not receive any SNAP^®-related treatment during the DTG period. Project staff has also indicated that the children did not participate in any comparable treatment activities during the waiting list period.

Both the experimental and DTG group were the same at baseline, indicating that the DTG group could serve as a feasible comparison group for the evaluation research study.

The evaluators are conducting a within-site analysis for each annual report, and a between-site analysis for the interim and final reports. The between-site analysis will determine the feasibility of implementing SNAP^® in a variety of settings.

Evaluation Research Limitations

The data in this report should be considered preliminary, as the evaluation study is still in progress, indicating that conclusions about program attribution should only be formulated at the end of the projects in 2014.

Due to the use of the DTG, the NCPC will only be able to make short-term comparisons (over a three month period) between the experimental and DTG group (children on the waiting list who did not receive the program). The sites have confirmed that children on the waiting list did not receive any SNAP^® treatment. The Edmonton site, for example, rigorously monitored the external services provided to the children on the waiting list. Just under one third of children on the waiting list participated in assessments and “one-off” visits to a mental health facility, social worker and school counsellor. A few children in DTG groups may have received some minimal service, but none of these services were comparable to the dosage provided in the SNAP^® program.

The sample size in the DTG groups is relatively small (n=18 and n= 9), reducing the reliability of the between-group analyses.

The SNAP^® sites being evaluated are NCPC funded community-based programs operating in a non-clinical environment. All of these sites were operating different programs prior to being funded by the NCPC. In reviewing the results in this summary, it should be noted that the sites had a relatively short period of six months to learn and implement the SNAP^® model. Scientific literature regarding implementation indicates that it takes approximately two to four years to effectively implement an evidence-based model program.

Outcomes to be Measured

Targeted outcomes include decreases in externalizing behaviours (rule-breaking, aggression, conduct, oppositional and attention problems) and co-morbid internalizing behaviours (anxiety and depression), and increases in pro-social behaviours (e.g., competency). For parents, other outcomes to be measured include improved child management strategies, reduced family risk factors, and improved relationships with their children.

Measurement Tools and Data Collection

The evaluators are using both quantitative and qualitative research instruments. The evaluation team uses the required standardized instruments that have been used and tested extensively. The research tools and approaches include the Child Behaviour Checklist (CBCL) and the Teacher Report Form (TRF). These measures include behavioural scales that measure rule breaking, aggression, conduct and oppositional problems, in addition to social competency scales that include strength-based measures that assess changes in social engagement, academic performance and social performance at school. The Total Problems Scale includes internalizing/externalizing behaviours, social problems, thought problems and attention. The DSM scales include measures related to affective problems, anxiety, oppositional defiant problems and conduct. Risk levels are assessed by the Early Assessment Risk Lists (EARL-20B and EARL-21G) developed by the Child Development Institute (CDI).

Program staff will administer some of the instruments, the evaluators will administer others, and the program developers (CDI) will participate in the fidelity audits. CDI created the initial fidelity tool. The evaluators further developed it by converting it into an instrument more suitable for multivariate analysis. The evaluators will also try to obtain school suspension data to supplement the survey instruments. Dr. Pepler notes that teachers' assessments demonstrate expected changes at the six-month phase and not immediately after the program. One explanation is that teachers may not be as sensitive as parents to changes in children's behaviours (Pepler et al, 2010). NCPC staff also noted that children often had more than two teachers during the assessment periods, making it difficult to achieve continuity.

Data is being collected at the pre-program stage, and post-group at 12 weeks, 6 months, and 1 year after admission. Annual follow-up measures will be sought, with the final follow-up being conducted 1 year after the project end date.

The evaluators will employ multi-level modeling using two statistical approaches. They will employ a variable-oriented approach that describes the relationships between variables, followed by a person-centred approach focused on the relationships between individuals.

In analyzing repeated measures data, individual differences in changes over time are typically captured by random effects using mixed modeling (the multilevel model for change). The advantage of using mixed-model statistical procedures over the traditional repeated measures ANOVA is that they can use all available information and can provide projections where there is missing data.

Interim Results

The following data provides results based on a within-site and between-site analysis conducted in June 2012. These preliminary results and trends may change in the final report (2014) when the projects have fully completed treatment for all cohorts and when treatment group numbers are high enough to produce statistical power. The following table provides information about the number of measures administered at the pre, post, and follow-up periods. Due to the limited comparison group results with the Cree site, the quantitative results will be shared in 2014. Qualitative information about the ability to reach the appropriate target group and program fidelity will be provided in this interim summary for the Cree site.

Note: The Cree Region site started six months later than the Edmonton and Toronto sites.

Program Fidelity (Edmonton and Toronto sites)

An important aspect of the SNAP^® program is matching risk levels with the appropriate treatment dosage. An analysis of variance (ANOVA) was conducted to examine the difference in the intensity of program services between the three risk groups (see Table 1). The Spearman correlation was used to examine the correlation between the overall risk and the intensity of program services. The hypothesis is that the children exhibiting higher risk levels would receive a greater dosage or intensity of services.

Findings suggest that there was a negative correlation (-0.27) between the overall clinical judgement of the children and the number of children or family group sessions. Higher-risk families and children received less SNAP^® group sessions than their lower-risk counterparts.

A chi-square test was conducted to examine the association between the clinical judgment and other SNAP^® treatment services attended, such as Individual Befriending (IB), Individual Family Counselling (IFC) and School Advocacy. The findings suggest that higher-risk children were more likely to receive other services, such as Individual Befriending (p=0.04) and additional parent counselling (p=.08). There was no statistical correlation between the child's risk level and other services related to School Advocacy and Individual Family Counselling. This latter finding suggests that children with varying risk levels all received the same amount of additional school and family-related services.

The final evaluation summary report will provide results that determine the extent to which program fidelity contributed to changes in the measures being tested in the SNAP^® program. To date, the sites have been reaching the appropriate target group and implementing the program as planned; however, improvements are needed in matching the child's risk with the appropriate treatment dosage.

Edmonton Site: Uncles and Aunts at Large

The results of the mixed models for each parent-rated CBCL scale indicates that children show significant decreases in almost all scales after completing the SNAP^® program. In Edmonton, the effect sizes for both externalizing and internalizing behaviours illustrate that SNAP^® participants made moderate to high-level reductions in such behaviours. Externalizing problems, including rule breaking (ES=0.61), attention (ES=0.40) and aggression (ES= 0.54) behaviours, show moderate reductions, with effect sizes ranging from 0.40 to 0.61. Internalizing behaviours, including anxiety (ES=0.68), withdrawal (0.27) and somatic (ES=0.58) behaviours, also show low to moderate reductions, with effect sizes ranging from 0.27 to 0.68. When these behaviours are reduced, this decreases the child's likelihood of future contact with police and the criminal justice system.

A review of the Teacher Report Form (TRF) indicates that two subscales showed moderate reductions: attention and withdrawal. The preliminary TRF results indicate that teachers did not see significant reductions in rule breaking, aggression, anxiety and somatic-related measures during the initial phase of treatment (12–16 weeks). It should be noted that literature regarding SNAP^® indicates that the accuracy of assessing changes in children's behaviours takes approximately six months longer than when the assessment is conducted by the parents (Pepler et al, Prevention Science, 2010). This suggests that conclusive statements about teachers' assessments should be considered at 6 months and 12 months post program to increase the confidence in the findings.

Preliminary results comparing SNAP^® program participants to a comparison group indicates that there was a statistically significant difference between the SNAP^® treatment group and the waiting list comparison group (moderate effect size of 0.57) in the Total Competency measure. Total Competency is measured by parent information related to children's engagement in community activities, social skills and academics. Higher scores indicate that children are able to function more competently across these contexts. When the statistical tests are examined on other behavioural subscales, there are no differences between the treatment and comparison groups. Given the low statistical power (N=9) of the three month post-program measurement (T2), these results should be considered preliminary until a larger cohort in the comparison group is used.

Toronto Site: St. Leonard's Society

The results of the mixed-model analysis for each parent-rated CBCL scale indicates that children show significant decreases in almost all sub-scales after completing the SNAP^® program. In Toronto, the effect sizes range from moderate (ES= 0.54) to high (ES= 1.17). Externalizing problems, including rule breaking, attention-seeking and aggressive behaviours, show moderate to high reductions, indicating that the SNAP^® program is achieving its targeted outcomes. Internalizing problems, including anxiety (0.46) and somatic problems (0.40), also show moderate to high level reductions.

A review of the TRF data indicates that aggression showed a statistically significant reduction.

Preliminary results from the TRF and CBCL measures comparing SNAP^® program participants to a comparison group indicate that most of the scales presented no statistically significant differences between the treatment group and the comparison group, however, the TRF DMS Anxiety sub-scale showed a statistically significant reduction for the treatment group, with a moderate effect size of 0.57. Given the low statistical power of the three-month post-program measurement (N=16), these results should be considered preliminary until a larger cohort in the comparison group is used.

Quebec Site: Cree Regional Authority (CRA)

The CRA site started project implementation later than the Edmonton and Toronto sites. As a result, quantitative data for this site will be available in early 2013. This summary will include process evaluation information related to the target group, fidelity and program implementation challenges and successes.

The project has provided treatment to 45 Cree Nation children residing in the remote Cree communities of Mistassini and Waswanipi, Quebec. A majority of the children fit the target group characteristics, including age (6 to 11), externalizing/internalizing clinical levels, and several children (boys) had police contact prior to commencing treatment in the program. The vast majority of children (80%) fit the high-risk category, indicating that the project is reaching an appropriate target group.

Fidelity to the program has been strong in some areas and weak in others. Visual examination of tapes of the children's sessions and observations of the program demonstrate that adherence to the SNAP^® curriculum is quite strong in both communities. To date, approximately 50% (20) have met the minimum requirement of eight sessions, indicating that children's program attendance is lower than expected. With respect to adherence to the parents' program, fidelity is lower due to several factors, including literacy and parental engagement. During program sessions, parents whose first language is Cree found it challenging to understand the curriculum exercises. Parent engagement and attendance has also been low. Key informant information provided indicates that cultural practices that promote early child self-sufficiency may be contributing to the mentality that the children are the only participants required in the program.

In addition to engagement and attendance issues, there have also been logistical challenges. The Mistissini site was originally to be held after class at the local school, but due to school renovations, programs had to be held at the Justice Centre, over a mile from the children's homes. This created logistical challenges in picking up children, diminishing program attendance. This logistical challenge was rectified in the fall 2012 term.

The community of Waswanipi has been short-staffed for some time, which has created pressures from a program delivery perspective. The ability to find qualified staff in small Cree communities is a concern, especially where, for example, the Cree Board of Health is offering similar pay and accommodation for staff. After some adjustments, the program is now fully staffed.

Finally, the Cree sites are struggling to complete the standardized instruments, EARLs, eco-systemic assessments, case files and frequent data entry. Time sheet analyses show that some employees spend 20 hours per week completing the forms and instruments, per the CDI license.

There are some successful aspects to the program that suggest that the challenges may be resolved during the final year of the program. There is strong support from community partners, including schools (to host programs) and referral source partners (schools, social services, parents, etc.). During the initial stages of implementation, the SNAP^® program was stigmatized as being a project for “bad kids”, however this stigma has been significantly reduced, resulting in an increased registration in the fall 2012 session. This increased registration of 48 children will allow the evaluation team to create an additional DTG group to strengthen the between-group analysis.

Cost effectiveness Analysis (CEA)

A cost effectiveness analysis (CEA) was conducted for the Edmonton and Toronto sites. Cost effectiveness uses the cost of the program and impact evaluation results to determine the cost of creating an effect on particular outcomes. During this interim analysis, the evaluation team conducted a CEA on the results related to the Total Competency effect size (as this presented a statistically significant difference between the two conditions) to determine whether it was also economically viable to achieve these favourable results. If competence can be achieved at an efficient rate, it may be feasible for other programs to make changes in children's ability to strengthen their social engagement, academics and other social skills. Total costs were obtained prospectively for a period of 12 months after SNAP^®. Expected costs were estimated on the basis of operating costs for children at different risk levels. Table 3 presents the incremental cost per standard deviation unit of the competency scale for the Toronto and Edmonton sites. A previous program funded by the NCPC (Banyan Community Services SNAP^® Under 12 Outreach Program) was also included as a reference site for further comparison.

Table 3 indicates that the cost per unit gained for Total Competency at the Edmonton site is $22, or 031.00 per child. Given the lower effect size (E=0.17), which is not statistically significant, it costs Toronto almost four times more ($84,688.00) to produce a gain in the Total Competency scale. An additional analysis using the mixed model technique demonstrated that the Toronto site could produce adequate gains in the Total Competency scale at one year post program with a higher effect size of 0.60.

Further uncertainty analysis is required to assess how the variability in program costs, risk levels, and statistical power in the comparison groups may contribute to changes in the cost effectiveness results. The final report will provide CEA results for all of the scales that have statistically significant effect sizes.

Note: The calculation of the CEA was completed prior to the preparation of Table 2. Table 2 provides a current count of participants who have completed evaluation measures as of August 2012.

The CEA analysis indicates that achieving a moderate effect size costs approximately $22,000 per child over a six month period to increase the Total Competency Scale with a moderate effect size. To identify the overall potential value, a cost benefit ratio can be calculated to express the amount of monetary gain realized in relation to the costs. Table 4 presents possible costs that will be incurred as a result of not participating in the SNAP^® program. These potential savings per year ($88,033) and the CEA costs for the Edmonton site ($22,031) were used to calculate the cost benefit ratio. The findings suggest that for every dollar spent on producing a change in the Total Competency scale at the Edmonton site, four dollars is saved each year. A cost benefit ratio greater than 1 means the benefits outweigh the costs, indicating that the treatment investment in this case is financially profitable or, in other words, it saves taxpayers a substantial amount of money that would have otherwise been used in downstream criminal justice system costs.

Note: All costs, with the exception of Ontario closed custody cost estimates, are standard Canadian costs based on the SiMPACT Strategy Group Inc. Financial Proxies

Reporting

Evaluation information and results will be shared in the following reports: an evaluation framework completed in September 2010, annual reports due in October 2011 and October 2012, October 2013, an interim report due in July 2012, and a final report due in 2014.

For more information or to receive a copy of the final evaluation report, please contact the National Crime Prevention Centre by e-mail at ps.prevention-prevention.sp@canada.ca.

If you wish to register for the NCPC mailing list to receive information from the Centre, please visit the subscription page at: http://www.publicsafety.gc.ca/cnt/bt/mlng-lst-eng.aspx

End Notes

1 This summary only presents the methodology. Only the interim results are being reported in this summary.

2 Evaluators: Astwood Strategy Corporation; Technical Authority and Contract Manager: Donna Smith-Moncrieffe, NCPC, Policy, Research & Evaluation.

3 Based on current estimate of Ontario closed custody facility costs which, in some institutions, range as high as $800 per diem, in addition to reported per diem youth closed custody costs in other Canadian Jurisdictions (e.g., Kids, Crime and Care, Office of the Provincial Health Officer of British Columbia, February 2009), $600 chosen as the lower end of cost estimate to house a youth in closed custody.

Date modified:

2022-08-16