Results from the Alternative Suspension Program^{Note 1}

Results from the Alternative Suspension Program PDF Version (199 KB)

Table of contents

• 1. Introduction
• 2. Program Description
• 3. Evaluation Objectives
• 4. Evaluation Methodology
• 5. Findings
• 6. Outcomes Measured
• 7. Reporting
• End Notes

Introduction

The Alternative Suspension (AS) program provides tailored interventions and support to youth who have been suspended from school or are at risk of being suspended in order to address their misconduct and help them persevere with their school work. It is based on a program that was first developed in Montréal by the YMCAs of Quebec in 1999 in response to a request from a local school that had noticed its suspended students would spend their time away from school with no supervision, whereas statistics showed crime rates were highest during school hours, with the most likely offenders being school-aged youth.

A national trial of the Alternative Suspension program is being implemented from 2009 to 2014 by the YMCAs of Quebec in Abbotsford and Chilliwack, BC; Edmonton, West and North Edmonton, Alberta; Moose Jaw and Regina, Saskatchewan; Winnipeg, Manitoba; Moncton and Riverview, New Brunswick; Halifax, Nova Scotia; and Corner Brook, Newfoundland, with $6,799,929.52 in funding from the National Crime Prevention Centre (NCPC).

Based on a previous evaluation of pilot AS projects from 2003 to 2005,^{Note 2} there was enough evidence for the NCPC to deem AS to be a promising program. However, more evidence is needed to assess the ability of AS to achieve its outcomes in a variety of settings, and with different target groups. Consequently, the NCPC is running a multi-site evaluation of AS in three locations: Chilliwack, West Edmonton and Moncton. This summary provides an overview of the multi-site evaluation of AS that is being funded by the NCPC.^{Note 3} The NCPC contracted the firm Malatest Program Evaluation & Market Research to conduct a multi-site impact evaluation of AS. The impact evaluation study, valued at $612,675, began in 2011 and will end in 2015.

Program Description

Alternative Suspension is an out-of-school program for youth facing school suspensions. The program works with youth 12 to 17 years old (during the 2011-2012 school year, median age calculated at 14 for the three sites) who are registered in school and are experiencing difficulties in their academic and social life on a recurrent or occasional basis. It also engages youth at risk of being suspended. AS provides program activities that help youth resolve conflicts and minimize negative behaviours that lead to suspensions. It aims to help them improve their attitudes toward school and their relationships with others. The AS program served 195 youth in the three sites during its first year of implementation (2010‑2011) and 164 in the 2011‑2012 school year.^{Note 4}

Youth in the program participate in group workshops that encompass a wide range of issues associated with at-risk behaviours and individual sessions that focus on their particular needs. The program's topics may include stress and anger management, responsibility, conflict solving, relating to authority figures, personal organization, self-esteem, violence and bullying, street gangs, and drug and alcohol abuse. Morning sessions are reserved for schoolwork; afternoon sessions are for workshops and individual interventions.

Target Group

Overall, the characteristics of AS program participants at the three sites being evaluated during the 2011‑2012 school year match the broad target group criteria. Two distinct groups are recruited, one characterized by higher risk students facing traditional out-of-school suspension, and a lower risk group consisting of students who may benefit from a time out period spent within the AS program as a preventive intervention. Students are being referred for traditional suspension and/or for other behaviours that are not worthy of suspension. However, the youth characteristics suggest all these students are at risk. Indeed, more than one quarter (27.8%) of students were referred to the program because of disruptive and impulsive behaviours, 22.2% for risky behaviour and 20.0% for physical or verbal violence. As shown in Table 1, most participants (95.2%) had experienced some level of behavioural difficulties prior to the program, 67.6% had previously been suspended and 43.9% were at risk of being transferred to another school when referred to the AS program. The majority had received warnings (86%) and disciplinary actions (81.7%), whereas help and support (76.2%) or referrals to services (40.2%) were provided to some participants prior to being referred to the AS program.

Source: AS Program. Essentials Form; Totals may not add to 100% due to multiple responses.

Program fidelity

The impact evaluators assessed the extent to which the implemented intervention adheres to the AS program model. At all three sites, students were hosted in a neutral out-of-school setting and the six-to-one recommended ratio of students to youth workers was rarely exceeded. As part of the fidelity to the program, personalized intervention plans were designed based on information provided by school administrators, youth workers' assessments, and objectives identified by the participants. A balanced intervention was also targeted through a combination of dedicated time to complete school work, group workshops, one-on-one meetings and complementary activities. According to the data, almost half of the participants (48.4%) completed 85% to 100% of their schoolwork, with 26.5% completing 100% of their work. Participation in group workshops has been 95.7%, while 68.3% of program participants received at least one one-on-one session every day, whereas participants received a one-on-one session, on average, 88.4% of the time. Complementary activities, including referrals to other programs/services, recreational activities and educational activities to complement other program components have been held but are not properly tracked in the monitoring system.

Regular communication with the school and parents regarding youth progress and daily activities is also seen as a key component of the program; 65.2% of referred students had some form of contact with the youth worker on day one of the program. The suspension period is typically between three and five days, but could last as long as 15 days to a maximum of three weeks for participants with a long history of behavioural problems and those presenting delinquent behaviours. The duration of the suspension period spent in the program averaged 3.4 days in West Edmonton, 3.8 in Chilliwack and 5.2 in Moncton. Ideally, 90% of program participants would engage in re-integration with the school administrator (and perhaps teacher) accompanied by the youth worker and a parent/guardian. In fact, 78.5% of participants attended a reintegration meeting, and all stakeholders were present only 58.3% of the time. Parental attendance has been challenging across all three sites, happening about 63.2% of the time.

Evaluation Objectives

The objectives of the evaluation are as follows:

• Implement a rigorous assessment of AS, as the prior evaluation did not employ quasi-experimental methods;
• Measure the program's effects on youth who are referred to the program; and
• Increase knowledge as to whether this program can achieve its outcomes with a variety of target groups and settings.

The evaluation takes place in three locations: Chilliwack, BC (site started March 2010), West Edmonton, Alberta (site started May 2010) and Moncton, New Brunswick (site started March 2010). These sites were selected to provide a wide range of geographical, cultural and implementation contexts.

Evaluation Methodology

The evaluation is based on a quasi-experimental methodology that uses a pre-post non-equivalent groups design, including two follow-ups after completion of the intervention (6 months and 12 months). The comparison group members include youth who were referred by schools and who were eligible for the program but did not participate. Due to a small number of youth in the comparison group (16), strategies to expand this group are being explored, including adding non-participants from nearby AS program sites.^{Note 5}

The evaluators use a mixed-method approach, integrating quantitative and qualitative data collected through multiple sources of evidence. For instance, program records and data collected by the youth workers are used, as well as school records, key informant interviews (program staff, parents, school administrators and partners), participant exit surveys, focus groups and interviews with youth. However, certain factors precluded the application of the planned design for the midpoint evaluation report due to the timing of the report and other issues related to data availability. As such, the current summary is mainly based on descriptive data collected by the program at entry and partial data for the short-term AS program follow-up period (four to six weeks). The descriptive analysis methods include the calculation of frequencies, cross-tabulations and means/medians to describe program statistics, participants' socio-demographic characteristics, school-related information and outcomes. Qualitative methods include inductive content analysis to identify convergent and divergent themes, drawing on open-ended questions used in various survey instruments and within each key informant group and across the different groups.

Testing for statistical significance (e.g., repeated measures t-test) will be completed by the evaluators once pre-post data is available to explore the extent to which the difference between pre- and post-program outcomes for school attendance, lateness, and school disciplinary actions are statistically significant for the program participant group. The net impact analysis will rely on more sophisticated statistical techniques (e.g. regression analysis) that will determine the within-site differences between the program participants and the control group members, as well as between-site differences.

FINDINGS

Outcomes Measured

The evaluation focuses on several outcomes, including short-term outcomes such as successful re-integration, learning and application of tools learned in the workshops, and involvement in community and school activities. Medium-term outcomes include completion of the school year, decrease of school disciplinary measures, and increased association in pro-social activities. The long-term outcome is to increase attachments and commitment to school. The ultimate goal is to reduce offending in targeted populations. At this point in the evaluation, it is too early to determine achievement of most medium-term and long-term outcomes since the first school year of the evaluation period was not yet complete when the mid-term report was prepared.

Short-term outcomes

Some preliminary findings suggest that behaviours and attitudes of program participants are influenced by what is learned in the AS workshops and interventions. Key indicators are: positive change in student attitudes; improved skills in problem areas; and positive change in student behaviour. According to youth workers, 67.4% of the AS program participants have given at least some thought (whether seriously or lightly) to their school situations and appeared willing (whether willing or having shown some willingness) to work on improving their behaviour. Their assessment of the participants' progress and status at the completion of the AS program is confirmed by 63.3% of the parents who reported that students have demonstrated either a noticeable improvement (9.2%) or a certain degree of change in their situation (54.1%). School behavioural assessments of the current situation of youth at program follow-up have noted an improvement in 65.7% of students, with 8.6% of students having demonstrated a noticeable improvement and 57.1% a certain degree of change in their situation. About 66.7% of AS participants felt they made some degree of improvement in the time since they were in the program as was noted at program follow-up (noticeable improvement: 54.9%; and significant improvement: 11.8%).

Medium-term outcomes

Once information associated with school-related outcomes is available, the evaluators will measure whether youth are more likely to complete the school year/course. Some preliminary key findings are available regarding the reduction of the occurrence of disciplinary actions for participating youth, subsequent suspensions after participation in the AS program, and school-related behavioural improvements.

School administrators noted that one of the outcomes of the AS program is that there are fewer office referrals and disciplinary actions, and that they have fewer negative interactions with the participants in the program. Overall, 15.3% of AS participants were suspended again after completing the program; however, this rate of re-suspension is only an estimate since data is incomplete at all sites. Suspensions were not reliably recorded for participants who did not complete the program, therefore a comparison cannot be made, and follow-ups were not conducted with participants who did not complete at least one day of the AS program. No AS participants had been transferred or expelled from their schools by the time the short-term follow-up occurred.

Based on the information collected from school administrators by youth workers during the post-program follow-up (four to six week after reintegration in school), AS participants seem to improve their academic performance 39.4% of the time. They also noticed that some students demonstrated a noticeable improvement in their situation (8.6%) or a certain degree of change in their situation (56.2%). About 5.7% of the time, student's behaviour deteriorated.

Cost analysis findings

Cost-effectiveness analysis will be conducted when outcome data is available, while preliminary descriptive cost analysis has been carried out. The cost of delivering the AS program at different sites varies due to local factors, involvement of community partners and location of the program. Total costs have been $500,865.13 ($403,212.65 cash cost) in Chilliwack, $446,190 ($444,825 cash cost) in Edmonton, and $383,997.24 ($319,984.99 cash cost) in Moncton. Implementation costs are $31,713.88 in Chilliwack, $26,156.50 in Edmonton and $30,132.11 in Moncton. The average cost per participant is $6,466.52 ($6,446.74 cash cost) in West Edmonton, $2,829.75 ($2,278.04 cash cost) in Chilliwack and $3,173.53 ($2,644.50 cash cost) in West Moncton based on data from 2009/10 (4 months), 2010/11 (12 months) and 2011/12 (12 months). The higher cost in Edmonton is due to a lower participation rate.

Reporting

The results presented in this summary should be considered preliminary and will be completed as more data become available. The final evaluation report is due October 1,  2015.

For more information or to receive a copy of the final evaluation report, please contact the National Crime Prevention Centre by e-mail at ps.prevention-prevention.sp@canada.ca.

If you wish to register for the NCPC mailing list to receive information from the Centre, please visit the subscription page at: http://www.publicsafety.gc.ca/cnt/bt/mlng-lst-eng.aspx

End Notes

1 The Technical authority for this evaluation contract at NCPC is Danièle Laliberté, PhD, Senior Evaluation Analyst.

2 C.A.C. International. (December 2005). Rapport final d'évaluation du programme Alternative Suspension.

3 This summary presents preliminary analysis included in the interim evaluation report submitted to the NCPC on June  30, 2012. More findings included in the upcoming annual reports and final report will be presented in a subsequent summary.

4 The current summary covers the period from September 2011 to May 30, 2012.

5 At the time of this summary, data for the comparison group was not available.

Date modified:

2022-08-03