The Alternative Suspension (AS) program, which aims to reduce criminal activity by increasing youth attachment to school, was evaluated. The evaluation sites were in Chilliwack (British Columbia), North/West Edmonton (Alberta), and Moncton (New Brunswick). The target group consisted of students 12 to 17 years old, who were experiencing difficulties in their academic and social life, and had been suspended or at risk of being suspended. A pre-post design with a non-equivalent control group was used. The evaluators collected data and also used school data and data collected by the implementation agency. Youth who were more likely to experience better behavioural outcomes had been referred to AS for substance use (29 times more likely), criminal behaviour (14 times more), or physical or verbal violence (12 times more). At the end of the school year, 75.2% of completers and 56.3% of the control group had at least one positive outcome. Overall 45.0% of AS completers successfully finished all their courses; 44.1% of these youth, and 28.8% of the control group “met or exceeded school academic expectations”. Overall, 59.0% of program completers and 38.1% of the control group improved their school behaviour. There was a decrease in disciplinary actions for 61.5% of completers and 39.6% of the control group. Around 3.8% of program completers and 14.3% of the control group had dropped out of school at the end of the school year. The average cost per completer was $1,340 in Moncton, $2,107 in Edmonton, and $1,693 in Chilliwack. Net average costs per positive outcome varied from $8,852 to $10,818, and marginal costs from $5,002 to $7,238.
Author's Note
The views expressed are those of the authors and do not necessarily reflect those of Public Safety Canada. Correspondence concerning this report should be addressed to:
The author would like to acknowledge Malatest Program Evaluation & Market Research; the YMCAs of Québec, the YMCA of Greater Vancouver (Chilliwack), the YMCA of Northern Alberta (North/West Edmonton), and the YMCA of Greater Moncton for their participation in the evaluation; the YMCAs in Abbotsford and Surrey (Greater Vancouver), the YMCA of Western Newfoundland (Corner Brook), The YMCA of Greater Halifax/Dartmouth, the YMCA of Cape Breton (Glace Bay), the YMCA of Exploits Valley (Grand Falls-Windsor), the YMCA of Moose Jaw, the YMCA of Regina, The YMCA-YWCA of Winnipeg (Seven Oaks and Winnipeg), South Edmonton (Northern Alberta), The YMCA of Newfoundland and Labrador (St. John's), for the control group; schools and school boards for their data. This research report is based on the evaluation report prepared by Malatest Program Evaluation & Market Research.
Introduction
Program Description
The Alternative Suspension (AS) program was first introduced by the YMCAs of Québec in 1999 in response to a request from a local school in Montréal. The organization was granted funding from Public Safety Canada (PS) to develop multiple sites across Canada in 2009–2015.Footnote1 The ultimate goal of the intervention is to reduce criminal activity by increasing youth attachment to school. It is expected that the participants will be deterred from dropping out of school, complete their academic year, have fewer disciplinary actions, and increase their participation in pro-social activities. The firm Malatest Program Evaluation & Market Research was contracted to conduct the multi-site impact evaluation of AS, valued at $612,675.00. Program sites located in Chilliwack (British Columbia), North/West Edmonton (Alberta)Footnote2, and Moncton (New Brunswick) participated in this evaluation. The target group of AS consists of students aged 12 to 17 years who are experiencing difficulties in their academic and social life in a recurrent or sporadic manner, and who are or have been previously identified at risk of being suspended.Footnote3 A structured out-of-school setting is provided by the program, with a pro-social focus, in which participants’ needs and development are addressed through supported homework time, tailored workshops, one-on-one discussions, and referrals. Students are also offered activities to help them resolve existing conflicts and change the negative attitudes and behaviours that led to their suspension. The AS youth worker develops a personalized plan and a school reintegration agreement with the student, who also receives a list of personal and educational resources.
Evaluation Objectives
The multi-site evaluation of the AS program aimed to:
Assess the extent to which AS was being implemented as intended;
Assess if the intended outcomes were achieved, and identify any unintended outcomes;
Determine if the program had been adapted to meet the needs of youth/community;
Identify lessons learned and make recommendations;
Provide a descriptive cost analysis and conduct a cost-effectiveness analysis.
Evaluation Methodology
The AS evaluation was based on a pre-post design with a non-equivalent control group developed from three categories of youth: 1) students who were accepted to the program but never participated; 2) students from other AS service sites who were accepted to the program but never participated; 3) non-completers from the evaluation sites and additional AS sites, who started the program but left early.Footnote4 The control group was not used for the short-term outcomes measures. The evaluators used program records and data collected by the youth workers with tools developed by the YMCAs of Québec.Footnote5 Some schools provided administrative data for AS participants and youth in the control group.Footnote6 A participant exit survey developed by the evaluators was administered by the youth workers at the end of student’s participation.Footnote7 Focus groups were completed with youth in 2012, as well as with AS coordinators and youth workers in 2014. Furthermore, 180 key informant interviewsFootnote8 were conducted during the 2011–14 school years and 21 for a follow-up of 17 youth from the 2013–14 cohort conducted in 2014–15. Footnote9 Descriptive statistical analysis included means, medians, modes, frequencies and cross-tabulations. Inferential statistical techniques were used to identify any significant differences in outcomes, between groups or sites (chi-squared tests, t-tests, logistic regression, and analysis of variance). Inductive content analysis of qualitative data was completed to identify areas of concurrence and divergence.
Results
Program Fidelity
Program fidelity was assessed by examining whether the intended target group was reached and the extent to which the implementation adhered to the essential elements of the AS model. The actual number of valid school referralsFootnote10 across the service sites (n=662) exceeded the expected target (n=540) over the three-year period (see Table 1). Six-hundred fourteen (92.7%) of all valid school referrals resulted in participation, while 91% of the 614 participations resulted in graduation from the program.Footnote11 The program was used by AS staff and school administrators as a corrective response to inappropriate student behaviour, such as a traditional form of suspension. AS was also used as a preventative measure to correct disturbing behaviour. These applications were in line with the referral strategies and represented the types of participants that were anticipated. Over 96% of participants were between the expected ages of 12 and 17Footnote12 (n=213), and 74% of participants (n=492) were classified as high risk by their school’s administrators.Footnote13
Table 1: Numbers of AS participants and referrals by site and total for the 2011–14 school years
School Years 2011–14
Chilliwack
N./W. Edmonton
Moncton
Total
Valid school referrals
190
223
249
662
Valid school referrals that did not result in participation
11
24
13
48
Total AS participants
179
199
236
614
Participants who did not complete the program
15
32
8
55
Total AS participants who graduated
164
167
228
559
Source: Essentials Form Data 2011–2014. Including multiple referrals for some students.
Program fidelity was assessed by examining the extent to which the delivery of the intervention adhered to the seven essential elements of the AS program model (see table 2). The first two criteria were fully met while the other ones were only partly applied. It should be noted that there was generally a regular communication between the program staff and the schools. In general, reintegration meetings were also held.
Table 2: Fidelity Assessment
Seven essential elements of AS
Fidelity assessment
1) Neutral location
The program took place outside of the school
2) Tailored intervention offered to small groups of youth (no more than 6)
A tailored intervention directly addressing the cause of the suspension was offered to small groups of youth
3) Balanced intervention including school work, group workshops, one-on-one meetings, and complementary activities
- 71.6% of AS participants who graduated (n=555) completed 75% or more of their assigned schoolwork
- 67.6% of youth participated in group workshops and 44.6% in complementary activitiesFootnote14 (n=614)
- 12.2% of completers (n=559) resulted in referrals to external programs/services, 42.4% included extracurricular activities and 0.7% educational outings/field trip
- 46.7% of completers had 100% of program service day involving a one-on-one meeting/individual counselling sessions (n=394)
4) Channel of communication
- AS staff were engaged in regular and ongoing communication with schools for 98.5% (n=458) of valid school referrals
- Parental presence on the first day of the program was low (35% of the time)
- Most parents who were interviewed reported that they were sufficiently informed about the program by schools and youth workers
5) Minimum of three days suspension standard (21 hours) to develop a relationship between the student and the youth worker
- The student’s suspension by the school usually reached between 3 and 5 days in 76.9% of the cases (n=614)
- It was 15 to 18 days for 1% of youth who had a long history of behavioural problems and delinquency
- The average number of suspension days was 4.7
6) Accompanied return to school involving all relevant parties (student, school administrator, youth worker, and parent/guardian)
- Reintegration meetings occurred in 92.7% of participants (n=559)
- All relevant parties were present at 41.3% of these meetings
- Parent(s)/guardians participated in 42.9% of the reintegration meetings
- About 54.7% of completers participated in the reintegration meetings, and youth participation was significantly lower (χ2=6.09, p=.01) in Moncton (44.7%, n=228) than in Chilliwack (54.3%, n=164) or N./W. Edmonton (68.9%, n=167)
7) Youth follow-up
Youth workers followed-up with 87% of participants (n= 614) to review their progress and reintegration to school
Program Outcomes
Short-term outcomes (up to 4-6 weeks following the program)Footnote15
Overall, while some students who participated in AS saw their reintegration plan as a useful tool which facilitated their return to school, others felt that the school administrators and teachers did not provide the support planned in their reintegration plan. Approximately 87% of students who completed the reintegration assessment (n=384) “somewhat agreed” or “strongly agreed” that they were ready to return to school, while readiness to return to school varied significantly by site (χ2=6.80, p<.05).Footnote16 The control group did not participate in AS, so these youth did not have a AS reintegration plan.
Students who participated in the program reported strong levels of understanding the reason for their suspension both immediately following return to school, and 30 days after returning to school. Around 89.4% of students (n=207) stated that they “somewhat understood” or “really understood” the reason for their suspension. There were significant between-site differences (χ2=8.11, p<.05).Footnote17 Understanding of the reason for the suspension showed a statistically significant increase between reintegration and follow-up (t = 2.024, p<.05), demonstrating that youth were becoming more reflective.
A majority of students who participated in AS “strongly agreed” that they learned something new at AS in the North/West Edmonton site (63%, n=146) and Moncton site (54.1%, n=135), while 56.7% of students at the Chilliwack site (n=104) “somewhat agreed” that they learned something new. Those site differences are significant (χ2=7.00, p<.05). Again, the control group members did not participate in AS, so the evaluators could not assess what they have learned from the program. However, data was collected to measure the results of the program. The evaluators made between-site comparisons. Across sites, 49.6% of students who completed the program thought that what they learned would be “somewhat useful” to their school careers, and 42.1% that it would be “very useful” (n=382). There were significant site differences: 62.5% (n=104) of completers in Chilliwack and 52.6% (n=133) in Moncton found what they learned was “somewhat useful”, while in the third location, 52.4% (n=145) of students believed what they learned was “very useful” to their school careers (χ2=12.17, p<.01). About 49.6% of all students found what they learned during the program “useful” to their life outside school, compared to 42.6% who said that it was “very useful” (n=383). The majority of students at the North/West Edmonton (51.7%, n=145) site believed that it was "very useful" to their lives outside of school. At the Chilliwack (60.6%, n=104) and Moncton (50.7%, n=134) sites, the majority of students found what they learned as "somewhat useful". Between-site differences were significant (χ2 =7.00, p<.05). Youth perception of usefulness of AS to their school careers and other aspects of their lives showed no significant change between their reintegration to school and the follow-up.
All youth who participated in AS noted an increase in self-esteem and/or confidence when the long-term follow-up data was being collected. AS helped them to learn more constructive and appropriate behaviours. They seemed to have greater insight into their behaviour, greater willingness and capacity to reflect on their actions and motivations. School staff noted that many participants who graduated had a tendency to return to school with a more collaborative approach to managing and dealing with problem behaviours, such as anger management techniques. School administrators believed that the youth had gained more control over their emotions following their participation in the program, and were better able to express their feelings. According to students, parents, and school administrators, students made better decisions and showed better self-regulation, which helped them to avoid the issues for which they were initially suspended.
Based on the school’s assessment, youth behaviour (n=462) improved significantly between the referral and the period 30 days after their participation in AS (t=5.287, p<0.01). Students’ (n= 319) academic performance also improved significantly (t=24.835, p<0.01). The mean change in behaviour ratings was small (0.25, on a four-point rating scale), and the mean change in academic ratings was more substantial (1.10, on a four-point rating scale). Among the completers (n= 80) who were re-suspended after their return to school, there was no statistically significant change in the severity of their reason for suspension.Footnote18 Completion of the program had a significant positive effect on youth getting caught up on their schoolwork, and improvements being maintained for at least 30 days after returning to school. Based on youth self-reports, there were significant changes about being caught up on schoolwork (F = 27.090, p<.01). There were significant differences in schoolwork before AS and upon reintegration (t = 9.402, p<.01) or at the follow-up (t = 6.298, p<.01). Only 23.6% of students (n=411) reported participating in extracurricular activities at the follow-up survey; the rate of participation of Chilliwack youth (47.9%, n=73) was significantly higher than North/West Edmonton (15.8%, n=120)Footnote19 and Moncton (19.7%, n=218).Footnote20 There was no evidence that AS increased interest in extracurricular or pro-social activities, although some youth, parents, school administrators and staff, AS coordinators and youth workers, said that there was an increase.
Medium-term Outcomes (up to 9 months after the program)
Across sites, 45% of AS completers (n=411) completed all of their courses for the school year. The North/West Edmonton site (n=120) had the highest proportion of students completing most or all of their courses (80.0%). Chilliwack (n=79) had the lowest percentage (68.3%).Footnote21 In Moncton, the percentage reached 70.3% (n=212). Based on data from the schools, only 17.7% of the program participants (n=401) did not meet academic expectations, compared to 40.4% of the control group (n=52). In addition, 44.1% of the program group “meet or exceed expectations”, and 28.8% for the control group.Footnote22 About 54.2% of completers at the North/West Edmonton site (n=118) and 43.3% at Moncton (n=201) were most likely to meet or exceed expectations, and youth at the Chilliwack site (n=82) to somewhat meet expectations (31.7%).Footnote23
Overall, 59% of completers (n=395) improved their school behaviour by the end of the year, and so did 38.1% of the control group (n=21) who improved their school behaviour.Footnote24 Completers at the Edmonton site (n=120) were most likely to show improvement (71.7%), and students at the Moncton site (n=194) were least likely to do so (50.5%),Footnote25 as were youth in Chilliwack (60.5%). A third (33.2%) of AS completers (n=325) were re-suspended by the end of the school year, and 18.2% of completers (n=439) were expelled or transferred to another school.Footnote26 The majority of program completers (61.5%, n=322) were reported by school administrators to have had a decrease in the number of disciplinary actions, compared to 39.6% of the control group (n=48).Footnote27
In order to determine if the AS program was more successful with some categories of students than with others, logistic regression was used. Several demographic student variables were included as independent variables while the dependent variables were behaviour, academic, and disciplinary outcomes at the end of year. Completers who were more likely to experience better behavioural outcomes were referred for substance use (29 times more likely), criminal behaviour (14 times more likely), or physical or verbal violence (12 times more likely).Footnote28 Completers from a visible minority group were three times more likely to have a positive change in academic outcomes than Caucasians.Footnote29 Female completers (odds ratio = 0.44) were significantly less likely than males to have an increase in disciplinary measures. When parents were present at the first day of youth participation, there were poor disciplinary outcomes less than half as frequently (odds ratio= 0.39).Footnote30
Long-term outcomes (a year following the participation in the program)
Across the sites, 54.6% of AS completers (n=317) and 45.8% of the control group (n=48) were registered at the same school for the following year; this difference was not significant, which suggests that AS did not increase the likelihood to remain at school. Only 3.8% of completers (n=504) and 14.3% of the control group (n= 119) had dropped out of school by the end of the year. A chi-squared test found a significant difference (χ2 = 23.00, p<.01). From the follow-up cohort, 16 out of 17 youth were promoted to the next grade.
Cost-Analysis
Methodology
The project financial reports and youth records were used for the cost-analysis of the program. Only the costs that were associated with program maintenance and use were included in the calculations. The cost-analysis was limited since the costs were not being directly tied to outcomes, whereas the actual program delivering involves a number of costs that were not easily definable or divisible. The cost-benefit analysis explored the costs of AS in the context of improvements to students’ well-being during their year of participation, based on four metrics: end-of-year outcomes,Footnote31 behaviour, academics, and disciplinary actions.
Cost of the AS program
The Moncton site received $418,515 in direct and indirect funding, the North/West Edmonton site $447,951, and the Chilliwack site $508,476.Footnote32 The amount of funding and cost of delivering varied between sites, which could be attributed to local factors, the involvement of community partners, and differences in the costs of service space and staff. The average costFootnote33 of the program per participant was $1,279 in Moncton, $1,735 in North/West Edmonton, and $1,598 in Chilliwack. The average cost of the program per graduation was $1,340 in Moncton, $2,107 in North/West Edmonton, and $1,693 in Chilliwack. The variation in the average cost per participant is primarily driven by the between sites difference in the number of students referred, participating and completing the program. The wide range of average costs also indicates significant regional differences in salaries and costs for spacing, and are not due to the methods employed for program delivery at the three sites.
The cost of adding another participant was $846 in Moncton, $1,190 in North/West Edmonton, and $1,055 in Chilliwack. The cost of adding one more participant who completed the program was $886 in Moncton, $1,446 in North/West Edmonton, and $1,118 in Chilliwack.Footnote34How the schools used the program contributed to the differences in costs.Footnote35 The level of effort (3,659 days), calculated based upon the number of participants and how long program resources are used, was 1,572 days in Moncton, 966 in North/West Edmonton, and 1,145 in Chilliwack. The average cost per day ($355) was $250 in Moncton, $447 in North/West Edmonton, and $422 in Chilliwack. The marginal cost per day ($237) was $165 in Moncton, $307 in North/West Edmonton, and $278 in Chilliwack.
Cost-Benefit Analysis
The students’ status at the end of the school year was examined (see Table 3). Nearly 75.2% of youth who participated in AS and 56.3% of the control group had a positive outcome at the end of their school year. About 60.4% of the completers improved their behaviour, as did 38.1% of the control group. Moreover, 44.3% of AS completers and 28.8% of the control group met academic expectations. Disciplinary actions decreased for 57% of AS students and 38.2% of the control group.
Table 3: Student Outcomes at End-of-Year (School Years 2009–10 to 2013–14)
Outcomes
AS Completers
Control group
End-of-Year Outcomes
n= 723
n= 119
Positive outcome
75.2%
56.3%
Negative outcome
24.8%
43.7%
Improving Student Behaviour
n= 594
n= 21
Behaviour improved
60.4%
38.1%
Behaviour deteriorated or stayed the same
39.6%
61.9%
Meeting Academic Expectations
n= 594
n= 52
Student’s academics meet expectations
44.3%
28.8%
Student’s academics did not meet expectations
55.7%
71.2%
Disciplinary Actions
n= 412
n= 55
Decrease in disciplinary actions
57.0%
38.2%
No change or increase in disciplinary actions
43.0%
61.8%
The net average cost per positive outcome and the marginal cost per positive outcome at the end of the year are shown in Table 4.
The median income of an individual with a high-school education is $5,044 per year higher than that of a high-school drop-out (Gilmore, 2010), and there is a $757 difference per year in taxable revenue. Assuming an entry in the workforce after graduation from high school, the Canadian government would recover the AS expenditure within 12 years. High school dropouts are 3.5 times more likely to be arrested than a graduate (Sum et al., 2009) and education attainment is closely linked to good health, lower death rates, and fewer risky behaviours (Freudenberg and Ruglis, 2007).
Table 4: Net Average and Marginal Cost per Positive Outcome at End-of-Year
(School Years 2009–10 to 2013–14)
Outcomes
Overall
End-of-Year Outcomes
Average Cost Per Positive Outcome ($/Outcome)
$8,938
Marginal Cost Per Positive Outcome ($/Outcome)
$5,900
Improving Student Behaviour
Average Cost Per Positive Outcome ($/Outcome)
$7,471
Marginal Cost Per Positive Outcome ($/Outcome)
$5,002
Meeting Academic Expectations
Average Cost Per Positive Outcome ($/Outcome)
$10,818
Marginal Cost Per Positive Outcome ($/Outcome)
$7,238
Disciplinary Actions
Average Cost Per Positive Outcome ($/Outcome)
$8,852
Marginal Cost Per Positive Outcome ($/Outcome)
$5,923
Discussion
The AS model is in direct response to crucial needs of the participating communities, such as high drop-out and juvenile delinquency rates, and criminal activities amongst youth. The program addresses school-related risk factors by providing a short-term intervention for students who have experienced at least one risk factor. The major components of the intervention are seen as best practices by empirical research, including: one-on-one counselling; development of a warm relationship and rapport between the youth and project staff; opportunities for parental involvement; life skills, anger management, and conflict resolution sessions.
Thirty days after their participation in AS, youth behaviour improved, and significant changes were noticed in being caught up on schoolwork at the reintegration meeting and at the short-term follow up. The program did not increase youth participation in extracurricular or prosocial activities 30 days after the participation. At the end of the school year, no significant differences between the two groups were measured regarding being expelled or transferred to another school. However, at that time more AS completers than youth from the control group had met school academic expectations and improved their behaviour at school. Their decrease in the number of disciplinary actions was significantly higher than for the control group. AS appeared to have positive effects on students; however, these outcomes were likely strongly moderated by the school context. Moreover, the intervention was more successful with students referred for substance use, criminal behaviour, physical or verbal violence, and youth from a visible minority. No significant effect was noticed specifically regarding re-suspensions and expulsionsFootnote36; however, more students from the control group had dropped out of school by the end of the year. AS could likely play a role in intervening in students’ trajectory from suspension to dropout.
The average cost of the program per completer was $1,340 in Moncton, $2,107 in North/West Edmonton, and $1,693 in Chilliwack. The cost of adding one more completer was $886 in Moncton, $1,446 in North/West Edmonton, and $1,118 in Chilliwack. These are small costs considering costs related to criminal activity. The cost of delivering AS was found to be indirectly recoverable through the difference in federal income taxes paid by individuals with a high school education and those who dropped out. Within 12 years after a student graduated from high school, the higher federal income taxes were sufficient to offset the cost of the program.
Malatest Program Evaluation & Market Research. (2015). Multi-site Evaluation of the Alternative Suspension Program. Final Evaluation Report. December 16th, 2015, submitted to Danièle Laliberté, Public Safety Canada.
In the case of 2) and 3), youth came from AS sites in Abbotsford, Corner Brook, Dartmouth, Glace Bay, Grand Falls-Windsor, Moose Jaw, Regina, Seven Oaks, South Edmonton, St. John's, Surrey, and Winnipeg.
Tools included the Essentials Form, Daily Tracker and Checklist, Reintegration Assessment and Agreement, Participation Summary Tool, and Short-term follow-up and end-of-year questionnaire.
Key informant interviews were conducted in 2011–14 with youth (n=58), parents/guardians (n=35), AS youth workers (n=13) and program coordinators (n=11), school administrators (n=35), school authorities (n=4), school teachers and counsellors (n=8), partners/community organizations and agencies (n=16).
Students from the Moncton and North/West Edmonton sites who were referred in 2013/14 and were also referred or attended the AS program again in 2014/15 (n=17). They were not typical of AS participants (most attended the program only once), and the sample size was small.
Calculations are based on valid school referrals that resulted in at least one day of participation in AS. Some students attended the program several times.
The target group was broadened to capture 31 students identified as “non-functional in the school environment”. Some students were not reached because they were detached in their school environment, lived far from service sites, or deterred by the program’s association with "suspension".
Complementary activities included a wide range of activities that occurred outside of the AS program and during the afternoons, such as basketball, badminton, board games, cooking classes, discussion groups, facilitated conversations, icebreakers, videos, and sport of weightlifting.
For both first and subsequent suspensions, students were typically suspended for an action that warranted a rating of approximately 4 on a 0 to 7 point severity scale.
Non-program costs (development, other sites) were $136,623 in Moncton, $190,588 in North/West Edmonton, and $220,688 in Chilliwack. The evaluation costs were $196,212 in Moncton, $191,379 in North/West Edmonton, and $219,079 in Chilliwack.
The calculation of average and marginal cost per participant and graduation excluded the second quarter of the 2014–15 fiscal year, as these costs were not associated with program maintenance or use.
Evaluation of the Alternative Suspension Program PDF Version (1.2 MB)