Results from the Youth Inclusion Program – Atlantic Canada

Results from the Youth Inclusion Program – Atlantic Canada PDF Version (269 KB)

Table of contents

• Introduction
• Program Description
• Evaluation Objectives
• Evaluation Methodology
• Outcomes to be Measured
• Outcomes Evaluation Findings
• Reporting
• End Notes

Introduction

The Youth Inclusion Program (YIP) implemented in Canada is a neighbourhood-based program that aims to reduce youth crime and anti-social behaviour by creating a safe place where youth can learn new skills, take part in activities with others, and receive educational support. The YIP was developed in 2000 by the Youth Justice Board as part of a national strategy for proactively tackling youth crime in England and Wales. Three trials of the YIP are being implemented in Atlantic Canada with funding from the National Crime Prevention Centre (NCPC): the Northside YIP in North Sydney, Nova Scotia (January 2010–April 2013); the Seeds of Change YIP in Spryfield, Nova Scotia (April 2010–October 2013) and the ONE Change YIP in St. John, New Brunswick (September 2010–October 2013).

There was enough evidence to deem the YIP to be a promising program.^{Footnote 1} However, more evidence is needed to determine, by using more rigorous methods, the YIP's ability to achieve its outcomes in a variety of socio-cultural contexts and with different target groups. Consequently, the National Crime Prevention Centre (NCPC) has contracted the firm NRG to conduct a multi-site impact evaluation of the YIP implemented in these three different locations.^{Footnote 2} This summary^{Footnote 3} provides an overview of the evaluation study valued at $472,884, which began in August 2010 and will end in January 2014.

Program Description

Consistent with the goals of the original UK YIP, the objective of the three YIPs included in this evaluation is to reduce criminal behaviour and substance use among at-risk youth. Specifically, the program aims to:

• Decrease risk factors and increase protective factors;
• Increase school attendance and school performance;
• Reduce youth offending; and
• Reduce the number of youth in the criminal justice system.

The YIP is a geographically based program aimed at reducing crime in specific neighbourhoods. This is achieved by identifying and targeting the 50 youth most at risk of offending in a geographically defined neighbourhood with high rates of crime. The target youth range in age from 11 to 20. Individualized action plans are implemented for each youth, targeting the youth's specific risk factors. Activities include a combination of one-on-one case management sessions and group activities such as life skills/training, mentoring and tutoring (peer or otherwise), outings, and youth and parent/guardian activities (in conjunction with community partners). In addition, the program connects participants to community resources by referring individuals to outside agencies.

Two of the three sites (the Northside YIP and the One Change YIP) expect to serve 50 youth per year, for a total of 150, whereas the other site targets 60 youth over a three-year period. Taking into account participant loss due to program attrition and non-consent to participate in the evaluation, the evaluators anticipated that approximately 38 participants per year will participate in the evaluation at each site, for a total sample size of 342 youth over the three-year evaluation period. According to the most recent estimates, the projects have been successful in reaching the targeted groups both in terms of risk level and age range (95% or more of youth at each site are within their target age range at admission). The Northside site recruited 58 youth, the ONE Change site 78 participants, and the Seeds of Change 60 youth, with an average age at entry of 14.65 (Northside), 13.25 (ONE Change) and 15.74 (Seeds of Change).

The average number of days participants spent in programs to date is 336 for Northside, 292 for ONE Change and 260 for the Seeds of Change site.^{Footnote 4} On average, youth are expected to receive 5 to 10 hours of intervention per week (i.e., approximately 500 hours per year) with approximately half of these hours spent with the YIP and the other half spent engaged in other community services. Actual average weekly hours per youth has been 3.1 for Northside, 3.7 in One Change and 1.2 for Seeds of Change. The percentage of participants who met the target dosage of 5 to 10 hours a week was 20% for Northside, 36% for ONE Change and 2% for Seeds of Change.^{Footnote 5} Drop-out rates for the program have been 19% in Northside, 5% in ONE Change and 17% in Seeds of Change.

Evaluation Objectives

The NCPC contracted the firm ^eNRG Research Group to conduct this multi-site impact evaluation of three YIP projects.^{Footnote 6} The impact evaluation study, valued at $780,000, started in January 2010 and will end in January 2014. The objectives of the evaluation are as follows:

• Assess whether the intended outcomes were achieved and whether there were any unintended outcomes;
• Identify lessons learned, exploring what has worked well in the program and what has not worked as well, and make recommendations to strengthen the project;
• Examine the relevance of the program by determining the extent to which the project/model corresponds to the needs of the youth and the community;
• Assess the extent to which the project is being implemented as intended; and
• Determine the cost of the program and conduct a cost-effectiveness analysis if feasible.

Evaluation Methodology

A quasi-experimental pre-post design with experimental and control groups was initially planned for the evaluation. However, as a result of difficulties in identifying a suitable comparison group of sufficient size, the design was changed to a single-group repeated-measures design for each site. The quantitative component of the evaluation now consists of a comparison of data collected pre-intervention, during-intervention and post-intervention. The collection of follow-up data at one-year post-intervention is also planned. The data came from a variety of sources, including some components of the ONSET (a structured referral and assessment tool) and official school and police records. The qualitative component of the evaluation consists of key-informant interviews based on a semi-structured instrument and conducted with participants, parents, program staff and community stakeholders to enhance the interpretation of quantitative data.^{Footnote 7}

Despite best efforts, it has been too difficult for all three sites to re-assess the participants every six months using the ONSET protocol; moreover, some participants were never re-assessed and data was often missing for those who were. For instance, re-assessment data is available for approximately half of participants in both Northside and ONE Change, while no data is available for participants at the Seeds of Change site. A careful analysis of missing data will be carried out to determine if the data available is valid and reliable to be representative of the group at each site. At this stage, available data should be considered very preliminary and extreme caution is needed in its interpretation, particularly since sample sizes are very low for many of the questions and not likely representative of the entire group. While memorandums of understanding are in place with appropriate schools and police agencies, data is not yet available to answer the intermediate and long-term outcomes evaluation questions. It is anticipated that school and police data will be available for subsequent evaluation summaries.

About 12 risk factors were assessed to determine if a youth was sufficiently high-risk to be suitable for the YIP program.^{Footnote 8} Professional judgement on the part of the program staff and the advisory committee was used to establish risk level. At this stage, the analyses are entirely descriptive. Heat maps are also used to gain a better understanding of the variability of the data both within and between sites. Once a better understanding of the missing data and a larger sample size are available, inferential statistical analysis will be conducted. It is extremely likely that missing data is not missing completely at random. Appropriate statistical tests will be used for conducting within-site and between-site analyses when adequate data will be available.

Outcomes to be Measured

The evaluation examines changes in participant risk factors over the life of their participation in the program. Intermediate outcomes to be measured include school absenteeism, anti-social and delinquent behaviours, as well as improvement in school performance. Over the long term, the evaluation aims at measuring a decrease in contacts with the criminal justice system and in criminal offending.

Outcomes Evaluation Findings

Decrease in risk factors

According to preliminary data, many participants noticed an improvement in their behaviour across time and reported having made progress towards achieving their intervention targets over time. The percentage of youth whose parents-guardians have noticed an improvement in the youth is 35% in Northside (n=17) and 92% in the ONE Change site (n=13). About 65% of youth in Northside (n= 26) and 95% (n=22) at the ONE Change site have noticed improvement in their behaviour. The percentage of youth who self-report having made progress towards achieving their intervention plan and targets is 92% in Northside (n=13) and 79% at the ONE Change site (n=43). The percentage of youth whose total score was reduced between assessment and reassessment-closure is 37% in Northside (n=27) and 94% at the ONE Change site (n=35). However, care should be taken when comparing values across sites because there is a large amount of missing data.

It appears that there is a considerable amount of variability and fluctuation in risk factors across time (approximately 12 months) for many of these youth. A heat map reveals that, at the Northside site, while some participants improved across a number of risk domains over time and reduced their risk assessment total score, others did not. The ONE Change's heat map is more uniform showing more positive improvements for most participants, with very few participants increasing in risk over time. At both sites, parents and participants indicate noticing positive improvements over time.

According to preliminary data revealed in these heat maps, favourable change has been noticed in the ONE Change YIP for 20–25 participants (out of 35) on risk factors such as emotional and mental (25), family and personal relationships (24), school and education (21), lifestyle (21), and attitudes toward offending (20). There has also been a favourable change for 10 to 16 participants regarding risk factors such as physical health (16), perception of self and others (15), thinking and behaviour (14), motivation to change (14) and substance use (10). Unfavourable changes were sometimes recorded (for 1 to 4 participants) for all risk factors but emotional and mental, school and education. Preliminary results available for the Northside YIP show there are never more than 9 participants who have had a favourable change for any of the risk factors (out of 27 participants). Most of the participants have had no change or have had an unfavourable change.

Cost analysis

A descriptive cost analysis was done for the Northside and the ONE Change sites, but basic requirements weren't met to conduct such analysis at the Seeds of Change site. There are variations in cost by site: the Northside YIP reported $740,757.96 in funding and in-kind resources and the ONE Change reported $683,605.10 over two years. The average cost per participant across the 2010–2011 and 2011–2012 fiscal years was $8,323.12 for the Northside site and $5,842.78 for the ONE Change site. The average cost per hour of dosage was $80.23 in the first case and $56.89 at the second site. A cost effectiveness analysis will be conducted later if feasible.

Reporting

The evaluation study will produce annual reports in April 2013 and 2014 as well as a final evaluation report in January 2015.

For more information or to receive a copy of the final evaluation report, please contact the National Crime Prevention Centre by e-mail at ps.prevention-prevention.sp@canada.ca.

If you wish to register for the NCPC mailing list to receive information from the Centre, please visit the subscription page at: http://www.publicsafety.gc.ca/cnt/bt/mlng-lst-eng.aspx

Date modified:

2022-08-02