The Achievers: Positive Alternatives to Youth Gangs (PAYG)

The Achievers: Positive Alternatives to Youth Gangs (PAYG) PDF Version (231Kb)

Table of contents

• Introduction
• Program Description
• Evaluation of the Program
• Evaluation Findings
• Lessons Learned and Recommendations
• Conclusion
• End Notes

Introduction

Toronto's Jane-Finch community suffers from one of the highest violent crime rates in the province of Ontario and is widely acknowledged as one of the most socially and economically disadvantaged communities in Canada. It is believed that the Jane-Finch community has the highest concentration of youth gangs in Canada, with well-known gangs such as the Bloods and Crips.

Researchers from the University of Toronto and officials from the City of Toronto have developed the Youth Crime Risk Index, a tool to identify neighbourhoods with a high risk of youth gang activity. The index demonstrates that Jane-Finch has the highest risk score in Toronto. This indicates that the community suffers from high crime rates, socio-economic disadvantage, and residents have limited access to community programs for youth.

Toronto Youth Crime Risk IndexFootnote 1

Image Description

The comparative percentages of crime factors versus socio-economic factors in regards to Toronto Youth Crime, are as follows:
Jane-Finch - Socio-economic factors equal 34.8% and crime equals 27.4%
Jamestown - Socio-economic factors equal 25.1% and crime equals 27.4%
Weston - Mt. Dennis - Socio-economic factors equal 18.0% and crime equals 29.5%
Kingston - Galloway - Socio-economic factors equal 10.9% and crime equals 31.5%
Scarborough Village - Socio-economic factors equal 7.6% and crime equals 34.5%
Eglinton East-Kennedy Park - Socio-economic factors equal 10.8% and crime equals 26.0%
Malvern - Socio-economic factors equal 15.0% and crime equals 18.7%
Flemington Park - Victoria Village - Socio-economic factors equal 9.6% and crime equals 22.6%
Dorset Park - Socio-economic factors equal 10.8% and crime equals 20.5%
Crescent Town - Socio-economic factors equal 6.3% and crime equals 24.6%
Lawrence Heights - Socio-economic factors equal 9.6% and crime equals 20.3%
Steeles - l'Amoreaux - Socio-economic factors equal 10.9% and crime equals 13.7%
Westminster - Branson - Socio-economic factors equal 0% and crime equals 20.2%

Given all of these risk factors, youth who grow up in this community are especially vulnerable to gang membership. In 1999, there were a few local programs that dealt with gang members and gang-related issues, but no programs were available for middle-school youth.

In 1999, managers of the San Romanoway high-rise buildings began consultations with key stakeholders to discuss gangs, violent crime, and vandalism in the Jane-Finch area. The Positive Alternatives to Youth Gangs (PAYG) project was proposed as a possible solution to these problems.

In 2008, the National Crime Prevention Centre (NCPC) contributed approximately $196,000.00 to implement and evaluate the PAYG. The project was launched in July 2008 and programming was completed in March 2011. The project served youth from two middle schools in Toronto's Jane-Finch neighbourhood. PAYG was modelled on cognitive behaviour theory, wraparound, social learning theory, a participatory learning pedagogy, an anti-oppression framework, an anti-racist philosophy, and a case management approach.

PAYG was implemented by several stakeholders, including the San Romanoway Revitalization Association in cooperation with Brookview and Elia middle schools and various community partners.

Program Description

The San Romanoway PAYG project aimed to prevent high-risk minority youth from joining street gangs or coming into conflict with the law. It also aimed to help youth, already in gangs, to exit their gangs safely. PAYG provided programs to youth and also brought together stakeholders – San Romanoway residents, two middle schools, and various community partners – to equip them with the knowledge and tools they needed to implement effective gang prevention activities in their community.

The project contained five core components, which are as follows:

School-based group program

In-school groups

This component used active and participatory learning methods to engage youth. Groups explored a range of psycho-social issues that affected the students' academic performance, their life choices, their propensity toward gang involvement and other forms of anti-social behaviour. Topics included self-esteem, violence, problem-solving and conflict resolution, life skills and communication, gender roles, sexuality, bullying, racism and culture shock, drugs and alcohol, and family and community violence. It also addressed gang resistance strategies, gang exit strategies, sexual harassment, and mental health.

One-on-one support

Staff met with those students who did not attend the in-school groups and/or who required additional individual support.

Mentoring

The goal of mentoring was to foster personal psycho-social development of at-risk youth and to provide positive alternatives.

Grade 9 and 10 student follow-up

The objective of the follow-up was to assist graduates from the project with their transition to high school, to provide any required tutoring, and to provide a support network to help them remain in school.

Gang prevention school assemblies

A series of educational workshops on gang prevention were provided to students in Grades 6–8. This allowed the project to reach a broader audience, including all students at each school, in these three grades.

After school program

The after-school program aimed to build trust and rapport with students, help them advance academically, build self-esteem through culturally-based activities, develop a wide range of pro-social skills, and become involved in pro-social activities.

Summer program

This component provided educational, social-recreational, sports and arts activities for project youth. Activities included life skills workshops centered on healthy living, an arts-based program with a different theme each week, sports, swimming, recreational activities, outings, and field trips.

Family support program

Parents and caregivers of the participants in the in-school group, after-school and summer programs were eligible for family support, although the primary focus was on parents whose children were in the intensive stream. By engaging parents, the PAYG program aimed to strengthen their capacity to keep their children gang-free, keep them attached to school, maintain their engagement in skill-building programs, and teach them effective parenting strategies.

Community program

Residents who were directly impacted by gang activities were invited to community forums, with about 50 participants at each event. The forums were also open to community groups and organizations located in the Jane-Finch area. The objectives of this service included creating awareness in the community and among families of the dangers of becoming involved in gangs, and of the factors that place minority youth at risk. It also aimed to identify prevention strategies, and to build community capacity to better address safety, crime, and gang issues. Partnerships with other community groups were developed as part of the community referral process.

PAYG targeted youth (aged 11 to 14 in Grades 7 and 8), their parents, and members of the Jane-Finch community. A total of 170 primary participants participated in the intensive and contact streams. The following is a summary of the different streams:

Intensive stream (n=101)

Students registered in the project who met the target profile and participated in the in-school groups. They received an average of 147.8 hours of services. Most evidence in the evaluation was based on this group.

Contact stream (n=69)

Students at both schools who attended the after-school program. They received an average of 90.4 hours of services.

Students from the community (n=34)

Students who attended the summer program

Parents (n=120)

Parents of intensive stream and contact stream students. They received an average of 3.3 hours of services.

In total, PAYG served 170 youth, of which 34 had graduated by the time of this evaluation.

The project focused most of its time and resources on the 101 intensive stream participants who were primarily male (61.4%), in Grade six (45.5%) or seven (41.6%), were of Black/African Canadian descent (82.2%), from a single-parent family (60.4%), and who spoke English as the main language in their home (90.1%).

Most participants in PAYG were not considered to be heavily involved in youth gangs, criminal activity or substance abuse. This is likely because the target group was below the age when youths are typically exposed to gang activity. Nonetheless, 52.5% of the participants in the intensive stream had intimate contact with gangs or had been involved in group-based deviant behaviour, while 9.9% claimed to have been a gang member. The initial risk assessment tool showed that just 57.4% of participants met the minimum risk criteria for admission into the program. However, after this tool was revised with a broader concept of risk, 87.1% met the minimum risk criteria.

The project delivered approximately 654 hours of services per month. However, it did not have a set length, as there was no firm start date. Once referred to the program, participants started at any time during the school year or during the summer months if they wished to attend the summer program. Youth could remain in PAYG as long as they were attending one of the two participating middle schools. The project also provided follow-up services for the 34 youth who had graduated from the project and started high school. Only 5% (5 of 101) of the intensive stream participants dropped out of the program.

Evaluation of the Program

The project evaluation began in January 2009, but in January 2010, a second evaluator assumed responsibility and conducted the evaluation until March 2011.Footnote 2

The outcome evaluation was conducted using a single group repeated measures design to determine if there were any changes in the outcomes of interest. Initially, the evaluators attempted to use a proxy comparison group used by the Ministry of Children and Youth Services in Ontario. However, the youth in this database were significantly different from the youth in the project in terms of criminal history and risk levels. This made the comparison group less feasible, and less likely to help attribute project effects. Consequently, the evaluators did not use the proxy comparison group in the evaluation.

The evaluator employed standardized survey instruments, but modified them to be more culturally sensitive. Their research tools included:

Pre-test (assessment/baseline) interviews with project participants

During the initial interviews with participants, the evaluators collected data on a large range of risk factors and outcomes. These interviews took, on average, one hour to complete.

Post-test (follow-up) interviews with project participants

To assess project impact, during this interview participants were asked to answer questions that mirrored the initial pre-test interview.

Parent questionnaires

Individual outcomes for project participants were also measured through a questionnaire administered to parents and/or guardians. The parents/guardians who were involved in PAYG were seen as participants, not as informants with respect to their child's behaviour.

Staff questionnaires

All staff were asked to complete a brief exit survey in January 2011. This questionnaire was designed to gather further information on the staff members' thoughts about the project and their recommendations to improve it.

Official school records

An alternative strategy for tracking participants' progress was to incorporate data from official school records. However, access was limited to intensive stream students and was not fully incorporated.

The client tracking form

Both evaluators developed client tracking forms. Project staff were to use these forms to record client attendance, how many hours of services they received, and any comments or observations that project staff wanted to make about participants.

Field observations and review of project records

Qualitative data was gathered through field observations. The aim was to document how well the project was being developed and delivered, and to acquire additional information on the number of youth being served, services being accessed, attrition rates, staffing issues and the project budget.

Participant evaluations of community training sessions

To evaluate the effectiveness of the program's community education efforts, the new evaluation team developed an evaluation questionnaire that was administered to participants at the Making it Real conference in May 2010. Respondents were asked about the overall effectiveness of the conference, the effectiveness of the speakers and what the participants may have learned about local gang activity and gang prevention. Respondents were also asked how to improve the conference in the future.

Participant evaluations of the summer program

In July 2010 the evaluation team administered an evaluation questionnaire to 23 of the 34 students enrolled in the PAYG summer program. This questionnaire asked them to discuss their general satisfaction with the summer program, its benefits and how it might be improved.

The project participants completed pre, post (6-months) and follow-up (12-months) surveys. There were 84 cases available for a pre-test/post-test analysis of project outcomes, with an attrition rate of 16.8%. Of the 17 respondents who did not complete a post-test interview, 15 had moved out of the target community, and two had dropped out of PAYG after a short period of involvement. For the follow-up measure, there were only 37 responses. Statistical tests used to analyze the data included chi-square, one-way analysis of variance (ANOVA), and linear regression.

Evaluation Findings

Process Findings

There was no rigorous fidelity assessment conducted to test if the project was implemented as planned, however, some mixed findings offer a perspective on project fidelity. Based on impressions, most staff felt that the project had been implemented as planned.

The risk assessment tool would normally show whether the project was reaching its target audience, serving as a key project fidelity tool. However, due to disagreements over the appropriate target audience for PAYG, the risk assessment tool was changed during the program, from an instrument that integrated a strict set of gang risk factors to a broader set of risk factors.Footnote 3 Consequently, the initial risk assessment tool showed that 57% of the program's intensive stream clients met the original risk criteria for project admission.

This low result suggests that the project was not effectively targeting at-risk youth, as 43% of participants were not at risk. However, the second risk assessment tool showed that 87.1% of participants met the criteria. Given that the risk assessment tool changed during the evaluation, it is not possible to assess whether the project improved in its ability to target appropriate youth over time or if the higher percentage was simply the result of the expanded risk criteria.

In total, the project cost $1,179,553 between August 2007 and January 2011. It supported 290 participants, and spent $4,067, on average, per participant. When these costs were examined in light of the three streams, the project spent an average of $8,083.98 per intensive stream student, $4,945.77 per contact stream student, and $182.27 per parent.

Outcome Findings

Attitudes

Pro-social attitudes

The project improved participants' attitudes toward civic responsibility. Youth in the moderate-risk category reported the most change, followed by those in the low- and high-risk categories. Analyses were conducted using an ANOVA to compare the mean score in improvement in civic responsibility and awareness by level of risk. Youth in the moderate-risk category reported the most change, followed by those in the low- and high-risk categories. The mean change scores were significantly different across the three risk categories (F(2, 81) = 3.508, p < .05) which indicates that the project made a contribution to increasing pro-social attitudes, especially for moderate- and low-risk youth.

Attitudes toward cheating

Mixed evidence showed some favourable changes, however, the project did not change attitudes toward cheating. Half of the youth in the low and moderate dosage categories reported an improvement in attitudes toward cheating, compared to 60.7% of youth with the higher education-specific dosage. While these results are in the expected direction, the chi-square test reveals that the observed differences are not statistically significant (χ²(2) = 0.862, p = .650).

Attitudes toward violence

Evidence suggests that the project did not directly contribute to any reduction in attitudes toward violence. An ANOVA demonstrates that an improvement of attitudes toward violence does not vary significantly by client risk level or dosage (F(2, 81) = 0.554, p = .577).

Risk and Protective Factors

Academic performance

There were mixed results with some favourable directional increases for certain grades, although overall, there was no significant change. Nonetheless, when this data is further analyzed by dosage, those with varying levels of dosage have similar outcomes. This suggests that the project may not have contributed directly to this change. Chi-square tests reveal no relationship between educational dosage levels and changes in grades over time (χ²(2) = 2.767, p = .251).

Teacher bonding

Mixed findings suggest that youths' interest in teacher bonding did not increase.

Employment

Although the relationship between the level of project exposure and changes in employment potential was not statistically significant, trends in the results suggest that the project may have helped youth improve their employment potential. Youth with high overall project dosage were more likely to demonstrate an improvement in employment potential (82.1%), followed by youth with low dosage (67.9%) and those with moderate dosage (53.6%). While the relationship between project dosage and change in employment potential over time was not statistically significant by conventional levels (p < .05), the findings suggest that high dosage youth are more likely to report improvements in employment potential over time than youth who receive low and moderate amounts of project services (χ²(2) = 5.240, p = .073). Qualitative results also illustrate that the youths' interests in pro-social employment may have contributed to a positive directional change between the pre and post tests.

Behaviour

Association with deviant peers

With mixed evidence, PAYG may have impacted participants' associations with deviant peers. On the one hand, pre-test/post-test analyses revealed statistically significant reductions in the participants' level of association with negative friends and/or friends who influence them to do “bad things”. After participating in the project for six months, it also appears that the participants were more likely to enjoy the company of friends who influenced them in a positive way. After being in the project for at least six months respondents were also less likely to report having friends who are in a gang or who do “gang-like” things. On the other hand, the findings show that lower-risk groups increased their relations with deviant peers.

Substance abuse

Overall, the results suggest that the vast majority of respondents have never used alcohol, cigarettes or illicit substances. During the pre test, for example, only 22 of the 84 respondents (26.2%) reported that they had used alcohol, illicit drugs or cigarettes during the past six months. With no statistically significant changes, it appears that the project did not influence participants' substance use for those that had self-reported alcohol or drug use.

Parental perception

The results of the evaluation revealed that participants' parents held favourable perceptions about the program. The majority reported that the project has improved their child's behaviour, both at home and in school. The majority also felt that the project had contributed to an improvement in their child's academic performance.

Community

Increasing community knowledge about gang prevention

The vast majority of community participants were very satisfied with the Making it Real conference and would recommend it to others, to increase knowledge about gang and crime prevention.

Evaluation Limitations

The primary limitation was the lack of a comparison or control group. To compensate for this weakness, the evaluators used triangulation of data sources to increase confidence in the findings. Their triangulation compared individual-level outcomes through parent questionnaires, staff member questionnaires, review of participant case files, and field observations.

Two separate evaluation consultants carried out the evaluation at different times. This situation brought many challenges to the evaluation, such as contrasting views on how to conduct the study, changes to core measurement instruments and differences in how project staff recorded data. These factors significantly complicated data analysis.

Additionally, the PAYG project found it challenging to determine the precise end date, making it difficult for the evaluators to articulate project dosage. This diminished the ability of the evaluation to isolate the program's contribution to the outcomes. Despite these complications, the study had many strengths. However, a more rigorous evaluation design is required to draw more confident conclusions about the impact of the project on youth.

Lessons Learned and Recommendations

Program Delivery

The project would benefit from one dedicated staff member to work with parents and another to deal with follow-up for students who have graduated from the program. A smaller staff-to-student ratio would ensure better service delivery and more one-on-one counselling. A better gender balance is needed among project staff. PAYG has the scope to expand its mentoring services. Thus, adult and peer mentors could both help the program, potentially recruiting youth who previously completed the project to serve as peer mentors.

Once a consensus has been reached with respect to the project screening tool, youth should not be admitted to the project unless they meet the minimum risk criteria. A detailed explanation should be required when youth are admitted who do not meet the minimum risk threshold. Youth may be in a high-risk neighbourhood but not all youth should be presumed to be at a stage where they can benefit from an intervention program. Admitting youth that are not at risk or are “too low” of a risk will result in a trend that may erroneously conclude that the project was ineffective.

In order to expedite the process of school referrals, it is crucial for project staff to meet with teachers to fill out the risk factors checklists. All teachers and school officials need to be fully aware of PAYG to help fill its quota of participants early in the school year. This would maximize the potential benefits of the program.

Branding PAYG as a “gang” project prevented several parents from allowing their children to participate. Emphasizing the crime prevention benefits for students and families, rather than gang prevention, needs to be central to the social marketing of PAYG.

The transition to high school is particularly challenging for students who are at risk. It is crucial for the project staff to meet early in the school year with clients who are graduating to high school. This will allow the staff to assist recent project graduates with any difficulties they may be encountering while transitioning to high school.

The project referral, screening and admission processes should take place in September, each school year. Then the project should develop a firm start date for the delivery of project activities, possibly in the first week of October. All participants should start at approximately the same time. This process will ensure that all participants are exposed to different project components at approximately the same time. Additionally, the project model should adopt a standard dosage that each client must meet before they can graduate from PAYG.

Evaluation

Agreement between the project staff, evaluator and the project funder is particularly important for establishing the target group and an appropriate screening or risk assessment process for selecting participants. The target group and screening/risk assessment procedure must be established before the project goes into the field. Moreover, stakeholders need to reach consensus regarding the project's model, target group, screening process, and evaluation protocol.

Each target population is unique. Some tools that work well in some social contexts may not work well in other settings. For example, survey questions designed for older youth may not be appropriate for younger youth. There is a need to pre-test research tools to ensure that they are appropriate for the target population with respect to age, ethnicity, risk level, and neighbourhood context.

Before the project is launched, there needs to be a strategy for producing an appropriate control group. Additionally, interviews with the control group should take place at the same time as interviews with the project participants.

When staff fully understand the purpose and importance of the evaluation, they are more likely to cooperate and facilitate the evaluation process. This understanding can be achieved by providing staff with evaluation training.

Evaluators need to spend time with the project staff. The presence of the evaluator is needed to build trust and cooperation between the evaluation team, project staff, and participants. Members of the evaluation team should spend time “on site” on a frequent basis. Additionally, they should hold monthly meetings with project staff to address any evaluation problems and ensure that evaluation tasks are being completed on time.

Utilizing project staff to do evaluation activities should be limited. The time staff spend on evaluation-related activities takes away from service delivery to participants. It can also create unnecessary tension between the project staff and members of the evaluation team. Most, if not all, evaluation activities should be conducted by members of the evaluation team. This will reduce demands on project staff members while protecting the privacy of participants who may not want to express their true feelings, attitudes or behaviours to the staff.

Conclusion

Although participants were not heavily involved in youth gangs, criminal activity or substance abuse, there is evidence that some gang-related risk factors were reduced after program participation. The project was very highly rated by participants, their parents, staff and members of their community. The project demonstrated an impact on pro-social attitudes, though it did not impact on attitudes toward violence. There were also mixed findings regarding PAYG's impact on risk and protective factors, academic performance, teacher bonding and attitudes toward cheating, although it may have had an impact on participants' employment potential.

Due to methodological limitations - such as the lack of an appropriate comparison group and correlation anomalies between project dosage and positive outcomes - the evaluators suggested labelling PAYG as promising. A more robust evaluation methodology is required for it to reach the status of a proven program.

Date modified:

2022-07-28