Prevention Intervention Toronto (PIT)
Introduction
Youth gangs and gang violence have been an issue within Toronto's socially disadvantaged communities for several decades. Toronto's gang-related deaths peaked in 2003 at 35, and have since ranged between 14 and 30 a year. In 2010, Toronto had the fourth-highest rate of gang-related homicides per capita of any major Canadian city, after Winnipeg, Vancouver and Montreal. In 2008, an in-depth analysis conducted by City of Toronto officials and researchers from the University of Toronto concluded that the gang problems in Toronto were most intense in three neighbourhoods located within the North-West section of the city: Jane-Finch, Weston-Mt. Dennis and Rexdale.
The City of Toronto received 3,956,802 million dollars of funding from the National Crime Prevention Centre (NCPC) to operate Prevention Intervention Toronto (PIT). The PIT program was administered by the City of Toronto and delivered in the community by Jewish Vocational Services (JVS).
The program was implemented between December 2009 and March 31st, 2012.
Program Description
PIT is a 36-week intervention that consists of three distinct phases: 1) Needs Assessment Phase; 2) Group Training Phase; and 3) Integration Phase. The Needs Assessment Phase is conducted during the first eight weeks of the PIT program. During this phase, PIT participants meet with their assigned case manager to identify specific risk and needs factors that form the basis of the individualized program plans. The Group Training Phase lasts 20 weeks where participants receive one-on-one counselling to discuss topics such as gang violence, victimization, education, employment needs, anger management, health issues, financial management, family and peer relations, drug and alcohol use, mental health and personal development. The Integration Phase has an eight week component that allows participants to meet with their case manager for further assistance with respect to achieving program goals. Case managers assist their participants in accessing various community activities that will facilitate their transition to a pro-social lifestyle.
Evaluation of the PIT Program
The evaluation of the PIT was conducted by an evaluation team from the Centre of Criminology and Socio-Legal Studies, University of Toronto. This quasi-experimental study was conducted using a multi-method strategy that involved a repeated measures design and a comparison group of youth who did not receive program services. Follow-up interviews were conducted at post-program (nine months), six-months following the post-program and one-year following the pre-test. The comparison group was selected from a comparable, high-risk neighbourhood located in North West Toronto.
There were some methodological limitations related to the comparison group. The comparison group recorded higher levels of negative behaviours and attitudes compared to PIT youth. To ensure the varying risk levels between the experimental and comparison groups were considered, risk levels were controlled in the multivariate analysis equations. This technique would ensure that we were able to attribute the PIT programs contribution to potential changes in the outcomes being measured.
The attrition rates, specifically in the comparison group, significantly lowered the sample size available for the examination of long-term effects. This reduced the validity of some of the long-term related results measured one year after program completion.
Recording challenges related to service delivery to program participants prevented the evaluation team from accurately determining how program dosage did or did not contribute to changes in the outcomes measures. However, the evaluation team had the type of data that allowed them to assess the amount of service delivered in relation to the expected benchmarks identified in the project's work plan.
A final concern was the fact that the research team was not able to acquire official data on the PIT participants' criminal records. Such data would have allowed researchers to compare the results of the participant and staff member interviews with official statistics and thus improve the validity and reliability of the evaluation results. However, NCPC's ethical review committee and the evaluation team could not guarantee the confidentiality of participants' names, which resulted in the need to forego the use of official police records.
Evaluation Findings
Process Findings
This study focussed on responding to a number of process evaluation questions. Two key questions relate to whether the project was able to recruit the appropriate target group and whether the project was implemented as planned.
Target Group
The program was able to reach the intended target group. PIT targeted youth aged 13 to 24 years who were either gang-involved or at risk of gang-related activity. Youth were referred to PIT from a variety of sources including schools, community organizations, youth workers and parents. PIT case workers also engaged in various community outreach activities in order to identify young people who might benefit from PIT program services. Prior to being accepted into the program, referrals were required to complete a screening process that was designed to determine program eligibility. Out of the 322 youth recruited and subjected to the PIT screening protocol, 312 were accepted into the PIT program. This represents an acceptance rate of 96.9%.
The PIT program was able to recruit the appropriate target group. The results indicate that 35.0% of accepted youth were current or former gang members, 57.2% met the Euro-gang definition of gang involvement and 81.1% met the minimum risk-score threshold for gang involvement. Overall, 87.6% of accepted youth met at least one of these three criteria. The criminal history of PIT participants indicates that they have had moderate to high levels of experience with the criminal justice system. For example, at the pre-test stage participants indicated that in the past six months they had been involved in making physical threats (61.4%); just over half (56%) had been engaged in a physical assault; 39.8% had been engaged in a gang fight; 29.5% were engaged in robbery/extortion and just over half (58.6%) had been arrested at least once prior to the program.
Program Implementation
The findings suggest that there were challenges implementing the program as planned, including the tracking and recording of activities at the individual level. According to the program plan, each PIT participant should receive 12 hours of one-on-one case management time during the Needs Assessment Phase of the program (1.5 hours per week over eight weeks). However, the average PIT participant received just 2.4 hours of case management during this period. Indeed, according to the official JVS tracking data, one-third of PIT participants (32.5%) did not receive any case management services during the Needs Assessment Phase. The required dosage in the needs assessment phase is important for producing individualized program plans for each participant.
During the first and second cycles of the PIT program, the majority of case managers reported that formally documented case management plans (i.e., plans that had been written down) had not been created for their individual participants. During the final cycle of the program, a structured process, including developing, completing and monitoring case plans could not be identified. The focus group findings with staff indicated that there was very little consensus regarding the specific objectives of the needs assessment phase. Some staff felt this phase was more about recruitment than assessment. Others felt that it was about relationship- or rapport- building between the case managers and the PIT youth. Few, however, identified this stage as a period of intensive needs assessment and individualized program planning.
During the training stage of the program, PIT youth were supposed to receive eight hours of group training per week or 160 hours over the entire 20-week period. The official JVS tracking data revealed that the average PIT participant received 29.2 hours of group training (or 1.5 hours per week)- less than 20% of the recommended group training dosage. Overall, none of the PIT participants received the recommended dosage of group training hours. Despite the weaknesses discussed above, most of the youth participants had a positive view of the group training.
The Integration Phase of the PIT program spanned eight weeks and was designed to: 1) respond to the most important participant needs identified during the assessment phase; and 2) phase out reliance on PIT supports by introducing youth to new community resources that could help them achieve their program goals. During this period, PIT participants were supposed to receive 12 hours of one-on-one case management time (1.5 hours per week). However, according to the official JVS tracking data, the average PIT participant received 5.4 hours of case management time during this period.
The focus group findings suggested that there were some program accessibility challenges. One of the site locations did not have the facility for staff to buzz participants into the building, thus creating delays and frequent participant absence. Several youth indicated that they exited the PIT program because they could not access program services during the 9 am to 5 pm period, due to work or school obligations. The evaluators and the City of Toronto program manager emphasized that this contributed to limitations in program implementation. Key informant and focus group findings indicate that the time between the needs assessment and training phase was too short. Case managers were focussed on recruiting new clients, resulting in a diminished focus on assessing clients, developing plans and implementing the full program requirements.
Overall, there were a number of challenges that suggested that the program was not implemented as intended.
Outcome Findings
The results outlined below refer to short- and long-term effects. Short-term effects correspond with measures that were taken post program (9 months after the program started) and long term effects correspond with measures taken at the 6 month and 12 month periods following the post program periods.
Attitudes
Attitudes towards Crime, Violence and Gangs
PIT participants showed an increase in pro-social attitudes towards crime, violence and gangs, while scores for the comparison group showed no change between the pre- and post-test periods. This finding suggests that the PIT program contributed to a positive change in attitudes towards crime, violence and gangs. Respondents with higher risk levels, as measured by the screener, also experienced more significant improvements in attitudes towards crime. It should be noted, however, that this positive effect was maintained over time, as both the experimental and comparison groups experienced the same rates of change six months after program completion.
Attitudes towards the Criminal Justice System
The results indicate that PIT participants experienced a statistically significant improvement in attitudes towards the courts in the short term. This suggests that the program contributed to positively changing thought patterns that contribute to strengthening protective factors. When testing changes between the pre- and the six-month post program follow-up period, there were no statistically significant differences between PIT participants and the comparison group, which indicate that the program's favourable effect is most salient at the program completion period.
Attitudes towards Education
There were no statistically significant differences in attitudes towards education between the PIT participants and the comparison group. Results indicate that the favourable increase in positive attitudes was similar for both groups at the post program and the six-month follow-up measure. These favourable increases cannot be attributed to the program since youth who did not receive the program also experienced an increase in favourable attitudes at similar rates.
Attitudes towards Employment
Based on measures taken prior to and after the program, the data suggests that PIT participants experienced a statistically significant improvement in attitudes toward employment when compared to youth who did not receive the program. These favourable results are not sustained at the 6-month and 12-month follow-up periods.
Risk & Protective Factors
Association with Gang-Involved Peers
Overall, there is no evidence to suggest that PIT participants experienced a statistically significant short-term (six months) or long-term (one year) decline in the number of gang-involved friends in their social network. Although PIT participants did report slightly fewer gang-involved friends over time, similar declines were observed among the comparison group; therefore program attribution cannot be established.
It should be noted, however, that the results also indicate that PIT participants who received high levels of anti-crime programming were more likely to reduce interactions with gang peers than those who received few crime-related program services. This finding provides some support for the efficacy of the PIT program.
Association with Anti-social Peers
The results indicate that PIT participants did experience a statistically significant increase in pro-social friendships between the pre-test and post-test interviews. These increases were greater than increases experienced by members of the comparison group, which indicates that the PIT program contributes to strengthening this peer-related protective factor.
School Attendance, Disciplinary Problems and Academic Performance
The data indicate that, between the pre-test and first follow-up interview, PIT participants were more likely to remain in school (or return to school) than members of the comparison group. Further analysis reveals that education-related program dosage was a significant predictor of school attendance. Between the pre-test and six-month follow-up interview, PIT youth who received a high volume of education-related services were more likely to stay in school or return to school than participants who received relatively few education-related services. The data, however, reveals that the PIT program had no long-term impact on school attendance.
The PIT program also had a positive impact on conduct at school. Between the pre- and post-test interview, PIT youth were more likely to experience decreases in disciplinary problems at school than members of the comparison group. However, long-term differences between the PIT youth and the comparison group could not be identified.
The results also suggest that, in the short term, PIT participants were actually less likely to report improved academic performance than members of the comparison group. This finding is contrary to program expectations. Dosage measures were also unrelated to changes in academic performance over time, which indicates that a greater focus on academic performance during the PIT program does not contribute to favourable changes.
Overall, the evaluation results suggest that the PIT program had a positive impact in the short-term on school attendance and conduct at school, but did not contribute to favourable change in academic performance.
Self-Esteem
Participation in the PIT program was associated with a significant increase in youths' self-reported levels of self-esteem between the pre- and post-test interview. However, further testing at the six month follow-up period indicates that there was no long-term treatment effect. Moreover, the amount of exposure to PIT services did not predict an increase in self-esteem; those who received relatively few PIT service hours improved over the short-term, just as much as those who received many hours.
Family Relationships
The results suggest that being in the PIT program was not associated with general improvements in family relationships. Although family relationships did improve significantly for PIT youth over time, the comparison group experienced similar positive changes. These changes cannot be attributed to the PIT program, however, due to similar rates of change made with the comparison group.
Participation in Pro-Social Activities
This study examined PIT's contribution to a change in involvement in pro-social activities (volunteering, attending church, working on hobbies, playing sports, etc.) for study participants. The results suggest that, over the short term, PIT participants were more likely to report an increase in pro-social activities than members of the comparison group. Differences between PIT participants and the comparison group, however, do not reach statistical significance over the long term. Overall, the results suggest that the PIT program increased participation in pro-social activities at the post program period.
Self-Control, Conflict Resolution and Anger Management
While youths' self-reported levels of hostility declined over time, this improvement cannot be attributed to the PIT program. PIT participants improved just as much as members of the comparison group. Data also suggests that the amount of exposure to PIT services was not related to changes in hostility over time.
The PIT program was associated with a significant decline in youths' self-reported levels of temper between the pre- and post-test programs but there was no long-term treatment effect. Moreover, the amount of exposure to PIT services did not predict a decrease in participants' anger-related levels.
Overall, the results of the evaluation suggest that positive self-control, conflict resolution and anger management were achieved for PIT participants; however, similar rates of favourable changes were also made for the comparison group.
Drug and Alcohol Use
There is evidence to suggest that PIT participants were more likely to reduce their level of marijuana use between the pre-test and post-test interviews than members of the comparison group. It should be noted, however, that further tests indicate that the reductions are unrelated to program dosage. Those that make favourable changes had the same amount of program services as those that made no change in their marijuana use.
The data suggests that PIT participants did not experience a significant decline in getting drunk between the pre-test and post-test interviews. Any favourable changes that did occur between the pre- and six-month follow-up period were similar to the trends experienced by the comparison group.
Data collected on dosage indicate that very few PIT services targeted issues of drug and alcohol use. During group training, for example, only one session per site addressed substance abuse issues. This might explain why there was relatively little program impact on drug and alcohol use.
Behaviours
Criminal Victimization
The results indicate that the PIT participants experienced a statistically significant decline in criminal victimization between the pre- and post-test interviews. In gang studies, the victimization rate is another indicator of a youth's increasing desistance of gang involvement. This rate of decline is significantly greater than the rate of decline observed among the comparison group. Overall, the evaluation results suggest that the PIT program did help reduce criminal victimization among program participants in the short term. These findings were not maintained at the one year follow-up period.
Criminal Offending
PIT participants experienced a greater short-term decline in non-violent offending than members of the comparison group. Although PIT participants also experienced a significant long-term decline in non-violent offending between the pre-test and six month follow-up period, this rate of decline did not significantly differ from the rate of decline experienced among the comparison group.
Further analysis indicates that total program dosage and crime-related program dosage are not related to declines in non-violent offending over time. In other words, those who received fewer PIT program services were just as likely to experience a decline in non-violent offending as those who received many services.
PIT participants also experienced both a short-term and a long-term decline in violent offending. During both time periods, the rate of decline among PIT participants was significantly greater than the rate of decline observed among the comparison group. This is consistent with program expectations. Further analysis indicates however, that neither total program dosage nor crime-related program dosage are related to declines in violent offending over time. In other words, PIT participants who received a low dosage of PIT services were just as likely to experience declines in violent offending as those who received a high dosage. Being part of the program matters, but the level of involvement does not.
Overall, the results of the evaluation suggest that the PIT program did, in fact, contribute to reductions in criminal offending among program participants. The results are relatively stronger with respect to violent offending.
Contact with the Justice System
The results suggest that PIT participants experienced a slight decline in self-reported arrests between the pre- and post-test periods. There was a large, statistically significant decline in arrests between the pre-test and 6 month follow-up period.
Further analysis reveals that both the short-term and long-term declines in arrests experienced by PIT participants did not statistically differ from those experienced by the comparison group over the same time period.
Gang Involvement
The evaluation results indicate that there was a statistically significant decline in gang membership among program participants over the study period. For example, while 34% of PIT participants admitted gang membership during the pre-test interview, this figure dropped to only 9.0% during the one year follow-up interview. However, despite a statistically significant decline in gang membership among PIT participants between the pre-test and six month follow-up periods, the rate of gang desistance was actually greater among members of the comparison group.
An examination of the PIT youth in isolation indicates that those who received extensive PIT program services were less likely to desist from gangs than those who received relatively fewer services. During their post-test interviews, the majority of PIT participants claimed that the PIT program had helped them stay away from gangs and avoid involvement with gang-related criminal activity. One-third (33%) of the participants, however, felt that the program had not helped them avoid gangs.
The majority of staff members also felt that the PIT program had helped youth avoid gangs. However, most staff members agreed that the program was more suited to prevention than intervention. According to the case managers, the program had only helped a few “hard-core” gang members exit the gang lifestyle.
Lessons Learned and Recommendations
Program Delivery
PIT case managers should screen all new recruits using the standardized screening interview. The screening interview should always be administered by a case manager in a private, one-on-one setting with the expectation that all sensitive questions be asked during the interview.
A standardized risk assessment should be mandatory for all program participants. The results of the risk assessment should be the basis for individualized program plans. Case managers should also be allowed to further assess their participants' needs through informal, rapport-building activities.
PIT supervisors need to closely monitor the activities of case managers to ensure that individualized case management plans are being followed. PIT should hire case managers who already have extensive case management experience and who are already familiar with the need to fully document and monitor individualized case management plans.
The PIT program should further develop its group training curriculum and include detailed lesson plans that address specific topics. These topics should be directly related to the predominant risk factors of program participants. Case managers should receive additional training pertaining to the purpose of the curriculum and how to deliver lessons to an audience of high-risk youth.
To improve program fidelity, PIT supervisors need to increase on-site visits during group training to ensure that the program curriculum is being delivered in a consistent fashion across different program locations.
A reduction in total caseload numbers (10 per case manager) and a slight increase in case manager hours (to 40 hours) could vastly improve the ability of the PIT program to deliver necessary services to individual participants.
The PIT program should change its hours of operation to increase the participation of high-risk youth residing in disadvantaged neighbourhoods. It is recommended that the hours of program operation be changed to weekdays between 1:00 pm and 9:00 pm. It is also recommended that the program operate during at least one afternoon during the weekend. This will better accommodate PIT participants who are either in school or employed.
A strict code of conduct and security protocol should be established and participants should be made aware of program rules during intake. Lack of respect for program rules should result in dismissal from the program.
The PIT program should develop two program streams – one for youth 13 to 16 years of age and the other for young adults between 17 and 24 years of age. This will ensure the curriculum is age-appropriate.
The PIT program should ensure that participant support does not end completely upon program completion. In an effort to continue to support youth once the program has ceased, program workers should refer youth to other support workers or community programs as needed.
Evaluation
It is essential for the evaluation research team to be fully accessible to all program staff and to address any questions or concerns regarding the evaluation process as quickly and thoroughly as possible. To accomplish this, it is necessary that they be on site when possible.
It was useful that members of the evaluation research team took the time to fully explain each element of the evaluation strategy to program staff. It is recommended that evaluators explain how the program evaluation can enhance program efficacy during the planning and implementation of the program.
Evaluators should consider the opinions and suggestions of program staff and stakeholders in the development of evaluation strategies. Reviewing proposed evaluation strategies with program staff before implementation will bolster researcher-staff relationships and may lead to valuable methodological revisions.
PIT program supervisors need to conduct periodic on-site visits to program sites and conduct regular discussions with researchers to ensure that proper evaluation-related procedures are being followed. Emerging problems should be dealt with immediately before they compromise the quality of the evaluation data.
Honorariums are required to recruit both program and comparison group respondents. Providing monetary incentives to youth who take part in evaluation-related interviews will encourage youth to complete the interviews, and reward them for their time and honesty.
To promote evaluation team safety, evaluation research assistants should travel and work in teams of two whenever they are in high-risk neighbourhoods. In addition, interviews should be scheduled only in public locations, such as community centres and schools.
Ensure that the comparability of experimental and comparison groups is tested in the early stages of the evaluation. Identify and report on the similarities and differences by considering the following factors: age, socioeconomic status, history of charges, history of arrests, history of offending, key gang-related risk factors, school status and other relevant factors. This will clarify the comparability between the two groups. Differences related to the predisposition of the group can be statistically isolated and considered when interpreting results.
All efforts should be made to reduce respondent attrition. In order to facilitate the re-connection with respondents for follow-up interviews, researchers should try to utilize a variety of contact strategies, including phone numbers, friends or family members, case managers, community workers, twitter accounts, texting and Facebook. During the pre-test interview, the interviewer should also ask the respondent about the best way to contact them in the future.
PIT program staff members need to better track the activities they engage in and the specific services they provide to youth participants. To improve case manager recall and the accuracy of tracking data, staff members should record their activities on a standard tracking sheet at the end of each business day. The retroactive recording of activities at the end of the week or the end of the month could contribute to both the under- and over-estimation of program services. The accurate recording of program activities and the specific services delivered to individual participants will better enable researchers to determine the aspects of the PIT program that are most effective and those aspects that are not working.
Some respondents may be sensitive to answering questions about their family members and friends. Whenever possible, such questions should be administered at the end of participant interviews after a rapport has been developed.
The evaluation of PIT and any other crime prevention program would greatly benefit from the acquisition of official criminal record data from the police as well as the participants' official school records. This would allow researchers to compare the results of the participant and staff member interviews with official statistics and therefore improve the validity of the evaluation results. However, the collection of this information needs to be conducted in a manner that does not disclose the names of PIT participants to police or school officials. In order to accomplish this, a strong partnership needs to be forged between program officials, the evaluator and both school and police officials. The police would need to sign a confidentiality agreement to ensure that the program youth would not be unnecessarily monitored or labelled as a result of the need to obtain data from official records.
Conclusion
The process evaluation revealed that the program did reach the intended target group, however, the PIT program was not delivered in a manner that was consistent with the original City of Toronto program plans. Overall, program fidelity was low, indicating that the average PIT participant received far fewer case management services and group training hours than was recommended.
The evaluation results suggest that over the short term, PIT participants were more likely than members of the comparison group to experience improved attitudes towards the criminal justice system in general. In the short term, PIT participants were more likely than members of the comparison group to experience reductions in victimization, marijuana use, criminal offending and association with anti-social peers. Comparable improvements were made in participation levels in pro-social activities and self-esteem.
The program was challenged at sustaining favourable changes in the long term and was not able to demonstrate its ability to impact measures related to association with gang-involved peers, alcohol use, academic performance, employment, risk-taking, arrests and gang involvement.
The results also indicate that due to limitations in the data related to dosage, the evaluation team could not make any conclusions about how the participation levels in the project contributed to the changes in the outcomes measured.
The majority of PIT participants reported that they benefited from their participation in the program and had developed a strong supportive relationship with their case manager. PIT staff members also felt that the program had helped many of the youth involved in the program.
The evaluation results suggest that the PIT and similar crime prevention programs need to standardize program procedures, ensure on-site supervision, clearly document program activities and follow the program plans and required dosage. Improvements in these areas might contribute to strengthening the program's ability to contribute to changes in gang involvement and other important gang-related risk factors.
For more information or to receive a copy of the final evaluation report, please contact the National Crime Prevention Centre by e-mail at ps.prevention-prevention.sp@canada.ca.
If you wish to register for the NCPC mailing list to receive information from the Centre, please visit the subscription page at: http://www.publicsafety.gc.ca/cnt/bt/mlng-lst-eng.aspx
- Date modified: