Results from the Towards No Drug Abuse (TND) Project

Results from the Towards No Drug Abuse (TND) Project PDF Version (291 KB)

Table of contents

• Introduction
• Program Description
• Evaluation Objectives
• Evaluation Methodology
• Measurement Tools and Data Collection
• Outcomes to be Measured
• Outcome Evaluation Findings
• Reporting
• End Notes

Introduction

Towards No Drug Abuse (TND) is a Blueprints for Violence Prevention model program designed to help youth reduce their tobacco, alcohol or drug use, and violent behaviour.

The John Howard Society of Hamilton is implementing TND between July 2009 and September 2014 in schools within the Hamilton-Wentworth District School Board and the Hamilton-Wentworth District Catholic School Board, in Hamilton, Ontario. The National Crime Prevention Centre (NCPC) is providing $1,556,000.00 in funding for the program.

Project TND has been tested in three true experimental field trials, in which approximately 3,000 youth from 42 schools participated. At the one-year follow-up, relative to comparisons, participants who received the 12-session program experienced a favourable reduction in the consumption of cigarettes (27%), marijuana (22%), hard drugs (26%), and alcohol (9%), as well as in victimization (6%) (Sussman et al, 2004). The program is being implemented in Canada to determine if this intervention can contribute to changes in drug abuse patterns in Canada.

Program Description

The program targets Grade 9 and 10 students in school, as well as youth at risk of committing drug and substance abuse-related crimes in the community. The school-based program is providing the intervention to the general population of high school students in Grades 9 and 10. The assumption is that school youth in the Hamilton region are all at risk of substance abuse, due to the disproportionately high rates of drug abuse in that region.

TND consists of 12 interactive sessions. Through these sessions, TND seeks to help youth decrease their use of substances, the carrying of weapons, and anti-social behaviour, while improving their attendance and academic performance. Each session lasts from 40 to 60 minutes. Sessions address active listening, stereotyping, myths and denials, chemical dependency, the impacts of drug abuse, marijuana, tobacco use cessation, stress, health and goals, self-control, positive and negative thought and behaviour loops, perspectives and decision-making, and commitment.

To further accommodate the program in the Canadian school system, the number of sessions was reduced to eight, however, all the original content is still being delivered as planned.

Target Group

The project's aim is to target those youth in schools and in the community who are at a high risk of committing crimes related to drug and substance abuse. To date, the project has recruited 2,332 school-based youth and 203 community-based youth. Approximately 40% of these youth (1,024) had parental consent to participate in the evaluation. The following table provides a summary of the baseline rates (prior to treatment) of cocaine, marijuana, alcohol and cigarette use for youth participating in the evaluation.

N=1001 (Note: The evaluators are using standardized TND questions in accordance with the Model Program.)

Table 1 indicates that the youth in the experimental and comparison groups used relatively no cocaine within the last 30 days prior to the start of the program. Their use of marijuana prior to the program is relatively higher, although a majority of the youth prior to the start of the program had not used marijuana within the last 30 days. The baseline data does suggest, however, that approximately one-third of the youth had consumed alcohol within the last 30 days prior to the program. Approximately 6% of the youth consumed cigarettes within 30 days prior to starting the program. Any baseline differences between the experimental and comparison groups were controlled in the multivariate analyses.

Evaluation Objectives

The NCPC contracted a firm to conduct this single-site impact evaluation of TND.^{Footnote 1} The impact evaluation study, valued at $250,000, started in 2010 and will end in 2014.

The objectives of this evaluation are as follows:

• Incorporate information that will assess the extent to which the project is being implemented as intended.
• Assess whether the intended outcomes were achieved, and whether there were any unintended outcomes.
• Provide a descriptive cost analysis for the project and conduct a cost effectiveness analysis.
• Identify lessons learned, exploring what has worked well in the project and what has not worked as well, and make recommendations to strengthen the project.
• Assess the extent to which each project has been adapted to meet the needs of the youth/community.

Evaluation Methodology

The overall impact evaluation for the school-based component is a quasi-experimental, between-group design using mixed repeated measures. The within-group repeated measures involve the administration of self-report questionnaires at four stages: pre-intervention (T1); immediately post-intervention (T2); 6 months post-intervention (T3); and 12 months post-intervention (T4). The comparison group recruitment began in the fall of 2011 using classes from schools that were not participating in TND during the semester.

The evaluation design for the community-component is a within-group repeated measures design similar to the school-based component, but with no comparison group. The data analysis for the evaluation findings in the current report includes participants from September 2010 to January 2012 (i.e. the fall 2010 to fall 2011 semesters). The quantitative analyses focused on providing descriptive profiles of participants using frequency and cross-tabulations, and identified significant differences between subgroups (e.g., gender, risk level) and time (e.g., pre, post, follow-up measures) on key outcome variables using one- and two-way ANOVAs, and differences between the intervention and comparison groups. The interactions between time and group (Time X Group) are the key results for determining whether the program is making a true difference in the outcome measures (program attribution). Qualitative analyses of focus group and key informant interview data were conducted using a content analysis approach to identify common trends in responses.

The experimental group sample includes both the school-based component (n=950) and the community-based component (n=51). There were relatively equal proportions of young men and women. The average age for the TND participants at program start was 14.20 years (SD=1.16), and the average age for the comparison group participants at the pre-test was 14.24 years (SD=.69). With respect to ethnic background, the majority of participants in both groups are Caucasian and speak and write English only. Based on tests of differences between the TND treatment group and the comparison group, the main differences were that all comparison group participants were from the school-based program, and there was a lower proportion of Caucasians and higher proportions of First Nations/Métis/Inuit and 'other' ethnicities in the comparison group. The following table provides information about risk levels, including prior offending and history of drug use, for both the experimental and comparison groups prior to starting the program. This analysis was completed to ensure the groups were similar enough to accurately assess the program's level of contribution to change in the outcomes being tested.

Research Limitations

The main limitations identified for the evaluation to date are the limited proportion of TND participants assenting/consenting to the evaluation component and, consequently, the limitation in follow-up data up to 12 months.

The recruitment of participants for the comparison group has been relatively slow, resulting in much fewer comparison cases in relation to the TND participants, which has reduced the statistical power and decreased the reliability of the between-group analysis. This limitation has also decreased the reliability of the cost effectiveness analysis, as comparison group data is needed to accurately calculate the cost in order to create effects related to drug and substance abuse reductions.

Fidelity and attendance are positive from a program implementation standpoint; however, there are limitations in terms of being able to determine whether varying fidelity levels have an impact on the results. Having more cohorts (classes) will allow for enough variability to test the relationship between program implementation and impacts.

Measurement Tools and Data Collection

The evaluators will use the following survey instruments and data sources: TND standardized surveys, a risk assessment tool, TND fidelity tool, a participant satisfaction survey, informant interviews, focus group discussions, information from school records collected on a data collection template, and a cost-tracking template. The TND tests are standardized instruments that will be used for pre- and post-tests.

For both the school- and community-based settings, the timing of measures is as follows: pre-intervention during the initial session; post-intervention immediately after the 12 sessions have been completed; and post-intervention 6 and 12 months after the intervention has been completed.

Outcomes to be Measured

The evaluators will assess the program's ability to increase participants' knowledge of the consequences of tobacco and substance abuse, and to increase their social skills, decision-making, self-control and coping ability. They will evaluate the program's ability to reduce the risk factors associated with drug-related crimes including cigarette use, hard drug/substance use, weapons carrying, and victimization.

Outcome Evaluation Findings

The outcome evaluation findings report on the experimental group's trajectory from the pre-test up to the six month post-program. When there are statistically significant differences, results will be reported regarding gender and risk levels. References to the “between-group analysis” relate to the reporting of statistically significant differences between the TND program participants and the group of youth who did not receive the TND program.

All of the behaviour-related results measure drug or substance abuse within 30 days of the administration of the survey.

Knowledge

The preliminary results reveal that knowledge of the consequences of alcohol, tobacco and hard drug use is significantly increased as a result of participation in the TND program. This pattern of increased knowledge was not identified in the comparison group, indicating that the program contributed increased knowledge about the negative impacts of drug and alcohol use. This increase in knowledge was sustained over a six-month post-program follow-up period. These findings were corroborated by TND focus group participants, and key informant interviews with program stakeholders. While the pattern of increased knowledge was found across all three risk profiles of the experimental group, the actual amount of knowledge as reflected in test scores was lower among the high risk participants. For two of the skill areas, female participants reported more confidence than males in being able to apply what they learned to various situations (quitting or reducing use; knowing other activities to do with friends other than using drugs).

Skills

In the initial pre-post period, the experimental group reported short-term increased skills and attitudes. The experimental group demonstrated an increased likelihood of using skills they learned in school to stay away from using drugs and alcohol. These skills would assist them in the areas of social situations, decision-making, self-control and coping. The self-reported increase in skills was not maintained at the six-month follow-up for two of the four skill-related measures. Further analysis indicates that those with higher risk levels had higher knowledge scores than the moderate- and low-risk program participants. Higher-risk youth and females were more likely to learn all four skills related to reducing drug and substance abuse habits.

There was a statistically significant difference between the TND program participants and the comparison group for one of the four skills measured. The TND participants were more likely to report the ability to utilize the skill of “making your household become or stay drug free” when compared to youth who did not participate in the program.

Behaviours:

Cigarette Use

In the pre-post period, the experimental group reported increased cigarette use, similar to the pattern amongst the comparison group. The TND participants with a medium- or high-risk profile were more likely to use cigarettes, and showed greater increases in cigarette use over the pre-post period. Males were also more likely to use cigarettes and show greater increases over this period. There were no statistically significant differences in the rates of change in cigarette use from pre- to post-program between the experimental and comparison groups.

It should be noted that both the TND participants and comparison group participants are moving to a new environment between grades 8 and 9. It is possible that the change from junior high to high school has contributed to an increased use due to increased exposure and access to cigarettes.

Alcohol

In the pre-post period, TND participants reported increased alcohol use. The between-group analysis indicates that the comparison group experienced the same rate of increase. Higher-risk youth showed relatively reduced drinking levels when compared to moderate- and lower-risk youth.

Marijuana

In the pre-post period, TND participants reported increased marijuana use, similar to the pattern among the comparison group. Higher risk youths and males showed increased use during the pre- and six-month post-program period.

Cocaine Use

A between-group analysis indicates that there were no statistically significant differences in cocaine use before and after the program. Higher-risk youth showed greater decreases in cocaine use when compared to their moderate- and low-risk counterparts. As indicated in Table 1 in this summary, less than 2% of the youth were using cocaine prior to the start of the program.

Weapon Carrying

Prior to the program, for both the experimental and comparison groups, the overall self-reported incidence of weapon carrying was very low. Males and participants with high-risk profiles were more likely to report carrying either a knife or a gun in the 12 months prior to the program. There were no statistically significant differences between the experimental and comparison group rates of weapon carrying before and after the program. High-risk participants were more likely to report a decrease in knife carrying during this period.

Reduced Victimization

TND participants experienced a reduction in victimization during the pre-post period with respect to the variable being injured on purpose without using a weapon. Pre- and post-test results indicate that higher-risk participants were more likely to experience higher rates of victimization in all four areas, including being injured, being threatened and being a victim of theft.

The comparison group reported greater reductions in being threatened with a weapon when compared with the TND participants during a similar period of time. Additional data from a larger comparison group will assist in explaining this finding.

Cost Effectiveness Analysis

The average cost per participant for the TND program was approximately $540, taking into account NCPC contributions, and JHS in-kind contributions. At this preliminary stage, the main significant incremental immediate outcome is increased knowledge regarding the consequences of tobacco use and substance abuse. The preliminary cost-effectiveness analyses, based on comparison group cases, determined that a 25% incremental gain in knowledge costs approximately $540 per student, or $117 per knowledge point gain. Taking into account the findings from other published studies of TND and the program logic, it can be determined that this incremental knowledge should contribute over time to decreases in alcohol, tobacco, marijuana, and hard drug use in the range of 7% to 50%, when compared to non-participants. Based on these estimates and the risk profile of participating youth, the lifetime benefits of reduction in alcohol and drug use is between $11K–$54K in estimated program costs, if societal costs such as involvement with the justice system, social services, and health care system are considered.

Reporting

The results in this evaluation summary should be considered preliminary. The NCPC will be receiving another annual evaluation report in October 2013 and a final evaluation report in July 2014. The final report will include a 6- and 12-month follow-up period for the final cohort so that long-term impacts can be considered. Final conclusions about the program and its potential contribution to preventing the use of drug and substance abuse-related crimes will be reported in July 2014.

For more information or to receive a copy of the final evaluation report, please contact the National Crime Prevention Centre by e-mail at ps.prevention-prevention.sp@canada.ca.

If you wish to register for the NCPC mailing list to receive information from the Centre, please visit the subscription page at: http://www.publicsafety.gc.ca/cnt/bt/mlng-lst-eng.aspx

Date modified:

2022-08-02