Final Program Evaluation – Towards No Drug Abuse

Introduction

Towards No Drug Abuse (TND) is an evidence-based program designed to help youth reduce their tobacco, alcohol or drug use, and violent behaviour^{Footnote 1}. TND has been tested in seven experimental field trials^{Footnote 2} in the United States. The most consistent finding (in 7 out of 7 trials) was a favourable reduction in the consumption of hard drugs. Favourable effects on alcohol use were found in 4 trials, on marijuana use in 3 trials and on cigarette smoking in 2 trials. Favourable reductions in violence victimization and weapons carrying were found in 3 trials.

The program was implemented in Canada to determine if this intervention can contribute to changes in drug abuse patterns in Canada.

The John Howard Society of Hamilton implemented TND between July 2009 and September 2014 in schools within the Hamilton-Wentworth District School Board and the Hamilton-Wentworth District Catholic School Board, in Hamilton, Ontario. The National Crime Prevention Strategy (NCPS) provided $1,556,000 in funding for the program.

Program Description

TND was originally developed as a classroom-based drug abuse prevention program to be used among alternative high school students, aged 14–19 (Sussman, 1996). However, the program has also been found to be effective with a general high school sample (Dent, Sussman & Stacey, 2001).

TND focuses on three factors that predict tobacco, alcohol, and other drug use, violence-related behaviours, and other problem behaviours among youth, including:

• Motivation factors (i.e., students' attitudes, beliefs, expectations, and desires regarding drug use);
• Skills (effective communication, social self-control, and coping skills); and
• Decision-making (i.e., how to make decisions that lead to health-promoting behaviours).

The model developer, Sussman, indicates that at the completion of the TND program it is expected that students will be able to:

• Stop or reduce the use of cigarettes, alcohol, marijuana, and hard drugs (i.e., cocaine, hallucinogens, depressants, and amphetamines,).
• Stop or reduce weapon carrying and victimization.
• State accurate information about the consequences of drug use and abuse, including environmental, social, physiological, and emotional consequences.
• Demonstrate behavioural and cognitive coping skills.
• Make a personal commitment regarding drug use^{Footnote 3}.

The American TND design consists of 12 in-class interactive sessions. Each session should last from 40 to 60 minutes. Sessions address active listening, stereotyping, myths and denials, chemical dependency, the impacts of drug abuse, marijuana, tobacco use cessation, stress, health and goals, self-control, positive and negative thought and behaviour loops, perspectives and decision-making, and commitment.

To accommodate the program in the Ontario school system, the same content and duration was implemented; however, the number of weekly sessions was reduced to eight. To compensate for this change, session length was extended to 90 minutes and the frequency of sessions per week was increased. If this transformation of the project was not considered, physical education departments in Ontario who only have 5 of the 16 days devoted to substance abuse would not be able to accommodate the program in Ontario schools.

Target Group

The project's aim was to target youth in schools and in the community who are at a high risk of committing crimes related to drug and substance abuse. The target group for the school-based program consisted of the general population of high school students in the Hamilton-Wentworth area, with a focus on grade 9 students. In defining the school-based target group, an assumption was made that all students in the Hamilton-Wentworth area have some degree of risk for drug-use. The target group for the community-based program consisted of higher-risk youth involved with community organizations including youth that have offended, and were being referred for extra-judicial measures. These participants were higher risk as a result of being part of regular or alternative education, expulsion and suspension programs.

The experimental group sample (TND participants) included both the school-based component (n=1,836) and the community-based component (n=81). The samples from the school-based and community-based components were pooled in order to enhance the sample size so that multivariate analyses could be undertaken. The TND group had a slightly higher proportion of females and the comparison group had a slightly higher proportion of males. The average age for the TND participants at program start was 14.24 years (SD=1.01), and the average age for the comparison group participants at the pre-test was 14.29 years (SD=.57). With respect to ethnic background, for both groups, the majority of participants is Caucasian and speak and write English only.

Based on socio-demographics measured, there were no statistically significant differences between the TND treatment group and comparison group. Based on a risk level categorization developed by the evaluation team, the majority of the TND participants were considered ‘low risk'(68%), while 21% were ‘medium risk', and a smaller proportion were categorized as ‘high risk' (11%).

The comparison group had a similar risk profile with the majority ‘low risk' (69%), 17% were ‘medium risk', and 14% were categorized as ‘high risk' (see Table 1). Chi-square test results confirmed that the experimental and comparison group were similar indicating that any differences in results between the TND intervention and the group without the program could likely be attributed to the intervention and not to the predisposition of the groups.

Evaluation Objectives

The NCPS contracted Goss Gilroy to conduct the impact evaluation of TND. The impact evaluation, valued at $197,412 started in 2010 and ended in 2014.

The objectives of this evaluation were as follows:

• Incorporate information that will assess the extent to which the project is being implemented as intended.
• Assess whether the intended outcomes were achieved, and whether there were any unintended outcomes.
• Provide a descriptive cost analysis for the project and conduct a cost effectiveness analysis.
• Identify lessons learned, exploring what has worked well in the project and what has not worked as well, and make recommendations to strengthen the project.

Evaluation Methodology

The school based component used a quasi-experimental, repeated measures between-group design. The comparison group used classes from schools that were not participating in TND during the semester. The within-group repeated measures involved the administration of self-reported questionnaires at four stages: pre-intervention (T1); immediately post-intervention (T2); 6 months post-intervention (T3); and 12 months post-intervention (T4). Due to the repeated efforts of the Senior Evaluation Advisor in the NCPS and evaluation team, innovative strategies were developed to ensure that the comparison group would remain long enough for  between measures to be taken at the 6 and 12 months follow-up stages^{Footnote 4}. The interactions between time and group (Time x Group) were the key results for determining whether the program made a difference in the outcome measures. These will be illustrated in tables in this report.

The evaluation design for the community-component was a within-group repeated measures design similar to the school-based component, but with no comparison group. This type of design tests changes in the target group before the program and at various follow-up points after the completion of the intervention.

The quantitative analyses included using descriptive frequencies, cross-tabulations (chi-square), used multivariate analyses (ANOVAs)^{Footnote 5}, to identify significant differences between the intervention and the comparison group,  subgroups (e.g., gender, risk level) and time (e.g., pre, post, follow-up measures) on key outcome variables.

Qualitative analyses of focus group and key informant interview data were conducted using a content analysis approach to identify common trends in responses.

When reviewing the tables in this summary, review the interaction effects (farthest column to the right) to determine if there were statistically significant differences in changes within time periods between the TND participants and the comparison group. The columns to the left labeled TND Participants identifies changes within the TND participant group as a whole^{Footnote 6} and also delineates change by risk and gender levels (subgroups).

Research Limitations

The main limitations identified for this evaluation were the smaller number of TND participants than intended, the limited proportion of TND participants assenting/consenting to the evaluation component, and, consequently, the limitation in follow-up data because participants had not obtained parental consent for the evaluation follow-up surveys. For example, while approximately 1,800 youth participated in the pre-post sessions conducted in class, less than half (n=765) provided parental consent for the evaluation team to follow-up with participating youth at the 6- and 12-month periods. This likely has resulted in some biases in results when combined with response rates to follow-up surveys in the range of 35%.

In addition, the recruitment of participants for the comparison group was initially quite slow, with few classes participating; however improvements were made in the last two years of the evaluation study and the total numbers of comparison group participants increased. Having much fewer comparison cases in relation to TND participants had an impact on the reliability of the analyses in terms of statistical power.

A further limitation related to the assessment of participants' risk level and subsequent categorization. Due to issues regarding the amount of time required to administer the risk assessment tool that was initially selected for use by TND, an alternative risk assessment approach was developed and implemented. Once these data were analyzed, however, there were limited participants deemed high risk and little variance on the items in order to conduct a factor analysis for construct validity purposes.

There were challenges in collecting individual level information on attendance, suspensions, expulsions and records pertaining to involvement with the law (e.g., police records, court records), therefore, self-reports were the only source of data. Not having data from these external sources limited the extent to which some of the longer-term outcomes such as decreased anti-social behaviour leading to suspension and expulsions, and reduction of criminal behaviour could be assessed.

Content fidelity was high throughout the program in that all content was delivered in all courses with the same required content and duration; however, fidelity with respect to the number of classes and frequency of delivery did not adhere to the model developer's requirements^{Footnote 7}. Given the Canadian educational context with longer instructional periods (90 minutes vs. 45–50 minutes) and block teaching (arranging for one subject to be taught intensely within a short period of time — five classes per week vs. one per week), the program was adapted in Year 2 and Year 3 to meet these structural differences. The evaluation was able to test whether the change in fidelity requirements had an effect on the outcomes being measured. The test revealed that there were no differences in outcomes between the group that used the developer's fidelity requirements and the group that adapted the fidelity to fit within the Canadian school structure^{Footnote 8}.

Data Collection Method

The data sources used in this evaluation were the model developer's self-administered and online questionnaires. Complementary methods included focus groups with school-based program participants, and key informant interviews with TND program stakeholders (i.e., teachers, principals, a representative from each school board, and community partners).

Outcomes Measured

The evaluators assessed the program's ability to increase participants' knowledge of the consequences of tobacco and substance abuse, and to increase participants' social skills, decision-making, self-control and coping ability. They also evaluated the program's ability to reduce the risk factors associated with drug-related crimes including cigarette use, hard drug/substance use, weapons carrying, and victimization.

Outcome Evaluation Findings

The overall summary of findings for anticipated outcomes is that approximately six months after participating in the program, TND participants have similar results to the comparison group on many of the outcomes. This indicates that participation in TND is having a limited impact on the anticipated skills and behaviour outcomes at six months post-program. One exception is that on the knowledge outcome, TND participants continued to have higher test scores than the comparison group indicating that TND participation had a positive impact on knowledge. Unfortunately, this increased level of knowledge does not appear to have been translated into similar positive changes in skills or behaviour.

Knowledge

Results indicate that TND participants' knowledge of the consequences of alcohol, tobacco and drug use was significantly increased as a result of participation in the program. The pattern of increased knowledge for TND participants was significantly greater compared to the comparison group of non-participants^{Footnote 12}. Furthermore, this increase in knowledge was sustained over 6^{Footnote 13}- and 12^{Footnote 14}-month follow-up periods after the intervention was completed (see Table 3). These findings were corroborated by focus group discussions with TND participants, and key informant interviews with program stakeholders indicating that participants' knowledge about drugs and drug-related statistics had increased as a result of their participation in the program. While the pattern of increased knowledge was found across all three risk profiles of TND participants, the actual amount of knowledge as reflected in test scores was lower among the high risk participants and males.

Attitudes and Skills

In the pre-post period, TND participants reported short-term increased likelihood of using what they learned in school to assist them in the area of social situations, decision making, self-control, and coping. These short-term increases were significantly greater compared to the comparison group of non-participants (see Table 4). TND participants' self-reported increase in skills was not maintained at the 6-month and 12-month follow-up levels. All four measures related to skill indicate a decreased likelihood that the student would be able to use what they learned in school to: 1. stay away or not increase their use of alcohol, tobacco or other drugs; 2. quit or reduce use; 3. know about other things they can do with their friends besides using drugs; or 4. make their household become or stay drug-free. These decreases were not significantly different from the changes in the skills reported by the comparison group.

Across the four skill measures, the participants with high-risk profiles reported a lower likelihood of being able to use what they learned in school to assist them in the various areas of decision making, social situations, self-control and coping. For three of the skill areas, female participants reported more confidence than males in being able to apply what they learned to various situations including: 1. quitting or reducing use; 2. knowing other things to do with friends other than using; and  3.making their household become or stay drug free.

Behaviours

Cigarette Use

TND participants reported increased 30-day cigarette use from the pre-to-post-test period (see Table 5). This increase was not statistically significantly different from the changes in the cigarette use reported by the comparison group. There were no statistically significant differences between the TND participants and comparison group observed at the 6 month or 12 month follow-up periods. The participants with a medium or high risk profile were more likely to use cigarettes, and showed greater increases in cigarette use over the pre-post period^{Footnote 20}. Males were also more likely to use cigarettes and showed greater increases over this period^{Footnote 21}.

It should be noted that both the TND participants and comparison group participants moved to a new environment between grades 8 and 9. It is possible that the change from junior high to high school contributed to an increased use due to increased exposure and access to cigarettes.

Alcohol, Marijuana and Other Drug Use

There were some unintended negative outcomes in the area of alcohol, marijuana, and stimulant use. In the pre-post period, TND participants reported increased 30-day alcohol and marijuana use. This short-term increase in alcohol and marijuana use was not statistically significantly different from the 30-day use of alcohol and marijuana by the non-participants in the comparison group. There was, however, higher marijuana use among TND participants' when compared to the decrease in use by the comparison group at the 12 month measurement.^{Footnote 22} However, the 12 month data is based on a relatively small comparison group sample.

There was also statistically significant higher stimulant use by the TND participants when compared to the comparison group at the 6 month and 12 month measurements. Males were more likely to use marijuana and show greater increases over this period. Participants with a high risk profile were more likely to use all types of substances, and showed greater increases on all substances over the pre-post period.

In focus groups, several TND participants stated that their behaviours did not change following TND because they did not take or consider taking drugs before their participation in the program. Most participants said that, if offered drugs such as marijuana, they would have said no even without the TND sessions. 

Reduced Weapon Carrying and Victimization

Among both the TND participants and the comparison group, self-reported incidence of weapon carrying was overall very low. For the initial pre-post-test analysis, the TND participants did not show the same increase in knife carrying as the comparison group.^{Footnote 23} For the 6 month follow up data, there were no significant differences between the experimental and comparison groups. At the 12 month measurement, the comparison group had a larger decrease in knife carrying.^{Footnote 24} As noted previously, the 12 month data is based on a relatively small comparison group sample.

Males and TND participants with high risk profiles were more likely to report carrying either a knife or a gun. As well, high risk participants were more likely to report a decrease in knife carrying during this period.

Among TND participants, there was a significant reduction in victimization with respect to “being injured on purpose without using a weapon” for all three post-intervention periods. TND participants also showed a reduction in the 6-month and 12-month data for “having had property deliberately stolen or damaged.” None of these reductions were significantly different from the changes observed for the comparison group. At the 12 month follow up, the TND participants had no change in reporting being “threatened with a weapon but not actually injured” while the comparison group had a decrease in being “threatened with a weapon but not actually injured”^{Footnote 25}.

Medium risk participants reported the greatest decrease from pre to post program on “being injured on purpose without using a weapon”. High risk participants reported “being injured with a weapon” significantly more often after the program. Higher risk participants were more likely to report all four aspects of victimization including injury, threats and stolen property. Male participants were more likely than female participants to have experienced “being injured with a weapon” as well as “threatened with a weapon but not injured”.

Cost Effectiveness Analysis (CEA)

During the interim evaluation of the TND program, there were some statistically significant differences between the TND participants and comparison group that allowed for the calculation of the CEA.

The average cost per participant for the TND program was $544, taking into account NCPS contributions and JHS in-kind contributions. Some in-kind contributions were not included in these costs such as the in-kind contributions from schools to host the TND program such as classroom use and school staff time to make arrangements for TND facilitators to work with their classes. It is important to note that the cost per participant includes the school and community participants as the financial data were not separated for each group, therefore it was not possible to conduct the cost analysis for each program component separately.

The main significant incremental outcome is increased knowledge regarding the consequences of tobacco use and substance abuse. The cost-effectiveness analyses determined that a 25% incremental gain in knowledge costs $544 per student and $135 per knowledge point gain. Theoretically, if one considers the findings from published studies of TND and the program logic, one could speculate that this incremental knowledge should contribute with time to decreases in alcohol, tobacco, marijuana, and hard drug use in the range of 7 to 50%, depending on substance and study, when compared to non-participants. Given the findings in the final analysis, it is questionable how much this increase in knowledge has had an impact on substance abuse. Based on these estimates and the risk profile of youth participating in the TND Project, the lifetime benefits of the reduction in alcohol and drug use would have to exceed $54K for the conservative impact estimate and approximately $11K for the upper impact estimate^{Footnote 26}.

The present study was unable to identify any sustained incremental changes in the outcome measures beyond knowledge scores. There were no demonstrated long term (at 6 months and beyond) impacts of TND on increased skills, reduction in substance use, weapon carrying or victimization. Consequently, the overall conclusion from the cost analysis is that the incremental costs in gains in knowledge do not appear to translate into incremental gains in other outcomes for TND participants.

Challenges and Lessons Learned

While overall there were high levels of satisfaction with the TND program among student participants, some students did make some suggestions for improvements in the curriculum such as: 1. providing more information about certain topics such as alcohol use, weapon carrying, peer pressure, and how drugs cause stress; 2. emphasizing the interactive nature of the instruction, and 3. focusing on the types of drugs and substances that they felt were more relevant such as alcohol and marijuana. The adjustment in focus towards alcohol and marijuana was supported by the stakeholders consulted during the key informant interviews. Stakeholders also indicated that the reduction from 12 to 8 sessions to adapt to the longer periods in the Ontario school system was appropriate, especially since the quality and quantity of the course was maintained.

The main challenges encountered with the evaluation revolved around the recruitment of a suitable comparison group and the rates of agreement by TND participants to be involved with the evaluation. As a result, the main lessons learned with respect to developing and implementing an evaluation of this type are: 1. to highlight the importance of having suitable comparison groups in the implementation of a rigorous evaluation; 2. to consider having participation in the intervention contingent upon participation in the evaluation; 3.to conduct a pre-implementation study to assess the demand for a specific intervention; and 4. to obtain support within the contribution agreement^{Footnote 27} to recruit an adequate number of TND participants and comparison group participants for the evaluation study.

Given the lack of demonstrated incremental impacts on identified outcomes for TND within the context provided, expanded or additional implementation of the TND Program within the Canadian context should be undertaken with caution. The results of the impact evaluation were unable to demonstrate that participation in TND leads to significant incremental impacts on identified key anticipated outcomes. Although there were high levels of satisfaction with the program among students, facilitators, and stakeholders, the program would likely need to be further adapted and/or changed to obtain the intended impact among Canadian youth.

Since addressing substance abuse is a key factor in reducing crime, some adaptations to the program may include focusing the intervention on the high risk youth group not currently in school or participating in alternative schools^{Footnote 28}. Focusing on secondary prevention ensures that resources are being used by those that need to make changes in their substance abuse patterns. In the present study, given the low risk nature of the school community, even if the intervention were effective, substantial change would not be visible within this low risk population. Modification of the curriculum to address the specific needs of the high risk user combined with revisions in the model developer's survey instruments would enhance the evidence-based nature of substance abuse interventions in Canada.

References

Dent, C. W., Sussman, S., & Stacey, A. W. (2001). Project towards no drug abuse: Generalizability to a general high school sample. Preventive Medicine, 32, 514-520.

Sussman, S. (1996). Development of a school-based drug abuse prevention curriculum for high-risk youths. Journal of Psychoactive Drugs, 28, 169-182.

GGI.^{Footnote 29} (2014). Towards No Drug Abuse (TND) Impact Evaluation: Final Report.

Smith-Moncrieffe (2010). Using a Fidelity Index to Increase Program Attribution, Canadian Evaluation Society

Smith-Moncrieffe, D.  (2014). Interim Results from the Towards No Drug Abuse (TND) Project. Retrieved from: http://www.publicsafety.gc.ca/cnt/rsrcs/pblctns/rslts-twrds-ndrg-bs/index-eng.aspx