Research report: Evaluation Summary of the Life Skills Training Program

Research report: Evaluation Summary of the Life Skills Training Program PDF Version (232 Kb)

by Giselle Rosario

2015–R013

Abstract

This research report provides evaluation results for the Life Skills Training (LST) program implemented in Edmonton, Alberta. LST is a prevention program targeting adolescent drug and alcohol use. Implementing and evaluating model programs provides an opportunity to assess what factors contribute to changes amongst different target populations in a variety of contexts. This evaluation research study was conducted between 2010 and 2014 and was adapted to an Aboriginal population. Valid conclusions regarding the effect of the LST program on substance use onset was unable to be drawn from the data available, as the project experienced challenges with enrolment, participant retention and drop-outs. Qualitative findings on most variables, however, were positive.

Author’s Note

The views expressed are those of the authors and do not necessarily reflect those of Public Safety Canada.  Correspondence concerning this report should be addressed to: Research Division, Public Safety Canada, 340 Laurier Avenue West, Ottawa, Ontario, K1A 0P8; email: PS.CPBResearch-RechercheSPC.SP@ps-sp.gc.ca.

Acknowledgements

Public Safety Canada acknowledges the work of the evaluation advisory committee, the program manager and staff responsible for implementing the project, and the contracted evaluation team led by R.A. Malatest & Associates.

Introduction

Premature and addictive use of substances is a well-established criminogenic risk factor that leads to criminal involvement. Although evaluations of Life Skills Training (LST) in the United States have shown the program to be successful, no evaluation has been conducted to determine its efficacy in a Canadian context, particularly in Aboriginal communities. In this context, the National Crime Prevention Strategy (NCPS) provided evaluation funding^Footnote1 to one LST project serving primarily urban Aboriginal youth in Alberta to test this promising program in Canada.

The program was delivered in Edmonton by the Ben Calf Robe Society (an organization that provides social support programs) to both Aboriginal and non-Aboriginal youth through an after school program. The National Crime Prevention Strategy (NCPS) provided $628,584 in funding to the Ben Calf Robe Society to implement LST from 2010-2013.

Program Description

Botvin Life Skills Training (LST) is a school-based prevention program that targets the early drug and alcohol use of adolescents, especially those in junior high school (grades 6 and 7). The main goals of the LST program are to prevent substance use amongst adolescents and to promote healthy alternatives to risky behaviour through activities designed to:

• Teach youth the necessary skills to resist social (peer) pressures to smoke, drink alcohol, and use drugs;
• Help youth develop a positive self-image;
• Enable youth to effectively cope with anxiety;
• Increase youth’s knowledge of the immediate consequences of substance abuse; and
• Enhance cognitive and behavioural competency to reduce and prevent a variety of health-risk behaviours.

The LST program uses these activities to reach its objectives of decreasing drug abuse risks by reducing personal motivation to use drugs, and reducing the susceptibility of youth to social factors that promote drug use.

The middle school program is delivered over 15 sessions in the first year, 10 sessions in the second year, and 5 sessions in the third. Each session lasts approximately forty-five minutes. The youth modules have three major component areas, which address domains associated with decision-making skills to prevent the use of alcohol, tobacco and drugs. The first component is designed to teach students a set of general self-management skills, the second focuses on teaching general social skills, and the third includes information and skills that are specifically related to the problem of substance abuse.

The booster intervention (sessions in year 2 and 3) is designed to be delivered throughout the eighth and ninth grades, and enable youth to practice the social and personal skills components of the program that were delivered in the primary year.

Ben Calf Robe used LST program recruitment criteria as a general guideline for participant selection. The criteria included youth who display behavioural problems at an early age, display hostility, aggression and a deficit in social skills, use drugs/alcohol, and/or have been involved with the law or Children’s Services at a young age. The screening form used by the program as a general guide did not measure risk factors. Rather, it was up to the program facilitator to assess eligibility on a case-by-case basis.

The program at Ben Calf Robe was adapted^Footnote2 and delivered to students in grades 4 through 9, beginning in grade 4 or 7. Four cycles of participants were tracked, with evaluation outcomes focusing on the latter two cycles.

Evaluation Objectives

The NCPS contracted an independent firm, R.A. Malatest & Associates, Ltd., to conduct the impact evaluation of LST^Footnote3. The impact evaluation study took place from September 2010 until March 2014. The objectives of the evaluation were to:

• Assess the extent to which the project is being implemented as intended.
• Assess whether the intended outcomes were achieved, and whether there were any unintended outcomes.
• Provide a descriptive cost analysis for each project.
• Identify lessons learned, exploring what has worked well in each project, what has not worked as well and make recommendations to strengthen the project for the benefit of others interested in implementing or supporting a project of this nature in the future.
• Assess the extent to which each project has been adapted to meet the needs of the youth/community.

Evaluation of the LST Program

This Evaluation Summary presents final outcomes from the impact evaluation of the Botvin Life Skills Training (LST) Program.

The evaluators employed a mixed-method approach using quantitative and qualitative methods.  A matched comparison group with a pre-post-post design was used to measure change over time on key outcomes. Because the impact evaluation team was not in place when the LST program was initiated (in the 2009/10 school year), the comparison group for Ben Calf Robe was only identified about twenty months later. The evaluation sought to obtain comparison group cohorts with similar characteristics as the LST participant cohorts to ensure that any differences between the program participants and the comparison group could be attributed to the intervention and not to the predisposition of the participants. Prior to the start of each cycle of the program, LST youth and comparison youth were generally comparable in terms of demographics and pre test scores, with minor exceptions.

In order to identify appropriate communities, the evaluators undertook community demographic research and risk assessments using demographic data to establish intervention/comparison group equivalency. They subsequently identified Dickensfield and Britannia-Youngstown (Edmonton, Alberta) as suitable communities from which to draw a comparison group.

To establish the equivalency of communities selected, the evaluator collected demographic data from a number of sources, and identified eighteen variables (such as number of lone-parent families, number of child welfare cases, number of crime incidents, school transiency rates, etc.). The evaluation recruited twenty-six comparison participants from the Boys and Girls Club in the Dickensfield and Britannia-Youngstown neighbourhoods.

Several measurement tools were used as part of the LST impact evaluation. The youth surveys included the following standardized instruments: LST Questionnaire, Rosenberg Self-Esteem Scale, Decision Making Scale, and the Casey Life Skills American Indian Assessment Supplement and Substance Use. Evaluators also conducted key informant interviews, focus groups, case studies and photo-voice activities. School data was to be used to track academic achievement as well as type and frequency of delinquent behaviours^Footnote4. Program fidelity was assessed by using the LST Fidelity Checklist.

The program expected to engage seventy-five students in grades 4 to 6 and forty-five students in grades 7 to 9. In total, one-hundred twenty students were expected to participate in the LST Program. Due to a change in the scope of the evaluation and attrition of youth, the anticipated numbers of youth participating in the LST program had been significantly reduced at Ben Calf Robe. As a result, analytical methods used to detect program effects on outcomes had to be altered^Footnote5.

Process Evaluation Findings

Five factors that contribute to an understanding of how the LST program was implemented by Ben Calf Robe are identified, namely: target population, program management, dosage, participant satisfaction and culture. These factors illustrate the extent to which the program was being implemented as planned in accordance with LST guidelines. It is understood that greater fidelity contributes to the greater likelihood of achieving desired outcomes.

Reaching Target Population

The program was not able to meet its target enrolment rate of 40 participants per year, and in fact only 13 youth completed three years or more of programming^Footnote6. While enrolment and drop-out rates improved somewhat over time, only one hundred sixteen of the one hundred sixty youth expected participated in the intervention.

Despite there being an interest in serving most of the Aboriginal population in this community, only 37% of the youth were of Aboriginal descent.

Program drop-out was also an issue, but improved over time as shown by the fact that dropout rates were reduced to eighteen percent from thirty-six percent in cycle 3, and further reduced to 9 percent by cycle 4. Key informants attributed the dropout rate to several factors such as: a lack of suitable and available transportation in the area for youth who are not in close proximity to the program site; families moving away from the area; scheduling conflicts and a lack of parental support or encouragement to attend or continue attending the program.

Program Management

Findings suggest that the program was managed effectively overall. There was some staff turnover but training was described as adequate. Program resources were a challenge due to a lack of dedicated program space within the host school, and a high turnover in principals at the school, which resulted in various levels of buy-in and promotion of the program on the school site.

Dosage

Dosage refers to the level of exposure (frequency, intensity and duration) of youth participants to the LST curriculum. The LST Fidelity Checklist as well as attendance records and key informant interviews were used to assess dosage.

Despite minor modifications in the method of class presentation and activity completion and the addition of supplementary materials, virtually all the curriculum was covered in each LST group with little variation from intended curriculum across all cohorts and cycles. Overall, the amount of time the facilitator spent on each lesson activity was fairly consistent across each level of the LST program. High fidelity was achieved in terms of the LST curriculum being implemented as intended.

The average percent of sessions attended by cohort varied from a low of fifty-six percent to a high of eighty-five percent. To bolster attendance and participation, staff provided incentives such as snacks and prizes, and this appears to have worked as attendance increased in the latter cycles. Throughout program implementation, participant responsiveness, as rated by facilitators, was consistently high.

Participant Satisfaction

The experiences of program participants and other stakeholders were almost universally positive. At least 80% of participants each year indicated that they were satisfied or very satisfied with the program. Specifically, stakeholders who were interviewed expressed their satisfaction with the program and its ability to aid youth in thinking critically, understanding life choices and the consequences of those choices. Participants noted their positive relationships with facilitators and their increased ability to make better decisions.

Cultural Component

The LST program was adapted at Ben Calf Robe to incorporate cultural components. The ability of the program to incorporate Aboriginal traditions was identified by focus group participants as a prime motivation for entering and staying in the program. Program staff had modified the program to include Aboriginal cultural components, such as sharing Aboriginal teachings (e.g., traditional foods, medicine wheel, teepee teachings, etc…) drumming, guest teachings from local Elders, and Sweat Lodge visits. It was widely felt that these modifications helped Aboriginal youth engage with and contribute to the program and develop pride in their culture.

Impact Evaluation Findings

Decision-Making Skills

Program participants showed frequent use of decision-making skills despite a lack of statistical evidence pointing to improvement in decision-making scores. No statistically significant differences were found between the mean decision-making scores of program youth and comparison youth. One possible reason for this is that the median scores were fairly high to begin with, and therefore there was little possibility for improvement.

The qualitative data strongly suggested that youth were improving their decision-making abilities as Elders, program staff, community partners and parents suggested that youth tended to be more confident and independent in making decisions on their own after having been through the program.

Participants’ Self-image

Results indicate that program participants had moderate levels of self-esteem that improved over time. Median scores on the Rosenberg Self-Esteem scale increased over time (27 to 30.5 from cycle 3 pre-test to cycle 4 post-test) as did those of the comparison group. There were statistically significant improvements in youth’s self-esteem during cycle 4 that saw a moderate improvement in score, however it is unclear what role maturation may have played in this outcome.

Qualitative findings show that program staff reported noticeable improvements in most youth’s self-image, and this was echoed by others in focus groups and case-study interviews. There was discussion amongst parents about their children’s increased assertiveness, positivity and confidence.

Coping Skills

Relaxation skills in the LST program are presented as coping mechanisms. At pre-test, both program and comparison youth had comparable median relaxation skills summary scores. While both groups improved their scores over time, these were not statistically significant, due primarily to small sample sizes

Qualitative findings suggest that the LST program positively impacted participants’ coping and relaxation skills. It was noted that sixty-two percent of those selected for case studies had demonstrated noticeable improvements in their ability to manage stress and anxiety. Parents and youth themselves identified several relaxation techniques that were learned and practiced as a result of the program. That being said, the scores also show that there were improvements but low retention, which suggests that additional focus on coping skills might be helpful for the long-term.

Anti-drug Knowledge

Program participation has had observable effects on elementary youth’s anti-smoking knowledge, with scores increasing from cycle 3 pre to post test. Further, the program group had higher overall, anti-smoking and life-skills knowledge scores compared to the comparison group at the cycle 4 pre-test. The findings suggest that knowledge gained during cycle 3 was retained, and that the increases were due to more than maturation.

For Jr. High program participants, there were higher overall knowledge scores over the course of cycles 3 and 4. The program participants had higher overall and anti-drug knowledge scores in relation to the comparison group.
Qualitative data supports the quantitative findings, and further suggests that youth have not only understood the in-session material, but have internalized, processed and shared what they have learned as well. Program staff identified improved understanding of the effects of substance use amongst almost all participating youth.

Cultural Knowledge

Cultural knowledge was not shown to have increased among LST participants.  Survey findings did not demonstrate any increases in cultural knowledge or affiliation. Data from the Casey American Indian Youth Assessment scale suggests that youth demonstrated moderate knowledge of and affiliation with their culture, and that these levels were similar between program and comparison youth.

Qualitative data suggests that Aboriginal participants enjoyed and appreciated the cultural program adaptations and opportunities. Elders, program staff and youth themselves spoke positively about the cultural elements of the program (teachings from local Elders, Sweat Lodge ceremonies, etc).

The evaluators point out that the lack of improvement in scores should not be taken to signify an absence of program impact on cultural outcomes, but needs to be understood in light of the Aboriginal participation rate which, at 37% for the entire program duration, was lower than the Aboriginal enrolment level expected. The Aboriginal cultural elements did not resonate for non-Aboriginal youth in the same way as for the Aboriginal youth.

Self-management Skills

It is unclear as to what extent, if any, the program has had in helping youth improve their self-management skills. One of the expected sources of information, school disciplinary reports, was unavailable.

The self-control scores of both participants and comparison youth were high to begin with, and the differences between groups were not significant. This combined with the small sample sizes and the fact that the self-control subscale had the lowest reliability of responses, means that the evaluation cannot comment on this outcome.

Social Skills

As with self-management skills, it is unclear as to the extent that the program helped youth improve their social skills. Overall, survey responses from program and comparison youth suggest that youth demonstrated relatively high levels of assertiveness, but that these did not noticeably improve over time, and further, there was a fair amount of variability in the scores.

Qualitative findings however suggest positive changes and improvements in social skills among participating youth. The LST program utilized role-plays and interactive activities and it was felt by youth, parents, and staff that these adaptations helped youth to improve their social skills. The fact that LST was delivered after school and facilitators were able to engage individually with youth (in part because of low participant numbers) further underscores the likelihood that social skills could have been learned and refined through the duration of the program.

Resistance Skills for Substance Use

There is evidence to suggest that the program helped youth maintain, rather than learn, resistance skills in relation to substance use.  At cycle 3 and 4 pre and post tests, program and comparison youth scored highly on drug refusal skills. During cycle 3, both program and comparison youth achieved higher drug refusal scores at the post test than the pre test. High scores were maintained in the program cycle 4 pre-test. None of the between-group and within-group comparisons were statistically significant.

Qualitative findings suggest that the program had a positive effect in helping youth to resist pressure towards substance use, as shown by parent interviews and case-study data. This indicator may suffer a ‘ceiling’ effect caused by high initial drug refusal skill scores, and more sensitive measures of change might be needed to have confidence in this measure.

Anti-drug Attitudes and Norms

There is a lack of clear, statistical evidence to suggest the development of anti-smoking, anti-drinking and anti-drug attitudes among program youth. Many participants in the program group scored near the maximum on these indicators and therefore the ceiling effect comes into play. The lack of statistical significance on most observations suggests that any observed changes in scores could be a result of chance variation.

Qualitative evidence however does support the development of anti-drug attitudes and norms through the course of the program. Photovoice activities graphically illustrated the negative opinions that some program youth formed towards smoking.

Parent/Child Communication

Focus groups and parent interviews found that parent communication with their children generally improved over the course of the program. Concrete examples of better parent/child communication were provided to the evaluators, and these improvements related to decision-making, stress-management and self-control. Parents generally attributed these communication improvements to their child’s participation in the LST program.

Cultural Pride

Most program staff, parents, and Elders reported noticing an increase in pride in Aboriginal culture among program participants.  Parents felt that attending cultural events made their children more proud of their culture. Generally respondents attributed the increase in cultural pride to the traditional activities that were part of the program.

Overall, the cultural components of the program were reported to be a very positive experience, but the evaluators note that the Aboriginal representation in the program, at 37%, was lower than targeted, and also that cultural pride can manifest in ways that are not easily identifiable.

Substance Use

Valid conclusions regarding the effect of the LST program on the onset of substance use cannot be drawn from the data available. Case study findings suggest that the program has had a positive impact on the intentions of youth to not use substances. Because survey data involved such low sample sizes, reliable conclusions cannot be made. Overall, few youth from either the program or comparison groups reported smoking, consuming alcohol, or using illicit substances at any time period; nor did these youth suggest any intention to begin using substances.

Cost Analysis

The evaluators provided a descriptive cost analysis that showed the overall project cost of delivering the LST program was $732,268.57. Overall, 116 youth enrolled in the LST program implemented by the Ben Calf Robe Society. This number fell significantly short of predicted enrolment numbers. The overall average cost per youth participant was $6,312.66, with costs rising to $10,286.55 for Cycle 4 given declining enrolment in the program.

The cost of the program per participant, however, should be interpreted with caution, and only as a description of costs, not as a measure of cost-effectiveness. In particular, because the LST program delivered by Ben Calf Robe was an adaptation of the program for a First Nations-oriented environment, and it was not delivered in a school/classroom setting, the cost of the program per client should not be compared to assessments undertaken in the United States. The costs of implementing and delivering LST are considerably lower in school-based environments.

Lessons Learned

The findings from the evaluation suggest that the LST program has demonstrated some modest success in meeting its short and medium-term outcomes. Qualitative findings were overwhelmingly positive for the outcomes of interest, but the quantitative analyses were limited by low sample sizes limiting the ability to make conclusive statements about program efficacy. Key recommendations and lessons learned include:

Parent Engagement

Parents should be more actively engaged in LST programming. Findings from the evaluation suggest that the involvement of parents in programming helps play a key role in the success of the program, particularly in terms of enhancing parent-child communication, facilitating referrals to other social agencies/services that may be needed, and increasing parent buy-in and support for the program – a critical component of ensuring program attendance and reducing participant dropout.

Aboriginal Culture

Aboriginal cultural components should continue to be incorporated into the program where possible. The inclusion of Aboriginal cultural components (e.g., Sweats) is well-received by participants and parents, and has helped increase participants’ enthusiasm for learning about their culture and participants’ feelings of pride in their Aboriginal culture.

Transportation

The transportation needs of participants and parents should be closely considered when implementing the program in a non-school-based (i.e., after school) setting. Findings from the current evaluation indicate that transportation to and from LST sessions was an issue for many participants and, moreover, is a critical factor in program dropout.

Non-school Based Programs

Non-school-based programs operating in schools must work closely with schools/school boards during the early stages of the program. The LST program delivered by Ben Calf Robe encountered several challenges related to its position as a non-school-based program operating within a school space, particularly around the lack of permanent space (i.e., no classroom) and lack of in-kind supports and supplies. Strategies that future non-school-based programs may wish to consider include providing school staff with more information about the program curriculum before program implementation, sharing feedback with school staff on a consistent basis (e.g., monthly meetings, presentations at school staff meetings, etc.), and sharing information about participants to school staff when necessary.

Curriculum Suitability

The suitability of program materials and curriculum should be closely considered before being delivered to target populations. The original LST program included frequent writing activities which severely hindered participants’ engagement, as the target population expressed challenges with writing skills.  Similarly, original supporting material was not deemed age-appropriate. As a result, introduction of the LST program to other similar populations across Canada must consider the skills and preferences of the target population before the program is delivered.

Conclusion

The evaluation of the LST program at Ben Calf Robe was limited by low sample size numbers due to reduced target enrolment, combined with challenges of participant retention and drop-out. This was an issue for both the program and comparison sites. Pragmatic issues faced by the implementing organization directly affected the kind of evaluation possible. However, the consistency and calibre of the qualitative data is promising in this evaluation. It shows that there is potential for LST to produce positive changes in risk and protective factors amongst participants, including Aboriginal youth.

Date modified:

2022-08-09