Evaluation Summary of the Velocity Adventure Program

Evaluation Summary of the Velocity Adventure Program PDF Version (635 KB)

By Giselle Rosario

Abstract

Public Safety Canada's National Crime Prevention Strategy (NCPS) has a mandate to develop and disseminate knowledge of effective crime prevention programs to help decision makers and practitioners in communities across the country to make the best use of their crime prevention resources. To achieve this mandate, the NCPS supports selected, evidence-based crime prevention projects in communities across Canada and seeks to provide practical answers to three key questions: what works (with whom and in what circumstances), at what cost, and how does this intervention create positive change? Identifying what works is achieved through selectively evaluating crime prevention projects that demonstrate sound program logic. Unpacking the 'how' is done through collecting and analyzing qualitative information from all organizations that the NCPS supports to implement interventions.

This summary provides an overview of findings from the impact evaluation of the Velocity Adventure Program (Velocity) that was funded by the National Crime Prevention Strategy.^{Note 1}

Author's Note

The views expressed are those of the authors and do not necessarily reflect those of Public Safety Canada.  Correspondence concerning this report should be addressed to: Research Division, Public Safety Canada, 340 Laurier Avenue West, Ottawa, Ontario, K1A 0P8; email: PS.CPBResearch-RechercheSPC.SP@ps-sp.gc.ca.

Introduction

Velocity is an adventure-based program aimed to reduce anti-social behaviour, increase attachment to school, and reduce substance-use among at-risk youth. It was identified as a promising intervention that helps youth overcome adversity, create and enhance their connections in the community, and make healthy lifestyle choices.

The NCPS provided funding of $574,014 to the Community Youth Network (CYN) in St. John's, Newfoundland, to implement Velocity from March 2009 until August 2012. The project was extended for two years to February 28, 2014.

Program Description

Velocity is based on research that demonstrates the effectiveness of outdoor adventure-based programs in helping troubled youth channel their energy into more positive behaviours.

Velocity targets youth, aged 13 to 18 years, who are at risk of, or who have already been involved in criminal activity. The program addresses key risk factors associated with involvement in crime, including aggressive and anti-social behaviour, substance abuse, and poor attachment to school. The program has three chronological components:

• Group-building day trip adventures (e.g. kayaking, rock-climbing) to establish program expectations, build relationships with staff and promote group cohesiveness;
• 7-day Adventure Camp with activities in a remote setting (e.g. zip-lining, horseback riding) in combination with life skills and personal development activities; and
• Engage-Connect-Shift, which provides ongoing adventure day trips, individual support from project workers and workshops.

Velocity's programming was comprised of trust and communication activities, goal-setting, life skills, experiential learning, high adventure pursuits and health promotion. Youth were also provided with individual support, community referrals and on-going encouragement towards healthy lifestyles. The length of the program was one year including the selection process. Selection of youth participants into the program involved two referral forms and all data was documented in a data system.

Evaluation of the Velocity Program

Objectives

The impact evaluation, conducted by Ference Weicker & Company Limited, began in March 2010 and ended in November 2013. The purpose of the evaluation was to determine the effectiveness of Velocity in reducing risk factors associated with criminal behaviour among participating youth. The objectives of the evaluation study were to:

• Assess the extent to which the initiative is being implemented as intended;
• Assess whether the intended outcomes were achieved and whether there were any unintended outcomes;
• Provide a descriptive cost analysis for the project;
• Identify lessons learned, exploring what has worked well, what has not worked well and to make recommendations to strengthen the project for the benefit of others interested in implementing or supporting a project of this nature in the future; and
• Assess the extent to which the project has been adapted to meet the needs of the youth/community.

Evaluation Design and Methodology

The evaluators used a mixed method approach, integrating both qualitative and quantitative methods. Quantitative data was collected through interviews, pre-tests, post-tests and second post-tests of Velocity participants, and Royal Newfoundland Constabulary (RNC) aggregated data.

Additional data sources included but were not limited to: administrative data, community partner surveys, document reviews, program advisory committee interviews, financial reports, literature reviews, observations, parent/guardian surveys, referral sources surveys, staff interviews, staff reports, participant focus groups and youth client surveys. The primary outcome evaluation data that were obtained from the Velocity participants was an extended version of the pre- and post-test assessments.

The main risk assessment tools of the Velocity evaluation were three standardized tests relevant to the targeted risk factors of this program. The assessment tools were also related to the targeted immediate and intermediate outcomes of Velocity.

The Violent Intentions Test (Bosworth and Espelage, 2005) measured intention to use non-violent strategies to control anger and conflict. The Hostility Test (Derogatis, Rickels, and Rock, 1976) measured symptoms underlying hostility related to aggression, irritability, rage and resentment. Both tools demonstrated high reliability (internal consistency= 0.84 and 0.73 respectively^{Note 2}). Finally, the General Assessment survey focused on behaviours targeted by substance use and addiction, school motivation, sense of community belonging, problem solving and self-esteem.

The three tools were administered by CYN staff with participants at program start (pre-test), at program completion (post-test) and one year after program completion. A descriptive analysis of participant and comparison group members was completed. Paired and independent t-tests, and multivariate repeated measurements of outcome variables were used to identify within and between group differences.

Sample

CYN had planned to deliver Velocity to 90 youth over 36 months through 12 month interventions. Ultimately 87 youth were selected to participate (71 of 87 completed the program) in Velocity and all 87 youth had the expected characteristics and risk factors, such as being involved with alcohol, drugs, early criminal activity, aggressive and anti-social behaviour, and lack of school attachment. The evaluation sample of youth were males and females primarily between the ages of 12 and 17 and they completed 67 pre-tests, 44 post-tests, and 25 second post-tests.

Comparison Group

The evaluation used a quasi-experimental design that involved the selection of a group of youth selected from a local high school. A total of 45 youth completed the pre-tests in November 2011 and 16 of those were available to complete the post-test in June 2012.

The level of targeted risk factors among comparison group youth were minimal as these youth reported stronger attachment to school (96% of comparison group attended school regularly as opposed to 18% of Velocity participants), less use of drugs and alcohol (13% used illegal drugs as opposed to 44% of Velocity participants), no criminal records (as opposed to 27% of Velocity youth), and more community involvement (89% as opposed to 34% among Velocity youth). Further, the gender differences were notable with 87% female in the comparison group and 37% female in the Velocity group.

Since the comparison group was different from the participating group on key criminogenic factors, this limited capacity to attribute the findings to the intervention. 

Findings Related to Process

The Process evaluation focused on project implementation, challenges, successes, best practices, lessons learned and areas for improvement.

The findings indicate that, over the course of the project, the three main components of Velocity activities related to group building, the adventure camp and engagement were conducted as planned. Activities were organized and delivered for three distinct target groups: boys aged 13 to 15 (n=25), girls aged 13 to 17 (n=26), and boys between 16 to 18 years of age (n=20).

The only exception in carrying out Velocity as planned was the number of participants per year. Velocity was planned to be delivered to 90 youth over 36 months (i.e., 30 youth per year) but was offered to 87 identified at-risk youth whereas 71 of these youth completed the program. The acceptance rate from referrals to the program was about 50% across the 3 cohorts. The main reasons for youth not getting involved or continuing Velocity were reported as: loss of interest, lack of ability to make a long-term (i.e., one year) commitment, lack of parental approval, as well as losing contact with youth to follow up.

The process evaluation findings suggest that the method for identifying and recruiting at-risk youth was successful. All youth selected to participate in Velocity (n=87) had the expected characteristics and risk factors such as being alcohol abuse, drug abuse, early criminal activity, aggressive behaviour, anti-social behaviour, and lack of school attachment. In addition, the majority of participants (89%) were between the ages of 12 and 17. Staff were consistent across the three cohorts in terms of selecting youth with similar characteristics and level of risk factors.

Qualitative data indicates that staff received ongoing and diversified training, and program resources were sufficient.  The program was effective in developing relationships with program partners, primarily through the creation of an advisory committee and targeted outreach activities. During the interviews, participants indicated that they found the staff were qualified, professional and a valuable asset and a success factor for the program.

A key program challenge related to the characteristics of the target group, in that participants' behaviours and lifestyles resulted in challenges in delivering Velocity. Many Velocity participants were in transition, involved in criminal activities, and unable to make a long-term commitment. Other challenges reported include transportation and its time-consuming nature, as well as staff turnover.

Findings Related to Outcomes

The Velocity program uses recreation and adventure as a catalyst for interaction, and through the connections made within the program components, identifiable outcomes are expected.

Short-term variables measured included the extent of increased knowledge of problem-solving and decision-making techniques, a sense of belonging to the community and motivation to participate in school. Further outcomes of interest had to do with acquiring pro-social skills, knowledge of substance abuse and strategies on how to reduce it as well as motivation to reduce substance use. Intermediate and longer-term variables related to the extent of decrease in aggressive behaviour, the extent of increase in healthy life choices, ability to deal with life stresses, the extent of substance abuse reduction, and decrease in drug-related criminal behaviour.

The outcome data was obtained through youth pre-tests and post-tests (both participants and comparison group), parent/guardian surveys, referral surveys, and RNC aggregated criminal records. All sets of variables were assuming unequal variances, and an alpha level of.05 was used for all of the statistical tests, unless otherwise noted. The next section outlines key outcome findings.

Attitudes and Knowledge

The Velocity intervention created a positive attitude amongst youth towards education. According to background data, 63% of participants did not attend school regularly and 62% had been suspended from school when they began the Velocity Program.

Pre and post-test follow-up data at 12 months  (within groups analysis)  indicate that the participants' attitude toward education significantly improved after participating in Velocity as they had indicated having relatively more interest in academic goals after participating in the Program.^{Note 3} The majority of parents/guardians (74%) and referrals (58%) agreed that the youth had heightened academic goals after participating in Velocity (74%), and that they were interested in furthering their education (68%).

The average knowledge ratings provided by participants (between-groups analysis) were still significantly lower than those ratings obtained from the comparison group^{Note 4} indicating that the intervention  does not appear to  make an impact in the knowledge and attitudes related outcome.

In terms of participants' feelings of self-worth, findings suggest that youth were less likely to report that they feel they are a failure one year after participating in Velocity^{Note 5}.  However, participant's average ratings of feeling like a failure were still significantly lower than the average ratings of the control group^{Note 6}.  This result indicates that the intervention made an impact on reducing the Velocity group's feelings related to failure, but not to any remarkable degree when reviewed alongside the comparison group results.  It was suggested that the participants' life situation and requisite challenges could be a relevant factor.

In terms of participants' ability to handle unexpected difficult problems, statistical significance was not found as results were similar to the comparison group, suggesting that this was a common challenge for youth this age.

Youth were less likely to agree that they felt a sense of belonging to the community one year after participating in Velocity. Between-group analysis was not statistically significant but within-group analysis showed statistically significant change^{Note 7} in an undesirable direction, with Velocity participants feeling less of a sense of belonging one year after Velocity was over. Possibly participants' sense of community belonging was diminished as they had less contact with Velocity staff after the program ended. Most of the parents/guardians and referrals however perceived that the youth in fact had gained a sense of local belonging to the community after participating in Velocity.

Social Skills

In the survey, there were questions that attempted to determine if youth participants' skills related to handling social situations had changed as a result of the intervention. In terms of youth's perception of social skills, findings were mixed.

Between-group analysis was statistically significant for the question about interest in expanding skill-set^{Note 8}.  This means that the comparison group expressed a greater desire to expand their skill set over the time period, but the participant group's initial ratings were high, indicating that youth maintained their desire to expand their skills throughout and after the program. The pre- 2nd post test within-group analysis indicates that there was no change in youth perception about their social skills. This suggests that the intervention may not have been as effective as desired in terms of social skills expansion.

Immediately after completing Velocity, youth felt that they were part of a team, but this declined in the 12 month (2nd post-test) assessment. This may be explained by the fact that their support network/team disappeared after Velocity ended.

Risk and Protective Factors – Drug Use

The pre and post-test within-group data showed that youth reported that they were able to handle their substance use problems and were motivated to reduce their drug and alcohol use, however, the overall level of drug use did not change much among participants. Background data indicates that 44% of Velocity participants identified using drugs when they were selected for Velocity, and for 25% it was unclear. Pre-post testing indicated 32% of Velocity participants used drugs and at program end 40% indicated that they were using drugs.

Interviews with Velocity participants suggested that this fluctuation may be related to the time it takes to build a trusting relationship with staff in order to disclose substance-use status. This result then may not be an indicator of the intervention's inability to facilitate the reduction in drug use. This is a common challenge found when asking sensitive, personal issues early on (baseline) in a crime prevention program.

Of the twenty three youth that reported their level of drug use, six reduced the frequency of their use (two of the six eliminated their use), fourteen maintained the same frequency of use, and three had increased their frequency of use one year after Velocity as compared to before. One plausible suggestion given for the increasing use among some youth was that more experimentation occurred as youth got older. Data also suggests that Velocity can help more frequent drug users reduce their use as there was a decline in the number of daily and occasional users^{Note 9}.

Results of the within-groups analysis pre- and post-tests indicate a statistically significant improvement in youth perception of their ability to handle substance use problems^{Note 10}  and motivation to reduce use.^{Note 11}  The parents of the youth reported that the participants were more knowledgeable about substance use, however they were not as confident about the participants' motivation to reduce/avoid drugs. Between-groups analysis shows a statistically significant greater motivation to reduce substance-use among comparison group participants^{Note 12}.

Aggressive behaviour

Aggressive and anti-social behaviours did not appear to change as a result of participating in Velocity. Two standardized psychological tests were used to measure anti-social behaviours among youth before and after their participation in Velocity. The results of these tests, completed before and after participating in Velocity, do not indicate any significant changes. Comparing these results with the results of the comparison group also indicates that the participants' responses were quite similar to the comparison group. 100% of staff/management interviewed, and most program stakeholder indicated that Velocity had in fact been effective in reducing anti-social behaviour amongst youth. These individuals explained that Velocity exposed youth to multiple perspectives, and taught empathy and critical thinking.

The within-group analysis data regarding the ability to manage stress does not show significant positive change, and in fact suggests that on average, youth are more stressed one year following Velocity^{Note 13}.  Nevertheless, most of the parents/guardians (74%) and referrals (64%) reported that participants were better able to handle the day to day demands of life after participating in Velocity. No explanation for the differing results was provided except for the fact that as youth got older, they would have more responsibilities and stressors, and that the comparison group had similar ratings.

The background data on this outcome measure provides further context in that 59% of Velocity youth had experienced traumatic events in their life that affected their current behaviour and 56% had received counselling before participating in Velocity. This context suggests that given youth's history of stress, perhaps the intervention would need to focus more activities to address this to make an impact.

Police contacts

The Velocity evaluation team was able to access aggregated data from the Royal Newfoundland Constabulary (RNC) on Velocity participants^{Note 14} for the time prior, during and one year after Velocity program involvement. RNC aggregated data on participant police contact suggest that youth participants had fewer police contacts. The total number of participants' police contacts before Velocity was 292, and was reduced by 61% to 114 incidents during Velocity. One year after Velocity the number increased to 148 incidents, which still represents an overall reduction of 49% compared to before Velocity.

In particular, the number of police contacts significantly reduced for each cohort one year after the youth completed the program. The average number of police contacts reduced from 4 incidents per participant before Velocity to 2 incidents per participant after Velocity.

When findings were analyzed based on the severity of the contact, the data shows mixed findings in that charges against youth decreased from 27 before Velocity, to 13 during, and increased to 41, one year after Velocity. However, the incidents of youth involved in crimes as witnesses, victims or suspects dramatically decreased during and after Velocity. Police contacts where youth were victims or witnesses decreased from 120 instances before Velocity to 40 after. Similarly, youth suspected of crimes decreased from 119 before, to 46 during, and 34 after Velocity.

Findings were also mixed in terms of the types of incidents for which youth were charged. While charges for break and enter and theft increased after Velocity (0 to 10 and 5 to 8 respectively), charges for assault and mischief decreased (11 to 8 and 6 to 4 respectively). The suggestion was given that part of the increase in anti-social behaviour among some youth could be explained by typical maturation trajectories. This proposed explanation has roots in criminological literature, and specifically the 'age-crime' curve that establishes that criminal activity peaks at age 17, and then declines as one enters adulthood.

Cost

The total cost of conducting Velocity in two years was reported as $385,968. During this time period, a total of 51 youth completed the program, which resulted in a cost of $7,568 per participant. The most costly component of Velocity was the 7-day camp in 2010-11 and Engage-Connect Shift in 2011-12. Qualitative data confirmed that the resources provided were sufficient. Velocity staff/management perceived that the adventure camp was the most cost-efficient use of resources, while the selection and referral process was perceived to be the least cost-efficient activity.

The evaluators provided cost-descriptive data and made brief recommendations that may facilitate future cost-effectiveness analysis. It was recommended that both quantitative and qualitative aspects and impacts of Velocity be considered for cost-effectiveness analysis.

Recommendations

Evaluation participants were all of the view that there was a strong and continued need for Velocity in their community, and perceived that Velocity fills a gap in services. The evaluators outlined areas for improvement where the program has had less of an impact, and has been more difficult to influence or change. There is a need to place greater emphasis on strengthening the following: targeted risk factors including:

• Strategies to avoid aggressive behaviour and promote the use of non-violent strategies to control anger and resolve conflict;
• Strategies to eliminate or avoid substance use;
• Skills to address life stresses; and
• Ways to connect with the community and feel part of a team outside Velocity.

The evaluators recommended that any future evaluations of Velocity attend to the following considerations:

• Allow several months to undertake surveys of youth in order to maximize data collection with this at-risk youth.
• Provide incentives to encourage youth to participate in the evaluation. Data suggested that higher incentives yielded a higher response rate.
• Provide youth with several options for questionnaire completion. While many (in-person, over the telephone, online) options were offered to participants, the highest response was the in-person option at intake.
• Assure confidentiality. Participants need to feel comfortable that their responses are confidential. Providing online assessment tests may help youth feel at ease to answer sensitive questions honestly.
• Work closely with program staff for questionnaire completion. Velocity program staff were found to be crucial in accessing youth, especially post-program as they had made connections with youth and their families.
• Explore other strategies to clarify the attribution of the program to the outcomes especially when the control group does not have similar risk factors as the participants in the Velocity Program.

Conclusions

The overall objectives of Velocity aim to reduce anti-social behaviour, increase attachment to school and reduce substance use. The majority of findings are mixed with regards to these targeted outcomes. The qualitative findings are consistently positive.

While Velocity is perceived by stakeholders as a worthwhile and relevant initiative that is being implemented as planned, attracting at-risk youth and moving towards its intended outcomes, the comparison group's lack of comparability with the Velocity participants is an evaluation weakness and limits ability to attribute observed results to the intervention. Velocity remains a promising but still unconfirmed crime prevention model.

Date modified:

2022-07-25