Results from the Multisystemic Therapy ProgramFootnote 1

Results from the Multisystemic Therapy Program PDF Version (277 KB)
Table of contents

Introduction

Multisystemic Therapy (MST) is a widely-used, evidence-based prevention program aimed at reducing anti-social behaviour and recidivism among at-risk youth (Henggeler et al., 2009). The program originated in the United States in the mid-1970's, when Dr. Scott Henggeler was hired by the Virginia State Department of Pediatrics to work with some of their most difficult cases. He decided that, rather than simply bringing adolescents into a clinic for therapy, a more effective strategy would be for treatment to occur directly within the context of clients' lives: their homes, their schools, and where they spent their leisure time. After several years of further study and development, MST Services was formed in 1996 in order to disseminate the intervention more broadly, granting program licenses and offering assistance with start-up and training, as well as providing ongoing technical and quality assurance support.

Beginning in April 2010, the Agincourt Community Services Association (ACSA), funded by the National Crime Prevention Centre (NCPC), implemented MST to address aggressive and socially unacceptable behaviour among at-risk youth in the Scarborough District of Toronto, Ontario. The main objective of the program was to avert these youth from initial or further involvement in the criminal justice system. To date, the NCPC has contributed approximately $2 million to fund the ACSA MST program.

In Canada, one evaluation of a multi-site implementation of MST has been conducted (London, Mississauga, Simcoe County, and Ottawa; Leschied & Cunningham, 2002), but no impact of the program on offending behaviour could be identified. Contrary to numerous MST evaluations conducted in the United States and elsewhere (Henggeler et al., 2009), the final results of the Canadian study showed no statistically significant differences between program participants and non-participants in terms of key criminal justice outcomes, such as the number of convictions and days in custody. Consequently, there is a need to conduct further evaluations of MST programs being implemented in the Canadian context, in order to obtain additional evidence regarding program effectiveness.

This summary provides an update on the process and impact evaluation of the ACSA MST program that is being funded by the NCPC.Footnote 2 The NCPC contracted an independent research firm, Harry Cummings & Associates, Inc., to conduct the impact evaluation. Valued at $250,000, the impact evaluation study started in October 2010 and will end in April 2014.

Program Description

MST is an intensive, short-term intervention of three to five months, providing approximately 60 hours of individualized treatment for each participating youth. The program focuses on the social ecology of at-risk youth, engaging both the youth and their families in order to decrease risk factors and increase protective factors with regard to family relationships, peers, school and community. The MST therapists work closely with the youth's parents in order to develop strategies to promote and monitor the youth's success at home, at school and in the community. The sessions occur at least twice a week and take place in the youth's natural environment, typically their home. The primary resources for managing and delivering the ACSA MST program consist of one supervisor, four therapists, one data clerk, and one intake clinician/research assistant.

Target Group

The ACSA MST program targeted youth aged 12–17 who were living with a primary caregiver within the boundaries of Scarborough and who met one or more of the following criteria:

Youth are referred to the program by the Toronto Catholic District School Board (55%), the Children's Aid Society (15%), parents (7.5%), and other sources (22.5%). A total of 58 youth meeting one or more of the aforementioned criteria had been accepted into the program as of March 2012, and a projected total of 96 to 111 participants are expected to be served by the program.

Evaluation Objectives

The objectives of the evaluation are as follows:

Evaluation Methodology

Due to some unforeseen challenges, the overall methodological approach to the evaluation has been modified since the beginning of the study. Originally, program effects were to be assessed using a “delayed treatment” design in which program participants would be compared with an eligible group of youth waiting to receive ACSA MST services. However, maintaining a lengthy waiting list was found to be incompatible with the philosophy and goals of the MST program. Further, a suitable comparison group could not be found at the ACSA centre or in the larger Scarborough area. Therefore, youth who did not complete the program will be used as the comparison group. Currently, 28 youth have graduated from the program, while 12 have left for other reasons (six were discharged due to a lack of engagement, three were placed in a setting where access was restricted, and three moved outside the program area). In the final analysis, statistical modeling will be used to control differences between the two groups (e.g., youth demographics, offending history, school performance, family characteristics, etc.) that might otherwise confound the assessment of program impact. In addition, program fidelity (i.e., degree of adherence by therapists to the principles of MST) will be used as a predictor of program outcomes, which will help mitigate the fact that an ideal comparison group is not available to examine the effectiveness of MST.

Further, a cost-effectiveness analysis of the program will be conducted, in an attempt to answer the question: “How much bang for our crime prevention buck?” Essentially, a cost-effectiveness analysis compares overall program costs to the observed differences in outcomes between the comparison and intervention groups, in order to determine the program cost required to produce changes in the key outcomes of interest.

Measurement Tools and Data Collection

In order to determine whether the ACSA MST program is being delivered as intended and producing beneficial effects for participants, data is being regularly collected on both program processes and outcomes using a number of standardized instruments, including the Referral, Screening and Enrolment Tracking Form, Child and Adolescent Functional Assessment Scale (CAFAS), Therapist Adherence Measure – Revised (TAM-R), Case Tracking Form, Program Implementation Review Tracking Form, Project Cost Tracking Form, Therapist Key Informant Interview Guides, Supervisor Key Informant Interview Guides, and the Family Key Informant Interview Guide.

Repeated measures will occur at intake (T1), immediate pre-intervention (T2), 3 months after intake (T3), immediate post-intervention (T3a), and 6 months (T4), and 12 months (T5) post-intake.

Where appropriate, tests of differences in means and percentages across groups, as well as more sophisticated statistical models, will be conducted to assess differences in the outcome measures between the intervention and comparison groups. In these analyses, demographic variables, offending history and other youth and family characteristics will be assessed and controlled in order to increase the validity of comparisons between the intervention and comparison groups.

Key Findings

Following is an overview of the key findings obtained thus far with regard to major process and outcome indicators. When considering these results, it is important to keep in mind the following two limitations: (1) the results only reflect the data available at the mid-point of the evaluation, and could therefore change over the remainder of the study; and (2) because statistical modeling and inference are not possible given the low sample size at this point in the evaluation, differences between the intervention and comparison groups cannot yet be adjusted for or controlled, and no statistical tests are available to judge the significance of the findings. Therefore, the findings should be considered preliminary, pending further confirmation and validation with a larger sample.

Process-related findings

The program is on track to meet enrolment targets. As of March 2012, a total of 58 youth have been accepted into the program, and it is expected that the final projected total of 96 to 111 participants will be achieved by the program. Further, all youth accepted into the program have met one or more of the inclusion criteria. The most common risk factor experienced by the referred youth is non-violent aggressive behaviour (about 75% of all referrals).

Regarding program fidelity, 65% of participants who provided scores on the Therapist Adherence Measure – Revised (TAM-R) rated their therapists as being sufficiently consistent with MST principles, but this is below the recommended target of 80% (Henggeler, Borduin, Schoenwald, Huey, & Chapmann, 2006). However, these results are only based on 31 discharged youth. A better determination of the overall level of fidelity can be made once a larger sample of TAM-R scores becomes available.

Of the 40 youth who were discharged from the program up to the end of March 2012, 28 (70%) graduated from the program, which exceeds the minimum completion target of 66%. However, six youth (15% of clients) were discharged due to lack of engagement during this same period, which does not meet the target of having fewer than 5% of clients discharged for this reason. Therefore, more focus is needed on methods for increasing client engagement in order to reduce program drop-outs and premature discharges.

Impact-related findings

A number of positive results have been obtained at this point with regard to key program outcomes. For example, of the 13 discharged youth (eight graduates and five non-completers) who were at imminent risk of out-of-home placement at program entry, 88% of program graduates were still living at home three months after intake, compared to 60% of non-completers. Further, at three months post-intake, 90% of graduates had managed to avoid arrest during treatment, as compared to only 67% of non-completers. Further, at program discharge, completers and their families were ahead of non-completers in a variety of other specific domains:

As well as the above outcomes, an additional goal of the impact assessment was to track change over time in the Child and Adolescent Functional Assessment Scale (CAFAS), a measure which combines information on problems experienced by youth across a number of different life areas (e.g., school, emotion, behaviour, etc.), as well as issues encountered by primary caregivers in particular domains (material needs and family/social supports). A reduction of 20 points or more in the overall CAFAS score represents a meaningful improvement (Hodges, 2005). However, due to problems with scheduling follow-up assessments with families, the availability of CAFAS data is currently too limited to reliably report and interpret any results. Therefore, the evaluation team will strive to increase youth/family engagement with and participation in CAFAS assessments.

In addition, the evaluators will conduct a cost-effectiveness analysis (CEA) of the ACSA MST program, which attempts to answer the question: “How much bang for our crime prevention buck?” Two components are required to perform a CEA: (1) total program costs; and (2) the difference in key outcomes between the comparison and intervention groups. Essentially, CEA provides the average cost for producing a unit change in an outcome variable. For example, in the current evaluation, the cost needed to produce a meaningful difference in the CAFAS measure (i.e., a reduction of 20 points or greater) will be calculated. Further, a cost-savings analysis will also be performed, in order to examine the monetary savings generated by preventing certain adverse outcomes, such as out-of-home placements which carry a high administrative cost.

Next steps

Over the next reporting period, efforts will be increased with regard to improving information management, with the aim of facilitating systematic and regular entry of information from therapists' case files into the key instruments and evaluation database, as well as maintaining participant engagement for data provision both during enrolment and beyond discharge from the ACSA MST program. In particular, youth/families will be continuously encouraged to complete the TAM-R and CAFAS measures.

Reporting timelines

A second annual report is due in December 2012, and the final evaluation report is due in March 2014.

References

Henggeler, S.W., Borduin, C.M., Schoenwald, S.K., Huey, S.J., & Chapman, J.E. (2006). MST Therapist Adherence Measure – Revised (TAM-R). Charleston, SC: Medical University of South Carolina, Family Services Research Center.

Henggeler, S.W., Schoenwald, S.K., Borduin, C.M., Rowland, M.D., & Cunningham, P.B. (2009). Multisystemic Therapy for Antisocial Behavior in Children and Adolescents (2nd ed.). New York, NY: The Guilford Press.

Hodges, K. (2005). Child and Adolescent Functional Assessment Scale. In T. Grisso, G. Vincent, & D. Seagrave (Eds.), Mental health screening and assessment in juvenile justice (pp. 123-136). New York, NY: Guilford Press.

Leschied, A.W. & Cunningham, A. (2002). Interim Results of a Four-Year Randomized Study of Multisystemic Therapy in Ontario, Canada. London, ON: Centre for Children and Families in the Justice System of the London Family Court Clinic.

For more information or to receive a copy of the final evaluation report, please contact the National Crime Prevention Centre by e-mail at ps.prevention-prevention.sp@canada.ca.

If you wish to register for the NCPC mailing list to receive information from the Centre, please visit the subscription page at: http://www.publicsafety.gc.ca/cnt/bt/mlng-lst-eng.aspx

Footnotes

  1. 1

    The technical authority on this evaluation contract at NCPC is Cameron McIntosh, Senior Research Advisor.

  2. 2

    All findings presented here are based on the results of the mid-term evaluation report submitted to the NCPC on September 17, 2012. Additional findings presented in the second annual and final reports will be included in a subsequent summary.

Date modified: