Final Evaluation Summary of the Multisystemic Therapy Program

Final Evaluation Summary of the Multisystemic Therapy Program PDF Version (222KB)
Table of contents

Abstract
Multisystemic Therapy (MST) is a widely-used, evidence-based prevention program aimed at reducing anti-social behaviour and recidivism among at-risk youth. The NCPC contributed approximately $2.1 million to fund the Agincourt Community Services Association (ACSA) MST program, from 2009 to 2014. The ACSA MST program targeted at-risk youth aged 12 to 17 who were living with a primary caregiver within the boundaries of Scarborough, Ontario. The outcome evaluation examined the extent to which the intended short-term outcomes were achieved. Over 80% of the 48 program completers experienced a drop of 20 points or more on the Child and Adolescent Functional Assessment Scale (CAFAS) between baseline and their time of discharge, which is considered to be a clinically significant improvement. As for the 9 non-completers, approximately 33% showed a similar drop in CAFAS points. Three months after intake, more youth who graduated from the program than non-completers were: living at home, attending school, vocational training or employed 20+ hrs/wk, and not arrested for an offence committed during MST. At discharge, 75% of youth who graduated from the program were involved with prosocial peers and activities, compared to 29% of those who did not complete the program.

Author's Note
The views expressed are those of the authors and do not necessarily reflect those of Public Safety Canada. Correspondence concerning this report should be addressed to: Research Division, Public Safety Canada, 269 Laurier Avenue West, Ottawa, Ontario, K1A 0P8; email: PS.CSCCBResearch-RechercheSSCRC.SP@ps-sp.gc.ca.

Product Information:
© Her Majesty the Queen in Right of Canada, 2015
Cat. No.: PS113-1/2015-15E-PDF
ISBN Number: 978-0-660-02170-6

Introduction

Multisystemic Therapy (MST) is a widely-used, evidence-based prevention program aimed at reducing anti-social behaviour and recidivism among at-risk youth (Henggeler et al., 2009). The program originated in the United States in the mid-1970's, when Dr. Scott Henggeler was hired by the Virginia State Department of Pediatrics to work with some of their most difficult cases. He decided that, rather than simply bringing adolescents into a clinic for therapy, a more effective strategy would be for treatment to occur directly within the context of clients' lives: their homes, their schools, and where they spent their leisure time. After several years of further study and development, MST Services was formed in 1996 in order to disseminate the intervention more broadly, granting program licenses and offering assistance with start-up and training, as well as providing ongoing technical and quality assurance support.

Beginning in April 2010, the Agincourt Community Services Association (ACSA), funded by the National Crime Prevention Centre (NCPC), implemented MST to address aggressive and socially unacceptable behaviour among at-risk youth in the Scarborough District of Toronto, Ontario. The main objective of the program was to avert these youth from initial or further involvement in the criminal justice system. Under the original project timeline, the contribution agreement for the MST program ended on March 31st, 2013. However, a year-long extension of the Agincourt MST program was approved in April 2013, and the evaluation study was correspondingly extended. The NCPC contributed approximately $2.1 million to fund the ACSA MST program, from 2009 to 2014.

In Canada, one evaluation of a multi-site implementation of MST has been conducted (London, Mississauga, Simcoe County, and Ottawa; Leschied & Cunningham, 2002), but no impact of the program on offending behaviour could be identified. Contrary to numerous MST evaluations conducted in the United States and elsewhere (Henggeler et al., 2009), the final results of the Canadian study showed no statistically significant differences between program participants and non-participants in terms of key criminal justice outcomes, such as the number of convictions and days in custody. Consequently, there is a need to conduct further evaluations of MST programs being implemented in the Canadian context, in order to obtain additional evidence regarding program effectiveness.

The NCPC contracted an independent research firm, Harry Cummings & Associates, Inc., to conduct the impact evaluation. Valued at $250,000, the impact evaluation study started in October 2010 and ended in April 2014. All findings presented in this summary are based on the results of the final evaluation report submitted to the NCPC on July 25, 2014.

Program Description

MST is an intensive, short-term intervention of three to five months, providing approximately 60 hours of individualized treatment for each participating youth. The program focuses on the social ecology of at-risk youth, engaging both the youth and their families in order to decrease risk factors and increase protective factors with regard to family relationships, peers, school and community. The MST therapists work closely with the youth's parents in order to develop strategies to promote and monitor the youth's success at home, at school and in the community. The sessions occur at least twice a week and take place in the youth's natural environment, typically their home. The primary resources for managing and delivering the ACSA MST program consisted of one supervisor, four therapists, one data clerk, and one intake clinician/research assistant.

Target Group

The ACSA MST program targeted youth aged 12 to 17 who were living with a primary caregiver within the boundaries of Scarborough and who met one or more of the following criteria:

Risk assessments were undertaken by the ACSA MST supervisor in conjunction with the MST consultant, based on information on the Referral Form and follow-up consultation as necessary (for example, with the school or parents). Referral forms were first reviewed by the MST supervisor. Since the program's inception, the MST supervisor has used the criteria noted above as a guide for refusing or accepting referrals into intake services. This procedure was later complemented by the administration of the CAFAS (Child and Adolescent Functional Assessment Scale). The CAFAS assesses the degree of functional impairment across eight behavioural and emotional domains in children and youth (school and work, home, community, behaviour toward others, mood / emotion, self-harmful behaviour, substance abuse, thinking). The CAFAS has been used to evaluate MST and is regarded as a measure that is sensitive to change in the areas that MST targets (Timmons-Mitchell et al., 2006).

Youth were referred to the program by the Toronto Catholic District School Board (50.5%), the Children's Aid Society (24.7%), parents (5.4%), and other sources (19.4%). A total of 94 youth were accepted into the program and discharged (69 of these youth successfully completed the program and 25 youth left before completing the program). Of the 25 youth who did not complete the program, 11 were discharged due to a lack of engagement, 4 were placed in a setting where access was restricted, and 3 moved outside the program area.

Evaluation Objectives

The objectives of the evaluation were to:

Evaluation Methodology

Due to some unforeseen challenges, the overall methodological approach to the evaluation was modified. Originally, program effects were to be assessed using a repeated measures “delayed treatment” design in which program participants would be compared with an eligible group of youth waiting to receive ACSA MST services. However, maintaining a lengthy waiting list was found to be incompatible with the philosophy and goals of the MST program. Further, a suitable comparison group could not be found at the ACSA centre or in the larger Scarborough area. Therefore, youth who did not complete the program were used as the comparison group. Ultimately, 69 youth graduated from the program, while 25 left the program for other reasons including lack of engagement.

A descriptive cost analysis was undertaken. However, given the absence of CAFAS data for a significant portion of the ACSA MST treatment group (38 of the 94 participants) it was decided not to conduct a cost effectiveness analysis. A cost-savings analysis was also not completed for the ACSA MST program given the small sample size and limited comparison group.

Measurement Tools and Data Collection

In order to determine whether the ACSA MST program was being delivered as intended and producing beneficial effects for participants, data was regularly collected on both program processes and outcomes using a number of standardized instruments, including the Referral, Screening and Enrolment Tracking Form, Child and Adolescent Functional Assessment Scale (CAFAS), Therapist Adherence Measure – Revised (TAM-R), Case Tracking Form, Program Implementation Review Tracking Form, Project Cost Tracking Form, Therapist Key Informant Interview Guides, Supervisor Key Informant Interview Guides, and the Family Key Informant Interview Guide.

The focus of the impact evaluation was on the following outcome indicators: youth living at home; attending school; participating in prosocial work; offence rate; anti-social behaviour; and higher functionality as measured by the CAFAS. Repeated measures using the CAFAS occurred within 14 days of intake (T1) or immediately prior to treatment (T1a), 3 months after intake (T2), at discharge, usually 5 months after intake (T3), 6 months after intake (T4), and 12 months after intake (T5). However, very few youth/families cooperated with follow-up efforts at T4 and T5.

Where appropriate, tests of differences in means and percentages across groups, as well as more sophisticated statistical models, were conducted to assess differences in the outcome measures between the intervention and comparison groups. In these analyses, demographic variables, offending history, and other youth and family characteristics were assessed and controlled in order to increase the validity of comparisons between the intervention and comparison groups.

The analysis focused on data for 94 youth up to and including (but not beyond) discharge, due to unsuccessful attempts to obtain post-discharge follow-up data. This group was made up of 69 program graduates (discharged by mutual consent) and 25 youth who did not complete the program (discharged for other reasons including lack of engagement). Baseline and corresponding follow-up CAFAS measures were collected for a total of 57 youth (48 youth who completed the program and 9 non-completers). 

Results                           

Process-Related Findings

The process evaluation examined the extent to which the project was implemented as intended.

The program complied with most of the program practices and characteristics. A key exception relates to the overall average duration of treatment. Over the course of the program up until the end of March 2014, the average length of stay in treatment fluctuated between 33 days to 51 days higher than the anticipated 120 day average. The overall average remained very close to the upper range target of 150 days. In the final reporting period that ended in March 2014, the average length of stay in treatment was 139 days, 19 days above the anticipated 120 day average. The extended length of treatment found in the earlier reporting periods was partly the result of therapist turnover issues.

The ACSA MST team received all compulsory training as well as additional training in cognitive-behavioural therapy (CBT), suicide assessment/intervention, and compassion fatigue. The MST therapist turnover rate at ACSA was 75% in the first year of the program. Three new therapists were hired and there were no additional human resource issues up until the end of the original contract in March 2013.

With respect to program participants, a total of 125 youth were referred to the program and a total of 94 youth (75%) were accepted into the program. Of the 94 youth that were discharged from the program, 73% (69) were discharged based upon mutual agreement and 27% (25) were discharged for other reasons (did not complete the program).

The program fell short of reaching its target of having fewer than 5% of clients discharged due to lack of engagement. Approximately 12% of clients were discharged due to lack of engagement. However, it is important to note that discharges due to lack of engagement declined substantially in the last two years of the program. Approximately 82% of the youth who were discharged due to lack of engagement were discharged during the 2010–2012 period, with the remaining 18% discharged during the 2013–2014 period.

The program fell below the minimum enrolment targets. ACSA had set a goal of treating 16 youth in the first year (4 youth per therapist), 32 in year two, 34 in year three, and 32 in year four for a total of 114 youth over the 4-year service delivery period. A total of 17 youth started the program in 2010, 26 started in 2011, 29 started in 2012, and 22 started in 2013 for a total of 94 youth over the life of the program. Staff turnover issues in combination with the uncertainty of funding for year 4, as well as the initial project restart process with referrals coming in at the end of April and May 2013, affected the ability of the team to transition to a full caseload.

As the continuation of the program was uncertain beyond March 2013, the team stopped taking referrals in November 2012 to ensure that no cases would have to be closed mid-way through treatment. In November 2012 the total number of open cases amounted to 14 or 3.5 cases per therapist and the average caseload declined over the next four months as the program wound down.

Within three weeks of the program ending in March 2013, funding renewal for another year was confirmed and three of four therapists returned to their positions. Succession planning was initiated at the beginning of this period and when the previous supervisor left the organization an experienced therapist quickly transitioned to the role of supervisor. Two new therapists were recruited and participated in the 5-day orientation in July 2013. Their development was enhanced as they joined an experienced team that could provide mentoring and job shadowing. The final intake occurred in December 2013 and the average caseload declined over the next four months as the program drew to a close.

The most common risk factor experienced by the referred youth was non-violent aggressive behaviour (81% of referred youth). Other risk factors experienced by the referred youth include violent behaviour causing injury (34%) and evidence of drug use (41%). The behaviours of 25% of the referred youth placed them at imminent risk of out of home placement and 22% had been convicted of a criminal offence prior to their referral to the program. Only a small proportion of participants (7%) had been expelled from or dropped out of school prior to their referral to the program. Approximately 47% of the referred youth had marked or severe functional impairment (high risk) based on a CAFAS entry score of 100 or greater.

The program met the target for overall average adherence (an average Therapist Adherence Measure [TAM-R] score of 0.61), and 76% of the youth discharged from the program had an average score above 0.61 which is very near the adherence target of 80%.

The program exceeded the minimum program completion rate of 66%. Of the 94 youth/families that were discharged from the program, 69 (73%) were discharged as a result of completing the program (discharged upon mutual agreement). Of the 25 youth who did not complete the program, 11 were discharged due to a lack of engagement, 4 were placed in a setting where access was restricted, and 3 moved outside the program area.

Impact Evaluation Findings

The impact evaluation examined the extent to which the intended outcomes were achieved and whether there were any unintended outcomes. All of the impact evaluation findings relate to short term outcomes.

Although the program experienced some challenges in the administration of the Child and Adolescent Functional Assessment Scale (CAFAS), the data that were collected on 48 program completers revealed that the large majority experienced a positive change in their functional impairment. Just over 80% of these youth experienced a drop of 20 points or more between baseline and their time of discharge, which is considered to be a clinically significant improvement. As for the 9 non-completers, only about 33% showed a similar drop in CAFAS points.

Three months after intake, several key differences were found between youth who graduated from the program and those who did not complete the program. These are summarized below in Table 1.

Table 1: Outcomes three months after intake

Outcome

Program graduates

Program non-completers

Living at home

96% (n=65)

61% (n=14)

Attending school, vocational training or employed 20+ hrs/wk

84% (n=58)

75% (n=18)

Not arrested for an offence committed during MST

84% (n=58)

63% (n=15)

At discharge, 75% of youth who graduated from the program were involved with prosocial peers and activities, compared to 29% of those who did not complete the program.

Although the sample was small, the statistical analyses identified groups with significant differences on certain outcomes. In particular, the results suggest that older program participants are less likely to be involved with prosocial peers and activities after discharge from the program. Furthermore, program participants that have fewer people living in the home are more likely to succeed at attending school, vocational training, be employed after discharge from the program, and experience a greater change in CAFAS. These latter findings suggest that ACSA should consider strategies for program participants living in larger family units, as these youth appear to be at greater risk of not succeeding following discharge from the program.

Unfortunately, the evaluation was unable to assess the maintenance of benefits beyond the point of discharge from the program. Very few youth/families cooperated with efforts to administer the CAFAS tool following discharge from the program (e.g. 6 month and 1 year post program) and in some cases there was no forwarding contact information available when families relocated.

There was no information obtained by the impact evaluation team that identified any unintended outcomes.

Cost Analysis Findings

Total program costs for the MST program over the 2009 to 2014 period amounted to approximately $2.1 million. Approximately 4% of the total costs were accumulated in the start-up year (2009/2010), with the balance being almost equally distributed across the following three years. The largest cost item associated with the program was wages for personnel (program coordinator and therapists), which accounted for almost $1.5 million (68%) of the total program costs.

The average treatment cost for the 94 youth that were accepted into the program was $22,973, whereas the average treatment cost for the 69 youth that completed the program (discharged by mutual agreement) was $31,297. The original goal of the program was to treat approximately 114 youth over the 4 year service delivery period. Had this target been achieved, the average treatment cost for the 114 youth would have amounted to $18,943. Given that most of the program operating costs were fixed costs, it appears that the projected extra 20 youth could have been served under the same budget.

Given the absence of CAFAS data for a significant portion of the ACSA MST treatment group (38 of the 94 participants) it was decided not to run a cost effectiveness analysis. Therefore, the program cost required to produce changes in the key outcomes of interest is unknown. A cost-savings analysis was not completed for the ACSA MST program given the small sample size and limited comparison group.

Discussion

MST Service Delivery and Fidelity Recommendations

Therapist turnover issues resulted in temporary disruptions to service delivery and limited the program from operating at its intended capacity for brief periods. As noted in the literature, MST is an intensive and demanding form of intervention that can result in therapist attrition issues. A further complicating factor in the ACSA MST experience was the short notice of the one year extension that expanded the program into a fourth year. The uncertainty of job security during this phase led to the additional turnover of therapists and lost momentum in the final year of the program.

It is recommended that therapist teams be adequately staffed/sized to ensure a buffer of additional capacity and support to respond to the intensive demands of the youth/family clients.

Data Collection / Information Management Recommendations

Two separate consulting firms were contracted to complete the process and impact evaluations of the ACSA MST program. The process evaluation was initiated at the start of the program while the impact evaluation plan was developed and initiated almost a full year into the program after families had entered the program and began treatment. The data collection tools were enhanced/expanded one year into the program and as a result the demographic/family information for the early MST cases was less complete and CAFAS data was not collected on 20 families that entered the program prior to June 2011. It is recommended that the impact evaluation team be engaged with the program during the design phase or at least prior to the implementation of the program to ensure that data collection tools respond to the key evaluation questions and baseline data is collected on all participants.

The ACSA MST team experienced ongoing challenges in gaining the co-operation of youth/families to participate in the CAFAS post program assessment and administering the CAFAS tool on a regular schedule across all participants. It is recommended that sufficient program staff hours be dedicated to maintaining a standardized and routine follow-up strategy (e.g. biweekly check-ins) to monitor and confirm the location of participating youth/families for the purpose of conducting data collection consistently and on schedule across all youth/families. Consideration should be given to enlisting the assistance of the MST therapists to conduct/support post program data collection. The therapists have established rapport and trust with the youth/families and this may facilitate greater interest and cooperation among the youth/families. It is recommended that agencies implement formal agreements with youth/families specifying the importance of the evaluation and the essential role that youth/families play in the evaluation. It is recommended that agencies develop strong relationships with relevant stakeholder groups (e.g. schools, police) that could potentially provide secondary data on youth outcomes.

References

Date modified: