Research Summary: Multisystemic Therapy Program Evaluation
Background
Multisystemic Therapy (MST) is a widely-used, evidence-based prevention program aimed at reducing anti-social behaviour and recidivism among at-risk youth. The program focuses on the social ecology of at-risk youth, engaging both the youth and their families in order to decrease risk factors and increase protective factors with regard to family relationships, peers, school and community.
The NCPC contributed approximately $2.1 million to fund the Agincourt Community Services Association (ACSA) MST program, from 2009 to 2014.
The ACSA MST program targeted at-risk youth aged 12-17 who were living with a primary caregiver within the boundaries of Scarborough, Ontario.
The objectives of the evaluation were to:
- Assess the extent to which the project is being implemented as intended;
- Assess whether the intended outcomes were achieved, and whether there were any unintended outcomes;
- Provide a descriptive cost analysis for the project and cost effectiveness analysis of key outcomes;
- Identify lessons learned, exploring what worked well and what did not work as well, and make recommendations to strengthen the project for the benefit of others interested in implementing or supporting a project of this nature in the future; and
- Assess the extent to which the project was adapted to meet the needs of the youth and the community.
Method
Due to some unforeseen challenges, the overall methodological approach to the evaluation was modified. Originally, program effects were to be assessed using a repeated measures “delayed treatment” design in which program participants would be compared with an eligible group of youth waiting to receive ACSA MST services. However, maintaining a lengthy waiting list was found to be incompatible with the philosophy and goals of the MST program. Further, a suitable comparison group could not be found at the ACSA centre or in the larger Scarborough area. Therefore, youth who did not complete the program were used as the comparison group. Ultimately, 69 youth graduated from the program, while 25 left the program for other reasons including lack of engagement.
The Child and Adolescent Functional Assessment Scale (CAFAS) was administered within 14 days of intake (T1) or immediately prior to treatment (T1a), 3 months after intake (T2), at discharge, usually 5 months after intake (T3), 6 months after intake (T4), and 12 months after intake (T5). However, very few youth/families cooperated with follow-up efforts at T4 and T5.
A descriptive cost analysis was undertaken. However, given the absence of CAFAS data for a significant portion of the treatment group (38 of the 94 participants) it was decided not to conduct a cost effectiveness analysis. A cost-savings analysis was also not completed for the ACSA MST program given the small sample size and limited comparison group.
Findings
The evaluation examined the extent to which the intended short-term outcomes were achieved. Baseline and corresponding follow-up measures were collected for 48 youth who completed the program and 9 non-completers. Over 80% of the 48 program completers experienced a drop of 20 points or more on the CAFAS between baseline and their time of discharge, which is considered to be a clinically significant improvement. As for the 9 non-completers, approximately 33% showed a similar drop in CAFAS points.
Three months after intake, more youth who graduated from the program than non-completers were:
- Living at home (96% of graduates; 61% of non-completers);
- Attending school, vocational training or employed 20+ hrs/wk (84% of graduates; 75% of non-completers);
- Not arrested for an offence committed during MST (84% of graduates; 63% of non-completers).
At discharge, 75% of youth who graduated from the program were involved with prosocial peers and activities, compared to 29% of those who did not complete the program.
The evaluation was unable to assess the maintenance of benefits beyond discharge from the program as very few youth/families cooperated with efforts to administer the CAFAS tool at 6-months and 1-year post-program.
Implications
MST is an intensive and demanding form of intervention that can result in therapist attrition issues. It is recommended that therapist teams be adequately staffed/sized to ensure a buffer of additional capacity and support to respond to the intensive demands of the youth/family clients.
In order to ensure that data collection tools respond to the key evaluation questions and baseline data are collected on all participants, it is recommended that the evaluation team be engaged with the program during the design phase or at least prior to the implementation of the program.
The ACSA MST team experienced ongoing challenges administering the CAFAS tool on a regular schedule across all participants. It is recommended that sufficient program staff hours be dedicated to maintain a standardized and routine follow-up strategy (e.g. biweekly check-ins) to monitor and confirm the location of participating youth/families for the purpose of conducting data collection consistently and on schedule across all youth/families. Consideration should be given to enlisting the assistance of the MST therapists to conduct/support post program data collection. The therapists have established rapport and trust with the youth/families and this may facilitate greater interest and cooperation among the youth/families. It is recommended that agencies implement formal agreements with youth/families specifying the importance of the evaluation and the essential role that youth/families contribute play in the evaluation. It is recommended that agencies develop strong relationships with relevant stakeholder groups (e.g. schools, police) that could potentially provide secondary data on youth outcomes.
Source
Harry Cummings and Associates Inc. (2014). Evaluation of the Multisystemic Therapy Program, Toronto, Ontario: Final Report.
For more information on research at the Community Safety and Countering Crime Branch, Public Safety Canada, to get a copy of the full research report, or to be placed on our distribution list, please contact:
Research Division, Public Safety Canada
269 Laurier Avenue West
Ottawa, Ontario
K1A 0P8
PS.CSCCBResearch-RechercheSSCRC.SP@ps-sp.gc.ca
Research Summaries are produced for the Community Safety and Countering Crime Branch, Public Safety Canada. The summary herein reflects interpretations of the report authors' findings and do not necessarily reflect those of the Department of Public Safety Canada.
- Date modified: