Research Summary: Review of Evidence-Based Registries Relevant to Crime Prevention

This report reviews American evidence-based registries relevant to crime prevention to assist in the development of a Canadian specific system for rating the level of programs’ effectiveness.

Research Summary: Review of Evidence-Based Registries Relevant to Crime Prevention PDF Version (108 KB)

Background

As part of efforts to promote an evidence-based crime prevention agenda, Public Safety Canada is exploring the feasibility of developing a Canadian specific system for rating the efficacy of programs which considers the variety of evaluation methods and designs. To assist Public Safety Canada in this work, WestEd's Justice & Prevention Research Center (jprc.wested.org) conducted a review of four prominent and internationally recognized evidence-based registries in the U.S.: 1)  University of Colorado's Blueprints for Healthy Youth Development; 2)  Substance Abuse and Mental Health Services Administration's National Registry of Evidence-based Programs and Practices (SAMHSA NREPP); 3) the Office of Juvenile Justice & Delinquency Prevention's Model Programs Guide (OJJDP MPG), and 4) the U.S. Department of Justice's CrimeSolutions.gov.

Method

This report is a review of the four selected registries that includes the collection of key information retrieved from the website of each registry and the relevant criteria used to rate studies. The guiding questions of the review included information about:  what agency funded the registry; the types of evaluation designs used to rate program effectiveness; if the development of the registry considered including qualitative/ non-experimental/ mixed-methods studies; and how potentially eligible evaluations are identified. The review also included information about other eligibility criteria that evaluation studies need to meet to be included in the registry; if a coding instrument is used to extract information from each evaluation study; what the system is for rating evidence; how effective programs are identified;  if a program scoring instrument is used;  and who conducts the reviews of evidence.

To supplement the information gathered from registries' websites, a telephone interview was conducted with a key person involved in the development of each registry. The designs and methods of each of the registries were examined, compared to the other registries, and summarized in the full research report.

Findings

Programs eligible for inclusion by the registries as “evidence-based” must have rigorous studies supporting their effectiveness, preferably through randomized control trials (RCTs), generally with replication, demonstrating effectiveness in achieving at least one of their stated outcomes in the area of interest of the registry (with minimal negative effects). Programs evaluated with rigorous quasi-experimental (QED) designs (e.g., regression discontinuity, interrupted time series, matched comparison groups) are also considered; however, Blueprints, for example, considers these programs supported by just QED evidence to be “promising,” rather than proven, with a “moderate confidence” rating in effectiveness.

Blueprints is often considered to use the most rigorous requirements for identifying an effective program. This system uses a checklist scoring system - if an evaluation does not meet all eligibility criteria, it is not rated. The checklist asks yes/no questions about design, sample, measures, outcomes, controls, attrition, fidelity, effect sizes, and long term effects. Programs are then rated as “Model Plus”, “Model” or “Promising”. Model programs have been evaluated by more than one experimental study, have significant impact and show sustained positive impacts for at least 12 months. Model plus programs meet an additional requirement which consists of an independent replication.

Crime Solutions and OJJDP MPG use a mathematical formula to determine the evidence rating for each program. The four-part rating instrument consists of a score for conceptual framework; design quality; outcome evidence; and program fidelity. Once values for each of the four categories have been tabulated, programs are assigned their final evidence rating (great effect, promising, or null) and programs are then classified as “Effective”, “Promising” or “No effects”.

The NREPP uses a scoring instrument that rates program outcomes on four dimensions: rigor, effect size, program fidelity, and conceptual framework. Numerical values are assigned to each element using the NREPP Outcome Rating Instrument and are combined to create an “outcome score.” Programs are rated as “Effective”, “Promising”, “Ineffective”, or “Inconclusive.” Effective programs have evaluation evidence with strong methodological rigor and positive short-term effects, while promising programs have evaluation evidence with sufficient methodological rigor and short-term effects that are likely to be positive.

For the four registries, if programs have evaluations indicating both positive and negative/unintended outcomes, programs are rated only if the positive outcomes are those which are prioritized by the registry. If a program has significant harmful effects, it is deemed ineffective, even if it also has positive outcomes.

While each registry has developed its own unique rating instruments, classification systems, and terminology, the aspects of evaluation methodology, program implementation fidelity, and outcomes that they examine are similar. All of the registries only consider RCTs and QEDs with comparison groups and pre/post data in rating program outcomes. However, Blueprints requires that if a program has not been evaluated by one RCT, it must have been evaluated by at least two QEDs evaluations; the other registries require only one.

Although none of the registries consider non-experimental evidence in program rating, NREPP's Learning Center will soon start to publish case studies of programs evaluated through qualitative and other non-experimental methods. Non-experimental studies are not included as evidence by these registries, as confidence in findings (internal validity) in studies using these designs are considered to be low by registry developers.

Implications

This review provides information that is critical for understanding what works and what may be useful to use in the Canadian context. While exploring the feasibility of developing a Canadian customized system, the consideration of similar work in the U.S evidence-based registries is useful for providing guidance. This review provides insight into these four registries while considering key aspects on how these systems were developed, maintained, and implemented to effectively assess the efficacy of each program.  The information gathered from this review is useful in providing information about strategies and methodological aspects that should be considered when developing a Canadian specific system.

Source

Morgan, C., Day, D., & Petrosino, P. (2017). Review of Evidence-Based Registries Relevant to Crime Prevention. Ottawa: Public Safety Canada.

Date modified:

2022-07-25