2010-2011 Evaluation of the Policy Development Contribution Program - Final Report

2010-2011 Evaluation of the Policy Development Contribution Program - Final Report PDF Version (302 KB)
Table of contents

Executive Summary

Evaluation supports accountability to Parliament and Canadians by helping the Government of Canada to credibly report on the results achieved with resources invested in programs. Evaluation supports deputy heads in managing for results by informing them about whether their programs are producing the outcomes that they were designed to achieve, at an affordable cost; and supports policy and program improvements by helping to identify lessons learned and best practices.

What we examined

The evaluation examined the Policy Development Contribution Program's relevance (ongoing need and alignment with federal government's roles, responsibilities and priorities) and performance (effectiveness, efficiency and economy). This study covers the period starting on April 1, 2005.

The Policy Development Contribution Program supports policy development through funding allocations to eligible recipients. The Department provides funding to three categories of projects: communication and information exchange projects in priority areas of criminal justice, emergency management and national security; innovation and research projects that support public policy issues; and training and skills development projects.

Over the five years from 2005-2006 to 2009-2010, an average of $2 million in annual funding was awarded to recipients, from four Public Safety Canada branches.

Why it is important

Grants and contribution programs are of central importance as instruments of public policy. Public Safety Canada's Policy Development Contribution Program supports strategic projects that contribute to policy-making and improved service delivery in the areas of public safety and emergency management. The overall objectives of contributions are to forge greater consensus and cooperation among various levels of government and other stakeholders through policy initiatives and activities.

What we found

There is a continued need for activities funded under the Policy Development Contribution Program to assist in the development of public policy for Public Safety Canada. The Program supports strategic projects that contribute to policy development and to enhanced service delivery in the areas of public safety and emergency management. There continues to be participation in the Program by funded recipients that assists in filling public policy needs. In general terms, contribution programs remain an important source of funding for recipient organizations that are highly dependent on government transfer payments.

The Program is aligned with government priorities and federal roles and responsibilities. This is evident in various government source documents and Speeches from the Throne. The Program's objectives remain relevant because they support the Department with respect to legislation, public policy development, programs and enhanced service delivery. In terms of the federal role, document review highlights the central importance of grants and contributions as instruments of public policy.

Key program management representatives interviewed for the evaluation stated that the Program achieved most of the expected immediate and intermediate outcomes. These respondents indicate that they regularly consult recipient organizations on numerous topics of interest to Public Safety Canada in order to deepen their knowledge and understanding of these issues and that the relationships allow the Department to obtain relevant advice and opinions on public policy development and on enhanced service delivery. These key informants also stated that funded projects have indeed helped to build the capacity of local communities to work with victims, offenders, their families, and local representatives. Among the criticisms raised during the interviews is the increasing difficulty of selecting new proposals that clearly meet program objectives, resulting in the repetition of some projects in various communities.

In terms of financial efficiency, the average amount used to administer the fund annually equates to 8.2% of the total annual amount of the contributions.

With regard to the dissemination of program information and daily management of the Program, although the Program is promoted on the Public Safety Canada website, for the most part it remains the domain of organizations that already have a relationship with the Department. Some project managers believe that they would benefit from training and better coaching for contribution administration in their respective branch. The evaluation found, however, that departmental training for the Program has been provided in 2010-2011, with more in 2011-2012, to provide guidance and develop common tools in the administration of grants and contributions, to address identified needs within the organization. Some program recipients are frustrated with the complexity and slowness of the Program's administrative processes.

With respect to governance, it appears that the governance mechanisms (direction-setting processes) of the Program have "eroded" from their original intent. Inception documents provided for strategic oversight regarding the management of program funds; however, over the years, this strategic view, even to coordinate decisions among the branches, has not been as present. The evaluation team found confusion and difference of opinion concerning program management, departmental authorities, the distribution of funding, as well as the roles and responsibilities of its various actors. For example, earmarked project funding allocations within individual branches (designated under separate decisions) do not necessarily take advantage of the strategic perspective that was to be offered by the Contribution Review Committee. The current governance of the Committee may not be suitable given the April 2010 decision to amalgamate funding from earmarked sources.

In terms of economy, the Program's key representatives consider the Program to be an economic means of achieving Public Safety Canada objectives, given the outcomes and the Program's low administrative cost.

Recommendations

It is recommended that:

  1. Clear governance and strategic oversight be established by assigning functional accountability to the Assistant Deputy Minister, Strategic Policy Branch.
  2. In light of the overarching decision to amalgamate earmarked and general funding into a single fund (in the amount of approximately $897,000), the Program's decision-making mechanisms should be redesigned to strengthen strategic management of the fund. The decision-making body, which could be a reconstituted Contribution Review Committee or a new committee, should be empowered to perform its strategic role. To this end, the committee should be of the appropriate size and composition so that there is participation at the appropriate level of the organization and so that strategic decision-making is facilitated. The project selection process and criteria should be revised in order that all projects are considered on an equal basis.
  3. An outcome-based Performance Measurement Strategy should be developed and implemented for the Program. The Performance Measurement Strategy should be aligned with current Treasury Board Secretariat guidance including the Guideline on Performance Measurement Strategy under the Policy on Transfer Payments. In this regard, outcome-based information may be required from funded recipients in order to sufficiently fulfill reporting requirements.

Management Response and Action Plan

The Assistant Deputy Minister, Strategic Policy Branch, accepts the recommendations of this evaluation and proposes the following management action plan:

Response to Recommendation 1:

Strategic Policy Branch agrees to assume the functional accountability of the Policy Development Contribution Program, noting the close partnership that will be required with Corporate Management Branch, which is expected to continue managing departmental transfer payment policies and processes, including for the Policy Development Contribution Program. The target date is January 2012. This date should allow for sufficient time to prepare for the new round of Policy Development Contribution Program proposals for the 2012-2013 fiscal year.

Response to Recommendation 2:

Strategic Policy Branch agrees that a more strategic decision-making mechanism is required, particularly with fewer resources available following Strategic Review, and will lead the redesign based on the recommendations of the 2010-2011 Evaluation of the Policy Development Contribution Program. The target date is January 2012. Involvement of all branches will be critical to the future success of the program.

Response to Recommendation 3:

Strategic Policy Branch agrees that an outcome-based Performance Measurement Strategy should be developed and implemented for the Policy Development Contribution Program. The target date is March 2012 to be ready for the 2012-2013 fiscal year, while also giving time for the new decision-making structure to be created and review the Performance Measurement Strategy before implementation.

1. Introduction

This report presents the findings of the 2010-2011 Evaluation of the Policy Development Contribution Program (hereinafter referred to as the "Program" or the "PDCP"). This evaluation, which covers the period starting on April 1, 2005, was conducted by the Evaluation Directorate.

Public Safety Canada's PDCP supports strategic projects by departmental stakeholder organizations that contribute to policy-making and improved service delivery in the areas of public safety and emergency management. Eligible recipients of the Program include provinces, territories, public and private non-profit organizations, Aboriginal governments, non-governmental organizations (NGOs).Note 1

This evaluation report presents a profile of the Program, determines the extent to which a demonstrable need has been met, whether the Program is appropriately delivered by the federal government, and whether it meets the needs of Canadians. The evaluation also aims to determine just how effective, efficient and economic the Program has been. The evaluation supports the obligation to report to Parliament and to Canadians as it fosters the creation of credible reports on the outcomes achieved with the resources invested in federal government programs.

2. Profile

2.1 Background

The PDCP is a transfer payment program, administered by PS that provides contribution funding in order to achieve its objectives. According to the Treasury Board Policy on Transfer Payments contributions are: "a transfer payment subject to performance conditions specified in a funding agreement. A contribution is to be accounted for and is subject to audit."Note 2 Through its contribution agreements, the PDCP contributes to the achievement of departmental legislative, policy development, and consultative objectives by supporting:

The PDCP was not originally conceived, nor managed as a single integrated program. Over the years, the Program was managed as a number of distinct program initiatives under different branches using the same Treasury Board-approved terms and conditions. Thus, several branches are involved in the administration of the Program. The Corporate Management Branch has had a facilitating/coordinating role for general funding under the PDCP with no functional, nor oversight role. The Community Safety and Partnerships Branch and the Law Enforcement and Policing Branch have participated in the administration of earmarked funding under the Program.

Over its history, the PDCP has undergone a number of changes. The precursor to the PDCP was created in 1983, under the former Solicitor General of Canada, and was named the Departmental Sustaining Grants and Contributions Program. After its inception, the Program was separated into two entities, one of them being the PDCP and the other being the Grants Program to National Voluntary Organizations (which is the subject of a separate evaluation).

Over the past several years, most of the funding for PDCP was provided by individual branches of PS under earmarked funding allocated under separate initiatives. These earmarked funds were designated for use by the individual branches, under separate decisions. The existing terms and conditions of the PDCP were utilized by the separate initiatives so as not to create duplication and so that funds could be dispersed within the terms of the PDCP. Limited general funding was also provided to the PDCP. Thus, over the years, PS has administered the PDCP using a two-pronged approach:

Four separate initiatives have used earmarked funding as follows*:

Community Safety and Partnerships Branch:

Law Enforcement and Policing Branch:

*Note - the foregoing initiatives have been the subject of separate evaluations, either by Public Safety Canada and as appear on the departmental website (see - http://www.publicsafety.gc.ca/cnt/rsrcs/pblctns/index-eng.aspxeng.aspx), or with PS participation in a horizontal evaluation led by another government institution, with reporting done via that departmental website, such as Department of Justice for evaluation of the National Anti-Drug Strategy.

In addition to having earmarked funding, the aforementioned branches have also accessed general funding. The Strategic Policy Branch and the Emergency Management and National Security Branch have only accessed general funding of the PDCP as they have not had earmarked funding.

In 2009-2010, a decision was made to create a separate contribution program for the National Strategy for the Protection of Children from Sexual Exploitation on the Internet. This program is the Contribution Program to Combat Child Sexual Exploitation and Human Trafficking. This removed the earmarked funding for the National Strategy from the PDCP.

In April 2010, a decision was made to change the PDCP into a single integrated program. This decision was meant to amalgamate and reduce funding in the PDCP so that by April 2011, funding formerly used for Aboriginal Community Corrections; Enhancing Community Corrections Infrastructure; and, the National Anti-Drug Strategy would be considered on an equal basis with other projects.

2.2 Program Resources

2.2.1 Financial

As illustrated in Table 2, from 2005-2006 to 2009-2010, the funding profile for PDCP was about $10 million or an average of about $ 2 million dollars per year.

Table 2 - Budget Planning for Earmarked and General Funding Contributions by Branch and Fiscal Year (dollar values rounded to the nearest thousand)
Branch/Separate Initiative 2005-2006 2006-2007 2007-2008 2008-2009 2009-2010
Earmarked Funding - Community Safety and Partnerships Branch
Aboriginal Community Corrections $225,000 $225,000 $225,000 $225,000 $225,000
Community Corrections $0 $0 $501,000 $500,000 $500,000
Subtotal $225,000 $225,000 $726,000 $725,000 $725,000
Earmarked Funding - Law Enforcement and Policing Branch
National Strategy to for the Protection of Children from Exploitation on the Internet $700,000 $700,000 $700,000 $700,000 $50,000*
National Anti-Drug Strategy $175,000 $175,000 $175,000 $175,000 $175,000
Subtotal $875,000 $875,000 $875,000 $875,000 $225,000
General Funding - All Branches
Subtotal $652,000 $652,000 $662,000 $662,000 $662,000
Total $1,752,000 $1,752,000 $2,263,000 $2,262,000 $1,612,000

*Note: As of 2009-2010, funds that were formerly listed as earmarked funds under the PDCP, were reallocated to the Contribution Program to Combat Child Sexual Exploitation and Human Trafficking (CPCCSEHT). The exception to this change in administration is that $50,000 for Crime Stoppers for 2009 2010 was not reallocated to CPCCSEHT and remained with PDCP.

2.2.2 Human Resources

A combined total of 1.0 FTE (0.5 FTE at the PM-01 and 0.5 FTE at the PM-05 level) from the two main branches mentioned above participate in the management of contribution agreements and related projects. The two other branches are estimated to expend 0.5 FTE at the PM-03 level to administer PDCP agreements under the general funding component.

Two resources from the Grants and Contributions Support Services Unit reporting to the Corporate Management Branch provide a coordination function for the overarching Contribution Review Committee (CRC) that decides on funding allocation for the general funding component only. These resources estimate that they spend about 10% of their annual workload (or about 0.2 FTEs) managing the Program, developing tools such as agreements and directives, in support of the management of grants and contributions, including templates for use by the branch for PDCP projects.

The CRC which comprises 10 members meets twice per year to review proposals, contributing 0.01 FTEs at the EX-01 and 0.01 FTEs at the EX minus one level to the Program.

2.3 Eligible Project and Recipients

Under the Program, the Department provides funding to three categories of projects:

Contributions under the PDCP may be provided to the following classes of recipients in support of its objectives:

  1. National, provincial, territorial, municipal, Aboriginal, community or professional organizations, societies and associations that have voluntarily associated themselves for a not-for-profit purpose, and that have a mandate to represent their membership or community;
  2. Canadian universities and educational institutions;
  3. Provincial, municipal and Aboriginal police services;
  4. Aboriginal and non-status governments, organizations and communities;
  5. Provincial, territorial, municipal and regional governments;
  6. International non-governmental organizations, including bodies associated or affiliated with organizations of which Canada is a member, that have as their purpose supporting public safety and/or emergency management as priorities; and,
  7. Not-for-profit organizations and associations serving the private sector.

Crown Corporations, for profit groups, and individuals are not eligible for funding under the PDCP.

2.4 Governance Structure

Governance of the PDCP, for general funding, is overseen by the CRC, which consists of representatives from all branches of the Department. The CRC is chaired by the Comptroller of the Corporate Management Branch, who is assisted by the Manager,

Grants and Contributions Support Services Unit. As mentioned previously, individual branches manage their own agreements under the earmarked portion of the Program.

2.5 Program Outcomes

The activities and outputs of the programs are expected to lead to the immediate and intermediate outcomes as outlined in PDCP inception documents. As indicated in the logic model in Figure 1, some of these outcomes are more directly associated with contribution funding, while others are related to internal PS policy development.

Figure 1: Logic Model for the PDCP

Logic Model for the PDCP
Image Description

Immediate Outcomes

  • Innovative approaches, lessons learned and best practices documents.
  • Shared knowledge and awareness and enhanced delivery of services.
  • Improved inter-sectoral/inter-disciplinary collaboration and delivery of services.
  • Increased knowledge and understanding for decision-making in Canada's public safety and emergency management systems.

Intermediate Outcomes

  • Increased public education and confidence in Canada's criminal justice, emergency management, and national security systems.
  • Increased community capacity to work with victims, offenders, families and local officials such as police and emergency first responders.
  • Recommendations to modify policies and legislation.

The intermediate outcomes are expected to lead to the following ultimate outcome.

  • A safe and resilient Canada.

While many other social and political factors are expected to affect this outcome, it is still expected that the programs will contribute in a positive way to the achievement of this outcome.

3. About the Evaluation

3.1 Methodology

The 2010-2011 evaluation of the PDCP covers the period starting on April 1, 2005. This evaluation meets Treasury Board requirements of providing evidence based on a neutral assessment of a program's relevance and performance, as set out in the Treasury Board Policy on Evaluation.

This evaluation aims to:

3.1.1 Evaluation Issues

In accordance with the Treasury Board Secretariat Directive on the Evaluation Function, the five core evaluation issues were addressed as part of this evaluation, concerning the Program's relevance and performance:

Relevance

Performance

3.1.2 Evaluation Framework

The methodology is based on an evaluation framework that includes issues, evaluation questions, and proposed data-collection methods. This evaluation framework was approved by the PS Director General, Evaluation, on January 14, 2011.

3.2 Data-Collection Methods

In the course of the evaluation process, the evaluation team used multiple lines of evidence in order to obtain data from multiple perspectives. The information and data collected were then synthesized and integrated in order to support the main findings and conclusions. Three lines of evidence were used in the data-collection process:

The paragraphs below summarize the three data-collection methods used as part of this evaluation.

3.2.1 Literature and Document Review

The literature and document review gave the evaluators an understanding of the Program's context, environment and evolution over the past few years. Moreover, the review provided reliable evidence for most of the evaluation indicators.

Various types of documents were studied including: previous evaluations, government and non-government reports, and administrative documents for the Program. Appendix A presents a comprehensive list of the works and documents consulted as part of this evaluation.

3.2.2 Interviews and Discussion Groups with Key Representatives

The evaluation team conducted interviews with 11 key representatives of the Program from departmental directorates. A single version of the questionnaire was used for all interviews, so as to ensure their consistency and uniformity.

In addition, the respondents were asked to fill out a questionnaire prior to the interview. The questionnaire consisted of nine open-ended questions and seven structured questions. For the structured questions, the respondents were asked to rate the statements using the following value scale:

Value Scale
1 (not at all) 2 (minimal) 3 (moderate) 4 (significant) 5 (to a high degree)
0.0-1.0 1.01-2.0 2.01-3.0 3.01-4.0 4.01-5.0

3.2.3 Case Studies

For in-depth analysis of a specific event considered representative of a typical situation, this report presents several case studies. Analysis of the case studies provides an overview of a few of the Program's meaningful outcomes. Information from the four case studies is included in Section 4.2.

3.3 Methodological Limitations

In the course of the evaluation process, the evaluation team had to manage methodological limitations that had a bearing on subsequent analyses and interpretation of the collected data. These are as follows:

Limited number of key representatives and available data

Given the nature of the Program (i.e., contribution program), its scope and its limited resources, the evaluation team was only able to meet with a small number of the PDCP's key representatives. In addition, limited outcome-based performance information was available for the Program. This information may have been difficult to find, given the current de-centralized administration of the Program, i.e. with Branch responsibility for projects and no cross-cutting "owner" of a performance measurement strategy.

Causal relationship between activities and Program outcomes

The Program's nature and context make it difficult to establish a causal relationship between activities and Program outcomes. Thus, it is important to note that some outcomes measured in this evaluation may be the result of multiple factors, both internal and external to the Program. In most cases, outcomes are attributable to a combination of activities and programs that complement the PDCP. However, the use of case studies as a line of evidence serves to partially address this limitation.

4. Findings

The following sections present the main findings concerning the two major issues covered by this evaluation: relevance and performance. These key findings stem from the data and information collected using the research methods described in Section 3.

4.1 Relevance

In order to assess the components of Program relevance, three issues were addressed, as illustrated in Table 3.

Table 3 - Relevance-related Issues
4.1.1 - Continued need for the program
4.1.2 - Alignment with government priorities
4.1.3 - Alignment with Federal Roles and Responsibilities

4.1.1 Continued Need for the Program

PS contributions to PDCP recipients are intended to support the Department with regard to the development of legislation, public policy and programs and the enhancement of service delivery. In this sense, most of the key PS representatives interviewed during the evaluation consider the Program essential to achieving these objectives.

During the interviews with the PDCP's key representatives, several benefits of the PDCP program were noted including the importance of recipient organization activities; the importance of recipient's representatives to strategic reflection and public policy development at PS; and, enhancements to the Department's service delivery. In this regard, one respondent stated that the recipients are the eyes, ears and voices of Program representatives and allow for policy-related dialogue through ongoing consultations.

Findings from document review indicate that there has been, and continues to be, "program uptake" related to filling public policy needs. Contributions to recipients meet a range of organizational needs related to public policy and to the Department's service delivery. The PDCP's budget planning allows for funding an average of close to 43 projects per year (within the categories of PDCP projects, i.e. training and skills development; innovation and research; communication/information exchange). The majority of the contributions envelope, or about 70% of the total amount, has been awarded to the five separate earmarked initiatives previously mentioned (as shown in Table 2).

Contributions earmarked under the National Strategy for the Protection of Children from Sexual Exploitation on the Internet enable the Department to fund eligible projects or initiatives that support education and public awareness as well as research and targeted initiatives, such as organizing or holding training sessions or conferences to further knowledge on human trafficking and the sexual exploitation of children on the Internet.

Funding for contributions under the Aboriginal Community Corrections and Community Corrections initiatives strengthens knowledge and expertise related to correctional services within Aboriginal communities. It also builds the capacity of these communities to take on new challenges and to implement innovative provisions for First Nations, the Métis, the Inuit and urban communities.

Funding for the National Anti-Drug Strategy supports research projects and knowledge advancement related to supply reduction and the Enforcement Action Plan under the National Anti-Drug Strategy, which is led by the Department. The National Anti-Drug Strategy, announced in October 2007, focuses on three priority areas: combating illicit drug production and distribution; preventing illicit drug use; and, treating and rehabilitating those with illicit drug dependencies. Specific to combating the production and distribution of illicit drugs, Public Safety Canada leads the Enforcement Action Plan, which proactively targets organized crime involvement in illicit drug production and distribution operations, including marijuana grow operations and clandestine laboratories. As the lead for the Enforcement Action Plan, Public Safety Canada is responsible for ongoing information and knowledge dissemination for the advancement of enforcement efforts, which includes the management of contribution funds (earmarked funds using PDCP terms and conditions) to support research projects and knowledge related to drug supply reduction in Canada.

The general funding component includes, for its part, projects concerning correctional services, national security, law enforcement and emergency management.Note 3

4.1.2 Alignment with Government Priorities

In recent years, the Government of Canada has repeatedly emphasized that public safety is one of its priorities, whether in the Speeches from the Throne from 2006 to 2010 or in the many public speeches by the Minister of Public Safety. For example, the Speech from the Throne to open the second session of the 39th Parliament of Canada stated that "there is no greater responsibility for a government than to protect (the) right to safety and security."Note 4

The primary objective of the PDCP is to support projects in priority areas in order to address public policy issues of strategic importance to the Department. Furthermore, the Program's goal is to improve public safety and emergency management. To this effect, the Program aims to align with departmental and governmental priorities respecting public safety and emergency management since all PDCP recipients work in the field of community safety, crime prevention or emergency management. The key representatives at PS fully agree that Program recipients help achieve the Department's priority objectives and, by extension, those of the Government of Canada. Having stated this, however, priority-setting among branches has been identified as a challenge.

4.1.3 Alignment with Federal Roles and Responsibilities

The federal government spends almost $27 billion per year on grants and contributions through more than 50 departments and agencies. These transfer payments (grants and contributions combined) represent about 13% of total spending by the Government of Canada. This sum includes funds that support a range of essential community services provided by non-profit organizations, First Nations, universities, educational institutions, research centres, other levels of government, etc.Note 5 Furthermore, the majority of the respondents during the evaluation confirmed that the PDCP was aligned with PS roles and responsibilities and, by extension, with those of the federal government.

Document review alsomaintains that "Federal grant and contribution programs represent one of the most important instruments through which the Government of Canada delivers on its responsibilities to Canadians. [...] All of these programs are regarded by ministers, members of Parliament, program managers and recipients alike as vital tools for the pursuit of objectives that are clearly in the public interest."Note 6

Thus, it is clear that the administration of a contribution program, such as the PDCP, falls within the federal role.

4.2 Performance

In order to evaluate the components of the Program's performance, two issues and five sub-issues were addressed, as presented in Table 4.

Table 4 - Performance-related Issues and Sub-issues
4.2.1 Achievement of expected outcomes 4.2.1.1 Immediate outcomes
4.2.1.2 Intermediate outcomes
4.2.1.3 Ultimate outcome
4.2.2 Demonstration of efficiency and economy 4.2.2.1 Efficiency of Program management
4.2.2.2 Program economy

4.2.1 Achievement of Expected Outcomes

Inception documents and the logic model (Figure 1) provide the basis for the evaluation of the PDCP's expected outcomes. These include four immediate outcomes, three intermediate outcomes and one ultimate outcome. The main findings with respect to these outcomes are discussed in the sub-sections that follow.

4.2.1.1 Immediate outcomes

Innovative approaches, lessons learned and best practices documents

The Program respondents interviewed as part of the evaluation claim that projects funded through the PDCP have contributed significantlyNote 7 to testing innovative approaches in order to favour best practices. The 2007 Crossing Communities Art Project is one example of a significant contribution to this immediate outcome. The purpose of this project was to recruit at-risk youth (based on selection criteria defined earlier by partner organizations) to participate in a skills development activity for problem-solving using artistic and cultural expression. The project also invited young people to continue their performance by demonstrating the skills learned at community centres. On this occasion, their artistic creations were presented as part of public activities where participants were called on to attend fora in order to identify solutions for reducing violence and to consider strategic alternatives to incarceration. Shows were held in Winnipeg and in Brandon.

As a complement to the previous example, Case Study 1 highlights the contribution of the Canadian Centre on Substance Abuse to achieving this immediate outcome.

Case Study 1 - Canadian Centre on Substance Abuse

Shared knowledge and awareness and enhanced delivery of services

Generally, the Program's key representatives agree that the projects and initiatives supported by the PDCP have contributed to the acquisition and exchange of information and knowledge in PS priority areas. In several cases, these exchanges have enabled the Department to be more sensitive to realities in the field and to better understand the limitations that their recipients face. The key informants stated that the PDCP has allowed PS to develop and maintain open communications with program recipients, thereby maintaining an effective and concrete interface with the field.

The Program respondents believe that the conferences, meetings, colloquia, and research activities funded by the PDCP have contributed significantlyNote 8 to a deeper understanding and knowledge of the issues as well as greater sensitivity to matters affecting public safety and emergency management, both with respect to the public and within the Department.

However, it is important to note that, after verification, the evaluation team found that some organizations that had received funding for conferences had not published the proceedings of those conferences.

Improved inter-sectoral / inter-disciplinary collaboration and delivery of services

The PDCP's key representatives state that Program-funded projects have contributed significantlyNote 9 to strengthening relationships between PS and PDCP recipients, particularly non-profit organizations, thus fostering access to a wide range of communities as well as to a network of volunteers and local organizations of every kind.

To illustrate inter-sectoral and inter-disciplinary collaboration and its influence on the Department's service delivery, one respondent cited the Metropolis project as an example. This project, funded in part by the PDCP, involved five international research centres and numerous researchers and authors with globally recognized expertise on a range of issues that are relevant to work being done across the Department. It supported the development of a Canada-US-Mexico network of scholars looking at informal networks. Interviewees indicated that it also contributed to relationship-building and facilitated the hosting of national conferences.

Increased knowledge and understanding for decision-making in Canada's public safety and emergency management systems

Generally, the key representatives at PS stated that they regularly consult PDCP recipient organizations on subjects of interest to the Department, such as offender reintegration, the fight against drug addiction, or the organization of emergency services in the event of major disasters. The stated goal of this consultation is to deepen the knowledge and understanding of PS stakeholders on these topics. Moreover, interviewees confirm that the PDCP has enabled them to develop and maintain a permanent relationship with recipient organizations, thus facilitating their ability to obtain opinions, advice and feedback concerning the development and update of PS public policy and research projects being designed or conducted, as well as on a range of topics concerning service delivery to the public. PDCP was also considered an effective mechanism to access specialized expertise and undertake sensitive projects.

Case Study 2 is a concrete example of a project funded by the PDCP that contributes to achieving this immediate outcome.

Case Study 2 - Demos

4.2.1.2 Intermediate outcomes

Increased public education and confidence in Canada's criminal justice, emergency management, and national security systems

The respondents interviewed agreed that the PDCP encourages the conduct of public education and information activities. They also say that these activities certainly contribute to the development of greater public confidence in existing public safety systems. One respondent stated in particular that the Canadian Association of Chiefs of Police, an organization benefiting from the PDCP, implemented public awareness and education activities on public safety subjects. For example, the following activities were organized:

Case Study 3 highlights the contribution of the Canadian Association of Chiefs of Police to achieving this intermediate outcome.

Case Study 3 - Canadian Association of Chiefs of Police

Increased community capacity to work with victims, offenders, families and local officials such as emergency first responders

The Program representatives generally agree that the contributions enable funded organizations to mobilize the resources required to carry out the projects and activities that meet the Department's mandate and priorities. The respondents interviewed recount that certain recipients clearly told them that this funding allowed them to offer services, thus, increasing community capacity. Moreover, interviewees indicate that these organizations assert that this funding enables them to hold activities such as meetings and conferences, intended for exchanges and sharing on topics of interest to them and to PS.

The Program's key representatives assess that PDCP-funded projects have contributed significantlyNote 11 to building the capacity of local communities to work with victims, offenders, their families, and local representatives.

The PDCP's contribution to the Canadian Centre for Child Protection is one example of such an initiative to help victims of crime associated with the sexual exploitation of children on the Internet. The Centre, a key partner under the National Strategy for the Protection of Children from Sexual Exploitation on the Internet, manages Cybertip.ca, Canada's national tipline to report suspected cases of Internet-facilitated child sexual exploitation. In addition to this important function, the Centre has conducted programs to strengthen the personal safety of children and to protect them against the risks of sexual exploitation. To this end, interviewees indicate that the Centre has held many awareness and prevention activities for children, families, schools and local communities.

Similarly, the PDCP contribution to the Native Counselling Services of Alberta supported a project on street gangs and gang violence. This project identified solutions adapted to the community in question and led to the development of a community strategy to better respond to this phenomenon. The initiative's outcomes are detailed in the report entitled Community Solution to Gang violence,Note 12submitted to PS in August 2007.

However, in the past few years, PS representatives say that the PDCP has received few new funding proposals. Under certain circumstances, the representatives explain that it is harder to find proposals that clearly reflect the Program's spirit and intention, which would explain the repetition of certain projects within various communities.Note 13

Case Study 4 presents a concrete example of a PDCP-funded project that helps build community capacity.

Case Study 4 - St. John Ambulance

Recommendations to modify policies and legislation

The representatives interviewed say that PS has benefited significantlyNote 14 from valid, important advice and opinions from PDCP-funded organizations concerning policy development and decisions affecting programming. However, with regard to the formulation of recommendations to amend policies or laws, the Program's key representatives state that PDCP-funded projects have contributed moderatelyNote 15 to this objective. An exception was in the case of projects generated by funds earmarked for the National Strategy for the Protection of Children from Sexual to Exploitation on the Internet where Program respondents emphasized the substantial contribution of funded projects in providing opinions and advice to PS, as well as recommendations to amend policies and laws accordingly, advancing the government's agenda to combat this crime.

4.2.1.3 Ultimate outcome

A safe and resilient Canada

The terms and conditions of the PDCP specify that activities and outputs must contribute to the Program's immediate and intermediate outcomes. In this regard, it is difficult to identify a causal relationship between program activities and achievement of the ultimate outcome. However, it is suggested that, at a high level, the PDCP has contributed to related outcomes of the departmental Program Activity Architecture (PAA) such as the following outcomes that pertain to the PS branches involved in the delivery of PDCP projects:

As such, it can be argued that, in contributing to these upper-level Departmental outcomes, the PDCP has made a contribution to "A safe and resilient Canada".

4.2.2 Demonstration of Efficiency and Economy

The following section presents the main findings concerning efficiency and economy of the PDCP.

4.2.2.1 Efficiency of Program administration

Financial efficiency

In order to calculate the efficiency of program administration, a measure of annual administration costs as a percent of contribution amounts was used. It is estimated that an average annual amount of $164,400 has been expended by all branches to administer the Program, while an average of $2,000,000 per year in contribution funding has been awarded. Thus, the percentage of administration costs versus funding amount is on average 8.2% per year. This percentage appears to be reasonable; however, without comparables it cannot be stated definitively that the Program is being administered efficiently.

Dissemination and communication of program information

The PDCP terms and conditions, project eligibility and information intended for recipients are available on the PS website.Note 16 The site presents a great deal of information, such as program objectives, expected outcomes, eligible recipients, the maximum level of government assistance for eligible expenses, selection criteria, the review procedure, timing of payment, and levels of approval. The Program has defined the selection criteria for each of the three PDCP project categories. These criteria are published and clearly identified on the PS website in the section on Contribution Program Terms and Conditions.Note 17The criteria are further detailed in Annex B.

Despite the public nature of the PDCP information, some respondents indicate that the Department tends to contact only those organizations that have received contributions in the past and that little effort is actually made to inform all organizations likely to show an interest in PS fields of intervention. In this regard, however, these respondents express reservations concerning the Program's capacity to meet an increased demand for contributions, given the PDCP's financial and organizational limitations.

Several departmental representatives indicate that there is little information available on the administration and management of PDCP contribution agreements, making existing information hard to access. Several staff members indicate that they would like clearer directives, a more user-friendly financial system for monitoring contributions, and a guide to help them manage the contributions for which they are responsible. Having stated this, the Grants and Contributions Support Services Unit has offered training and has provided standard tools on-line including an agreement template, approval form and cash flow report.

In short, these findings are the same as those observed at the public hearings of the Independent Blue Ribbon Panel on Grant and Contribution Programs, which suggested, to correct the situation, organizing the information in such a manner as to be useful for both recipients and program managersNote 18 and to simplify the application process for grants and contributions in order to make it more transparent and easily accessible.Note 19

Selection of PDCP projects and funding recipients

Inception documents from 2006 provide a description the intended step-by-step process for the selection of recipients. The first step in this process is that plans were to be established and prioritized at the branch level to meet key priorities. Branches then had the responsibility of contacting potential recipient organizations to identify project areas relating to the objectives of each directorate. Once proposals were received, the approval process varied according to the amount of the proposed contribution.

Contributions of $20,000 or less were to be discretionary within the program area but were based upon a review by Corporate Management Branch/Accountability Division. The Contribution Agreements drafted from these proposals were to be routed through the Comptroller for review and recommendation. Proposals were ultimately approved within each Branch by individuals with delegated signing authority. Thus, provided the projects and activities were consistent with the Department's mandate and priorities and complied with the PDCP's terms and conditions, the Policy on Transfer Payments and other applicable policies and laws, the approval of contributions of $20,000 or less was left to the discretion of the Branch and depended on fund availability.

For general funding, contributions of more than $20,000 were to follow the same process as those of $20,000 or less with the exception that they were required to be reviewed and recommended for approval by the Contribution Review Committee (CRC). The CRC was to provide recommendations but had no authority to provide final approval. Final approval still rested with the individuals with delegated signing authority within the Branch concerned.

In terms of governance bodies, the composition of the CRC was also described in inception documents. The CRC was to be made up of representatives from branches across the department as well as a representative from the Royal Canadian Mounted Police, Correctional Services Canada and the National Parole Board (now, Parole Board Canada). The Assistant Deputy Minister or the Comptroller, Corporate Management Branch, chaired the CRC.

Inception documents also include the requirement for the CRC to provide strategic focus in ensuring that projects met criteria established and were relevant to the Portfolio's mandate and priorities, and by ensuring that a horizontal approach to funding and information exchange existed. With regards to budget reallocation, the CRC was to be the forum through which annual contribution funding was allocated, for the general funding portion. Each branch/directorate that sought access to the funds submitted an annual contribution plan for the upcoming fiscal year. Following discussions between program officials and CRC members, budgets were to be allocated on a pro-rated basis and based on a combination of need and priorities, and availability of funds.

Interviews with program representatives describe a process that delineates between the earmarked funds versus the general funds that was not outlined in the inception documents. Program representatives indicate that the CRC, in recent years, has played a different role for earmarked funds versus the general funds. For the earmarked funds, the CRC has been informed of projects but has not reviewed project funding allocations and is not involved in their selection. They indicate that the CRC had representation from the Portfolio in early years, but has not recently, given the reduced envelope of available general funding (indicating that the current level of general funding is $127,000 annually). Program representatives indicate that the CRC is involved in reviewing and allocating funding for the general funding component. Program representatives also indicate that after general funding has been allocated, the only Requests for Project

Approvals for PDCP projects that require formal review (i.e. CFO attestation) are the ones that exceed $100,000. These require Deputy Minister or Minister's Office approval as per Delegated Financial Signing Authorities.

The meetings with the Department's key respondents confirm the existence of a process for selecting and approving contributions. These persons describe biannual meetings of representatives for the directorates involved in awarding contributions. However, some respondents report uneasiness with CRC participation and functioning. They say that the CRC has had to postpone its meetings several times due to lack of attendance by persons in authority, which has held up the progress of the projects. These interviewees note a lack of communication concerning how decisions are made and would like to see more communication in this regard. Program representatives also indicate that processes of the CRC have led to unreasonable delays, resulting in the late signing of agreements.

Finally, it should be noted that the key representatives have stated that recipients regularly complain about the complexity and slowness of administrative processes, including payments, as well as about accountability. For example, one international agency that received a PDCP contribution told the departmental project manager that, of all the levels of government with which the agency has dealt for similar projects, the experience with the Government of Canada was by far the most painful.

An April 2010 decision has amalgamated funding from the two branches that have been managing earmarked funds (i.e. Community Safety and Partnerships Branch and Law Enforcement and Policing Branch); the intention is that funding will be considered on an equal basis with other projects. Key program representatives differ in opinion regarding this orientation and confusion remains regarding the implementation plan for the amalgamated fund. Some look kindly on a new distribution of all sums available, earmarked and from general funding that would allow them to include projects that could not be funded or that were likely to be refused. Others oppose it, indicating that, by appropriating funds that were previously approved by separate decisions for specific programs, the Department is diverting them for other purposes and departing from their original intent.

4.2.2.2 Program economy

Those in charge of the Program state that the contributions are an economic way for PS to obtain knowledge and collaboration in developing and updating public policies and in delivering services.

In this respect, document review indicates that "grants and contributions are a cost-effective way of utilizing the work and the skills of non-governmental actors (often volunteers at the community level) for public policy purposes."Note 20

Without further outcome-based performance information, it cannot be determined if the Program has been economic (achieved outcomes at the lowest cost). The fact that Program projects are administered by the branches themselves for all practical purposes makes it difficult to obtain program-wide financial data and corporate follow-up for its projects. The data concerning each contribution are in fact held by each branch based on specific standards. It was difficult for the evaluation team to obtain a comprehensive departmental report on PDCP contributions at the PS-wide level.

5. Conclusions

Relevance

There is a continued need for activities funded under the PDCP to assist in the development of public policy for Public Safety Canada. The PDCP supports strategic projects by departmental stakeholders that contribute to policy development and to enhanced service delivery in the areas of public safety and emergency management. There continues to be participation in the PDCP program by funded recipients that assist in filling these public policy needs. Contribution programs remain an important source of funding for recipient organizations, in general, which are highly dependent on government transfer payments.

The PDCP is aligned with government priorities and federal roles and responsibilities. The PDCP falls under the Government of Canada's roles and responsibilities as well as its priorities, as evident in various government sourcesand Speeches from the Throne. The PDCP's objectives remain relevant because they support the Department with respect to legislation, public policy development, programs and enhanced service delivery. In terms of alignment with the federal role, document review emphasizes the central importance of grants and contributions as instruments of public policy.

Performance - Effectiveness

The key program management representatives have stated that the PDCP achieved most of the expected immediate and intermediate outcomes. These respondents indicate that they regularly consult recipient organizations on numerous topics of interest to PS in order to deepen their knowledge and understanding of these issues and that the partnerships developed allow the Department to obtain relevant advice and opinions on public policy development and enhance service delivery. These key informants also stated that PDCP-funded projects have indeed helped to build the capacity of local communities to work with victims, offenders, their families, and local representatives.

Among the criticisms raised during the interviews is the increasing difficulty of selecting new proposals that clearly meet program objectives, resulting in the repetition of certain projects in various communities. There is a need for outcome based performance information.

Performance - Efficiency and Economy

In terms of financial efficiency, the average amount used to administer the fund annually equates to 8.2% of the total annual amount of the contributions.

With regard to the dissemination of program information and daily management of the Program, although the Program is promoted on the PS website, for the most part it remains the domain of organizations that already have a relationship with the Department. Some project managers believe that they would benefit from training and better coaching for contribution administration in their respective branch. The evaluation found, however, that departmental training for the Program has been provided in 2010-2011, with more in 2011-2012, to provide guidance and develop common tools in the administration of grants and contributions, to address identified needs within the organization. Some PDCP recipients are frustrated with the complexity and slowness of the Program's administrative processes.

With respect to governance, it appears that the governance mechanisms (direction-setting processes) of the Program have "eroded" from their original intent. Inception documents provided for strategic oversight regarding the management of Program funds; however, over the years, this strategic view, even to coordinate decisions among the branches, has not been as present. The evaluation team found confusion and difference of opinion concerning Program management, authorities within the Department, distribution of funding, as well as the roles and responsibilities of its various actors. For example, earmarked project funding allocations within individual branches (designated under separate government decisions) do not necessarily take advantage of the strategic perspective that was to be offered by the Contribution Review Committee. The current governance of the Committee may not be suitable given the April 2010 decision to amalgamate funding from the earmarked sources.

In terms of economy, the Program's key representatives consider the Program to be an economic means of achieving Public Safety Canada objectives, given the outcomes and the Program's low administrative cost.

6. Recommendations

It is recommended that:

  1. Clear governance and strategic oversight be established by assigning functional accountability to the Assistant Deputy Minister, Strategic Policy Branch.
  2. In light of the overarching decision to amalgamate earmarked and general funding into a single fund (in the amount of approximately $897,000), the Program's decision-making mechanisms should be redesigned to strengthen strategic management of the fund. The decision-making body, which could be a reconstituted Contribution Review Committee or a new committee, should be empowered to perform its strategic role. To this end, the committee should be of the appropriate size and composition so that there is participation at the appropriate level of the organization and so that strategic decision-making is facilitated. The project selection process and criteria should be revised in order that all projects are considered on an equal basis.
  3. An outcome-based Performance Measurement Strategy should be developed and implemented for the Program. The Performance Measurement Strategy should be aligned with current Treasury Board Secretariat guidance including the Guideline on Performance Measurement Strategy under the Policy on Transfer Payments. In this regard, outcome-based information may be required from funded recipients in order to sufficiently fulfill reporting requirements.

7. Management Response and Action plan

The Assistant Deputy Minister, Strategic Policy Branch, accepts the recommendations of this evaluation and proposes the following management action plan:

Response to Recommendation 1:

Strategic Policy Branch agrees to assume the functional accountability of the PDCP, noting the close partnership that will be required with Corporate Management Branch, which is expected to continue managing departmental transfer payment policies and processes, including for the PDCP. The target date is January 2012. This date should allow for sufficient time to prepare for the new round of PDCP proposals for the 2012-2013 fiscal year.

Response to Recommendation 2:

Strategic Policy Branch agrees that a more strategic decision-making mechanism is required, particularly with fewer resources available following Strategic Review, and will lead the redesign based on the recommendations of the 2010-2011 Evaluation of the PDCP. The target date is January 2012. Involvement of all branches will be critical to the future success of the program.

Response to Recommendation 3:

Strategic Policy Branch agrees that an outcome-based Performance Measurement Strategy should be developed and implemented for the PDCP. The target date is March 2012 to be ready for the 2012-2013 fiscal year, while also giving time for the new decision-making structure to be created, and review the Performance Measurement Strategy before implementation.

Appendix A: References

Literature and Document review

Legislation Review

Database Review

Website Review

Appendix B: Selection Criteria

Communications/Information Exchange Projects

Innovative and Research Projects

Training and Skills Development Projects

Notes

  1. 1

    Public Safety Canada.

  2. 2

    Treasury Board of Canada Secretariat Policy on Transfer Payments, http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=13525&section=text.

  3. 3

    Government of Canada. Inception document

  4. 4

    Privy Council Office. Speech from the Throne to Open the Second Session of the 39th Parliament of Canada, October 16, 2007.

  5. 5

    Ibid., p.1

  6. 6

    Independent Blue Ribbon Panel on Grant and Contribution Programs. Op. Cit., p.11.

  7. 7

    Here "significantly" refers to a rating of 4.29 out of 5.00 based on the value scale presented in Section 3.2.2.

  8. 8

    Here "significantly" refers to an average rating of 4.13 out of 5.00 based on the value scale presented in Section 3.2.2.

  9. 9

    Here "significantly" refers to an average of 4.50 out of 5.00 based on the value scale presented in Section 3.2.2

  10. 10

    Canadian Association of Chiefs of Police. http://www.cacp.ca/index/main

  11. 11

    Here "significantly" refers to an average rating of 3.20 out of 5.00 based on the value scale in Section 3.2.2

  12. 12

    Erickson, Karen, Patti LaBoucane-Benson et Jana Grekul. Community Solution to Gang Violence: A Collaborative Community Process and Evaluation Framework, 2007.

  13. 13

    Government of Canada. Inception document

  14. 14

    Here "significantly" refers to an average rating of 4.00 out of 5.00 based on the value scale in Section 3.2.2

  15. 15

    Here "moderately" refers to an average rating of 3.00 out of 5.00 based on the value scale in Section 3.2.2

  16. 18

    Independent Blue Ribbon Panel on Grant and Contribution Programs. Op. Cit., p.viii.

  17. 19

    Ibid., p.ix.

  18. 20

    Independent Blue Ribbon Panel on Grant and Contribution Programs. Op. Cit., p.11.

Date modified: