ARCHIVE - 2008-2009 Summative Evaluation of the Joint Emergency Preparedness Program (JEPP)

Adobe Acrobat version (PDF 440KB)

Executive Summary

In compliance with Treasury Board of Canada Secretariat (TBS) requirements, an evaluation of the Joint Emergency Preparedness Program (JEPP) was completed in January 2008. This report documents the evaluation.

Through the Joint Emergency Preparedness Program, the Government of Canada (GC) in consultation and cooperation with provincial and territorial governments contributes to projects that enhance Canada’s national capacity to address all types of emergencies.

Available funds over the past five years have averaged approximately $5 million annually.

Provincial and territorial governments are the targeted recipients of financial contributions provided through JEPP. A province or territory may submit a proposal whose originator is the provincial or territorial government, its agencies or a municipality. First Nations Reserves are also eligible to apply for JEPP funding through the respective province or territory. Beneficiaries are emergency management organizations and first responders as well as the Canadian public.

JEPP is administered by Disaster Financial Assistance and Preparedness Programs, Emergency Management Policy Directorate, Emergency Management and National Security (EMNS) Branch of the Department of Public Safety and Emergency Management, referenced within this document as Public Safety Canada (PS). The operational responsibility for the program lies with the Senior Assistant Deputy Minister (SADM) of the EMNS Branch.

Initial terms and conditions were approved on January 29, 1981 and amended on July 31, 1985. Current terms and conditions were approved in November 2004.

This is the second recorded evaluation of JEPP that has been completed during the life of the program. An initial evaluation was done in 2003 that considered the period from program inception in 1980 through to November 2003. This second evaluation focuses its analysis on the period starting in November 2003 through December 2007, although it may include references to the previous timeframe.

Key Findings

1. Rationale and relevance:

• JEPP continues to be consistent with government-wide priorities related to emergency preparedness.
• There is evidence that there is a need for the federal government to continue contributions to build capacity and capability for emergency management organizations at the provincial and, especially, the municipal levels.
• Emergency management officials across the country agree that emergency preparedness at the community level would be significantly and negatively impacted by the absence of JEPP.
• The program has clearly identifiable strategies that are being followed.
• Although a clearly articulated vision could not be found in JEPP documentation, there are themes that suggest, and could be used to comprise, a relevant vision.
• JEPP has three defined objectives that could be more clearly articulated to define a preferred end-state.

2. Success:

• Planned activities have substantially been implemented and are largely producing the expected outputs.
• There are improvements that could be made to enhance results.
• There are barriers that impact recipients’ and beneficiaries’ ability to participate in the program.
• There is evidence that the program has had a positive effect on ensuring emergency preparedness equipment and personnel are in place to respond to all emergencies (immediate outcome 1).
• There is evidence that emergency plans are available to facilitate emergency response (immediate outcome 2).
• There is some evidence of public awareness of emergency management (immediate outcome 3).
• Determining whether the investment in emergency preparedness capacity by GC, provinces, territories and municipalities has been “wise” is difficult to ascertain (intermediate outcome 1).
• There is evidence that active partnerships exist between levels of government and stakeholders to ensure emergency preparedness response (intermediate outcome 2).
• There is considerable evidence of significant program reach and uptake.
• JEPP provides a funding vehicle for other federal government initiatives to use to reach and affect emergency management capacity and capability at the provincial and local level.
• Recipients and end-beneficiaries have been known to limit their applications to avoid the requirement of a provincial/territorial project audit.
• Project cost-sharing is between the federal and municipal governments more so than between the federal and provincial/territorial governments even though all federal contributions flow through the provinces.

3. Cost-effectiveness and alternatives:

• JEPP has an established governance structure.
• The program has instituted controls to ensure that approved applications/proposals meet appropriate criteria, and terms and conditions are followed.
• The program uses the completed and signed federal application form as the formal funding agreement between provincial/territorial recipient and the federal government.
• There is limited evidence that a risk-based approach has been taken to the management of projects or the program itself.
• National priorities identified are not approved by senior officials and do not include provincial or territorial input.
• The JEPP program office has limited direct contact with provinces or territories, or other organizations that could contribute independent advice, recommendations or guidance.
• The provincial/territorial governments have JEPP administrative organizations that carry out tasks and activities that are often equivalent to those completed by the federal JEPP program office.
• The application and claims processes from end-beneficiary to federal government can be long, include a large number of approval points, and feature duplicative effort and activities.
• Program outcomes are not well-formulated making it difficult to align with objectives and to measure performance.
• Information and data about defined performance indicators is not collected, analyzed or used on an on-going basis.
• The program has not implemented formal processes or mechanisms for the identification, sharing and use of lessons learned across jurisdictions.
• There has been some consideration of options/alternative to increase cost-effectiveness or efficiency in delivery.
• The JEPP program appears to be efficient but, due to lack of personnel, this is accomplished at the expense of effective program monitoring and reporting, and strategic activities.

Recommendations

1. Fill open positions in the JEPP Program Office. A key priority of the Director, Disaster Financial Assistance and Preparedness Programs is to fill the open positions in the JEPP program office and obtain a commitment for their on-going funding. At the time of writing this report, the current JEPP human resources were almost exclusively utilized managing project applications and claims with little time left over for any other activities such as monitoring and improving performance, risk management, reporting, or stakeholder engagement. Until this recommendation is substantially completed, addressing other recommendations will be difficult.

2. Engage provincial, municipal and industry stakeholders. Stakeholders should be engaged for a number of reasons including:

• To ensure that the program office is fully informed about the current state of emergency management at the provincial/territorial and municipal levels and the options available to improve emergency preparedness so that it [the program office] can provide the best investment advice and make the best investment decisions;
• To share lessons learned across jurisdictions related to program planning, provisioning and delivery;
• To share the impact and identify the results that completed projects have achieved during specific emergency events including reducing injuries, loss of human life and property damage;
• To build the national emergency management community;
• To inject understanding and knowledge into the program not found in the immediate JEPP community.

Engagement should take the form of an annual conference in which stakeholders can physically meet in the same location rather than a teleconference which can limit participation and the sharing of information.

Engagement may also take the form of an advisory committee that would provide strategic advice to the program on standards, trends and future direction as well as daily operational issues such as governance and process improvements.

3. Review and update end-to-end application and claims processes. As noted in the findings, the end-to-end application and claims processes can be long and, across jurisdictions, can involve many duplicate tasks and approvals. Also, critical dates defined by the program can conflict with critical fiscal dates at the municipal level. The federal JEPP program office should take the lead in reviewing the processes, identify options for improvement (e.g. reducing duplication, increasing time to completion, mitigating municipal constraints), and facilitate implementation. Consideration should be given to:

• Differences in size and financial capabilities of provinces/territories and municipalities influencing the types of tasks that are executed at provincial/territorial or federal level;
• Differences in the complexity and risk of projects influencing the level of administrative burden as well as where the final authority resides to recommend the project or claim for approval by the PS EMNS SADM;
• Additional support (e.g. training, mentoring, support materials, etc.) enabling greater responsibility and accountability at the front end of the processes (i.e. provinces/territories, Regional Offices) and fewer burdens on the JEPP program office at the back end;
• Automation (e.g. expanding the Ontario application automation initiative);
• Modifying the application or claims cycle to mitigate municipal constraints and/or align with municipal fiscal dates (e.g. consider implementing two application cycles within the federal government fiscal year).

4. Update program design. Findings identified a number of issues related to program design suggesting a comprehensive review is required which should ultimately result in an updated program design. During this exercise, consideration should be given to:

• Defining and/or validating end-beneficiary needs;
• Defining and/or validating recipient needs;
• Creation of a program vision;
• Definition of objectives;
• Clarifying/updating activities and outputs;
• Identifying how program activities and outputs provide value to end-beneficiaries or recipients through the affect on needs;
• Defining/updating outcomes specifically identifying how the program will affect change to a target group (e.g. end-beneficiary, recipient, Canadian citizen) and its needs;
• Updating the logic model;
• Improving performance management including the identification, data collection and monitoring, and reporting on key indicators; reporting vertically on the effectiveness and efficiency of the program and horizontally on the quality of activities and outputs;
• Conducting a threat/risk assessment of the program;
• Validating governance and making explicit accountabilities, roles and responsibilities;
• Completing the JEPP Standard Operating Manual and Samples.

The updated program design should also consider the relationship of related initiatives including HUSAR and CIP, and how they should be integrated into the overall design.

The updated design should be used to inform an update of the RMAF/RBAF, the program terms and conditions, and the JEPP Manual.

5. Update terms and conditions. The program’s terms and conditions should be reviewed and updated. In particular, consideration should be given to the following.

• Prescribing the use of a formal contribution agreement defining each project. Alternatively, consideration should be given to developing a single contribution agreement with set of general terms and conditions applicable to all recipients, a set of terms and conditions that are specific to a province or territory, and a final set of conditions specific to a particular project and are appended to the agreement as an annex or appendix. Care must be taken that the contribution agreement is acceptable to recipients and does not place undo burden on the administration of the program from all perspectives (program office, recipient and end-beneficiary). A formal contribution agreement may be a requirement of the new TBS policy on transfer payments.
• Strengthening and prescribing the specific reporting requirements expected of recipient provinces and territories. Alternatively, this could be considered within the context of the contribution agreement.
• Reviewing the requirement to have a provincial auditor review projects with a federal contribution of $50,000 or more. $50,000 has been the prescribed level since 1996. It significantly impacts the administrative burden of the province/territory or municipality. In addition it impacts the program office because it must often follow up on projects that have a PAYE status because they are awaiting the completion of an audit. Consideration should be given, based upon an assessment of risk, to increasing the threshold.

6. Adjust the manner in which projects are promoted. The manner in which the program promotes its involvement in projects and why it promotes itself should be reviewed. Consideration should be given to:

• Identifying in the program design the target group(s) of the promotion (e.g. Canadian citizens, senior officials responsible for emergency management, etc.) and the result that is expected to be achieved;
• Setting a threshold on the project contribution levels that require recipients to ensure federal government involvement is effectively communicated to Canadian citizens and other stakeholders;
• Direct promotion by the JEPP program office who should implement alternate means such as an annual (or semi-annual) promotion through, for example, a media release or posting on government web-sites of major completed projects that have been funded through JEPP.

Table of Contents

• Executive Summary
• Table of Contents
• Revision History
• 1. Introduction
  • 1.1 Purpose and Scope of the Evaluation
  • 1.2 Audience
  • 1.3 Structure of Report
  • 1.4 Contact Information
• 2. Program Profile
  • 2.1 Overview
  • 2.2 Program Logic Model
  • 2.3 Governance
  • 2.4 Financial Signing Authority
  • 2.5 Program Resources
  • 2.6 Delivery
• 3. Evaluation Methodology and Constraints
  • 3.1 Evaluation Questions
  • 3.2 Lines of Inquiry
  • 3.3 Evaluation Constraints
  • 3.4 Scope Constraints
• 4. Findings
  • 4.1 Rationale and Relevance
  • 4.2 Success
  • 4.3 Cost-Effectiveness and Alternatives
• 5. Conclusions
  • 5.1 Rationale and Relevance
  • 5.2 Success
  • 5.3 Cost-Effectiveness and Alternatives
• 6. Recommendations
• Appendix ‘A’ – Documents Reviewed
• Appendix ‘B’ – Participants
• Appendix ‘C’ – Total Number of Applications
• Appendix ‘D’ – Total Number of Completed Projects (Including PAYE)
• Appendix ‘E’ – Percentage of Applications Approved (Including PAYE)
• Appendix ‘F’ – Value of National Priority Applications Completed (Including PAYE)
• Appendix ‘G’ – Total Federal Funds Contributed or Committed
• Appendix ‘H’ – Timeframe to Process Claims at the Federal Level
• Appendix ‘I’ – Program Logic Model
• Appendix ‘J’ – Observed Activities

Revision History

1. Introduction

In compliance with Treasury Board of Canada Secretariat (TBS) requirements, an evaluation of the Joint Emergency Preparedness Program (JEPP) was completed in January 2008. This report documents the evaluation.

The JEPP Summative Evaluation has been completed by Public Safety Canada (PS) and specifically the Disaster Financial Assistance and Preparedness Programs (DFAPP) Division which has responsibility for administering the program. The evaluation was done by an independent consultant under the direction of the Director, DFAPP. It was conducted with the guidance of the Guide for the Review of Evaluation Reports prepared by Treasury Board’s Centre of Excellence for Evaluation, January 2004.

This is the second recorded evaluation of JEPP that has been completed during the life of the program. An initial evaluation was done in 20031 that considered the period from program inception in 1980 through to November 2003. This second evaluation focuses its analysis on the period starting in November 2003 through December 2007, although it may include references to the previous timeframe.

1.1 Purpose and Scope of the Evaluation

The evaluation was undertaken to provide evidence-based answers to the Treasury Board evaluation policy questions related to:

1. Relevance (i.e. is the program still consistent with department and government-wide priorities, and does it realistically address an actual need);
2. Success (i.e. is the program effective in meeting objectives, within budget and without unwanted consequences);
3. Cost-effectiveness/alternatives (i.e. are the most appropriate and efficient means being used to achieve objectives relative to alternative design and delivery approaches).

The evaluation will be used as an initial step in the renewal of the program, to action improvements, and as a basis for recipient and end-beneficiary consultation.

1.2 Audience

The audience for this report includes PS and TBS program officials and executives with roles and responsibilities associated with and impacting JEPP.

1.3 Structure of Report

This report has the following sections:

   Section 1 introduces this report and highlights the purpose and scope of the
   evaluation;

   Section 2 provides a description of the program;

   Section 3 discusses evaluation methodology and constraints;

   Section 4 presents findings;

   Section 5 summarizes conclusions;

   Section 6 presents recommendations;

   Appendices provide supporting materials for the body of the document.

1.4 Contact Information

Queries related to this evaluation can be directed to:

   Dave Neville, Director

   Disaster Financial Assistance and Preparedness Programs

   Emergency Management Policy Directorate, Public Safety Canada

   Telephone: 613-990-3110

   e-Mail: dave.neville@ps-sp.gc.ca

2. Program Profile

2.1 Overview

Through the Joint Emergency Preparedness Program, the Government of Canada (GC) in consultation and cooperation with provincial and territorial governments contributes to projects that enhance Canada’s national capacity to address all types of emergencies.

Provincial and territorial governments are the targeted recipients of financial contributions provided through JEPP. A province or territory may submit a proposal whose originator is the provincial or territorial government, its agencies or a municipality. First Nations Reserves are also eligible to apply for JEPP funding through the respective province or territory. Beneficiaries are emergency management organizations and first responders as well as the Canadian public.

Available funds are divided into three major allocations.

1. JEPP Funds: The original funds provided since program inception in 1980 when JEPP was established. Available funds over the past five years have averaged approximately $5 million annually. On a yearly basis these funds are allocated based upon, but not limited to, projected provincial and territorial population, and national priorities identified annually by the JEPP program office.
2. Critical Infrastructure Protection (CIP): Commencing in 2003-2004, $400,000 per year on-going was allocated to JEPP to fund CIP projects. National critical infrastructure comprises government and privately owned or managed physical and information technology facilities, networks and assets, which if disrupted or destroyed, would impact the health, safety, security or economic well-being of Canadians or the effective functioning of governments in Canada.2
3. Heavy Urban Search and Rescue (HUSAR): Funding of $20 million over six years (2001-02 to 2006-07) with $3 million on-going was allocated in the 2001 federal budget through the Public Safety and Anti-Terrorism (PSAT) envelope to develop a national HUSAR capability for all hazards, including the consequences of terrorism.3

To receive JEPP funds, the recipient and GC enter into a cost-sharing agreement. The maximum percentage of sharing of eligible costs that PS will provide is 75 percent. Although the ratio is negotiated on an application by application basis and depends on the specifics of the project and the funds available, the usual ratio for JEPP funds is 50 percent federal and 50 percent provincial/territorial.

Other stakeholders can participate with the proviso that the combined contribution of other federal institutions to a project cannot exceed 25 percent of the non-federal share.

Eligible projects for JEPP funds are required to:

• Have a clear objective aimed at enhancing national civil preparedness for emergencies or critical infrastructure protection;
• Have an articulated statement of expected outcome(s) for the project to demonstrate, in as concrete terms as possible, how national emergency preparedness capability is expected to increase as a result of the project;
• Be based upon an agreed, identifiable beginning and end with measureable project points as appropriate;
• Include a statement on how the federal financial contribution to the project will be recognized;
• Include a provincial or territorial financial commitment to the project;
• Build on existing emergency preparedness arrangements: thus a province, territory or municipality must already have an existing emergency plan in place in order to apply for funds; otherwise the proposal must seek to develop an emergency plan.

Projects must be completed in the fiscal year in which they are approved. Projects that require more than one year to complete must re-apply in each successive year for approval with the understanding that there is no guarantee of continued funding.

Subject to the maximum annual total amount of funding approved by Cabinet, the maximum amount payable to any province or territory is $4 million in any given fiscal year. The maximum payable for an individual project is $3 million in any given fiscal year.

2.2 Program Logic Model

The program logic model is included as Appendix ‘J’.

2.3 Governance

JEPP was established in October 19805. On April 4 2005, the Department of Public Safety and Emergency Preparedness Act came into force and the Department of the Solicitor General Act was repealed.

JEPP is administered by Disaster Financial Assistance and Preparedness Programs, Emergency Management Policy Directorate, Emergency Management and National Security (EMNS) Branch of the Department of Public Safety and Emergency Management, referenced within this document as Public Safety Canada (PS). The operational responsibility for the program lies with the Senior Assistant Deputy Minister (SADM) of the EMNS Branch.

Initial terms and conditions were approved on January 29, 19818.

2.4 Financial Signing Authority

Authority to approve, sign or amend contribution agreements is delegated by the Minister designated for the purposes of the Emergency Management Act (previously the Emergency Preparedness Act) to the PS Assistant Deputy Minister (ADM), EMNS or the ADM’s successor or delegate. Similarly, signing authority to approve payments by certifying they are in accordance with the applicable contribution agreement is delegated by the Minister.

2.5 Program Resources

The program has four approved full time equivalent (FTE’s) resources: one PM-06, two PM-04 and one PM-02. The program also uses a portion of an EX-01 (Director, DFAPP) and a portion of an AS-01 (Administrative Assistant, DFAPP).

2.6 Delivery

The program delivery described in this sub-section focuses on the GC delivery of contribution funds to provincial and territorial recipients, and specifically application, claims, and information dissemination and reporting activities. Although not described herein, each province and territory interacts with end-beneficiaries through similar, often equivalent, activities.

2.6.1 Application for Contribution

Potential recipients (provinces or territories) submit proposals to the GC using a standard JEPP application form. PS Regional Offices provide support to recipients during the application process. Completed applications are submitted to PS Regional Offices for initial review, assessment and recommendation for (or against) approval. The Regional Office forwards applications with its recommendations to the Director, DFAPP at PS headquarters in Ottawa.

The Director chairs a PS JEPP Review Committee that reviews, evaluates and recommends appropriate applications. Applications competing for funds are evaluated based upon agreed-upon criteria, including annual national priorities, and ranked. The PS JEPP Review Committee recommends approval to the SADM, EMNS in order of priority until all funds have been committed. Applications that are eligible but are not approved due to lack of funds are placed on a wait list. If funds become available throughout the year because approved projects are cancelled or do not spend their funding allotment, additional applications on the list may be recommended for approval.

2.6.2 Claim for Contribution Payment

Claims for payment (i.e. reimbursement) for approved projects are submitted by recipients to the PS Regional Office on a PS JEPP claim form. Claims forms include detail and proof (e.g. receipts) of the expenditures incurred and, to receive approval, must correspond to the estimated expenditures and details of the approved application and subsequent amendments.

Claim forms require the signature of the provincial or territorial Minister responsible for emergency preparedness or a delegated official. Claim forms for projects with a GC contribution of $50,000 or more also require an auditor’s signature; otherwise forms must be signed by a designated official with appropriate fiscal accountability under the respective provincial or territorial financial administration act. The PS Regional Director (RD) signs the form to recommend payment and forwards the claims to the Director, DFAPP at PS headquarters in Ottawa. JEPP program officials review the claims, ensure they are complete and forward to the SADM EMNS or designate to approve payment to the recipient. Subsequently, payment to recipients is processed by the Financial Operations Division.

Projects and the recipients accounting records are subject to audit at the discretion of PS.

2.6.3 Information Dissemination and Reporting Requirements

Table 1 identifies critical dates in the delivery process.

Table 1: Critical Dates in the Delivery Process

National priorities, funding arrangements and other changes to the program which affect its administration are described in Annual Update Instructions (AUIs). Changes usually take effect twelve months subsequent to the publication date. AUIs which are the primary means of disseminating information to recipients have tended to be communicated in November, although it has occurred as early as September.

Information about JEPP including manual, terms and conditions, RMAF/RBAF, and forms is found online at Joint Emergency Preparedness Program (JEPP).

3. Evaluation Methodology and Constraints

This evaluation has been informed by and follows, where possible, the Evaluation Strategy outlined in the Results-Based Management and Accountability Framework (RMAF) / Risk-Based Audit Framework (RBAF) for the Joint Emergency Preparedness Program (JEPP), dated November 2004.

3.1 Evaluation Questions

Nine specific evaluation questions have been addressed that are identified in the evaluation strategy.

Rationale and relevance:

1.    Is there a clear and relevant vision and objectives for JEPP?

2.    Does JEPP continue to be consistent with government-wide priorities relating to Emergency Preparedness?

3.    Is there a continued need for JEPP?

Success:

4.    Are planned activities actually being implemented and producing the expected outputs?

5.    Have the expected results/outcomes and reach been achieved through JEPP?

6.    Have there been any unexpected outcomes?

Cost-effectiveness and alternatives:

7.    Are there appropriate management and decision-making structures in place to meet JEPP objectives?

8.    Has there been an assessment and strategic use of lessons learned?

9.    Has there been consideration of options/alternatives to increase cost-effectiveness or efficiency in delivery?

3.2 Lines of Inquiry

The following types of data and methods were used to conduct the evaluation.

• Document review: documents reviewed are identified in Appendix ‘A’.
• Interviews and consultations with stakeholders: individuals consulted are identified in Appendix ‘B’.

3.3 Evaluation Constraints

The Evaluation study encountered the following constraints.

• The RMAF/RBAF dated November 2004 was used as the basis for the evaluation. It identified the evaluation questions to be considered. It also described program objectives, activities, outputs and outcomes (immediate and intermediate) that were evaluated and reported upon. Performance indicators identified within the RMAF/RBAF were used whenever possible.
• Consultations were limited to federal and provincial government representatives. Municipalities and other end-beneficiaries were not included in the consultation process.

3.4 Scope Constraints

The following related programs or initiative were not within the scope of the evaluation.

• Disaster Financial Assistance and Preparedness Programs manages other initiatives on behalf of PS. These programs were not in the scope of this evaluation.
• The HUSAR initiative uses JEPP as a vehicle to contribute funds. This initiative was not within the scope of this evaluation; however it may be referenced to provide background, set context and formulate conclusions and recommendations.
• The CIP initiative uses JEPP as a vehicle to contribute funds. This initiative was not within the scope of this evaluation; however it may be referenced to provide background, set context and formulate conclusions and recommendations.
• This was not an evaluation of PSAT although it may be referenced to provide background, set context and formulate conclusions and recommendations.

4. Findings

4.1 Rationale and Relevance

This section addresses the following issues:

• Is there a clear and relevant vision and objectives for JEPP?
• Does JEPP continue to be consistent with government-wide priorities relating to Emergency Preparedness?
• Is there a continued need for JEPP?

4.1.1 Clear and Relevant Vision and Objectives

Although a clearly articulated vision could not be found in JEPP documentation, there are themes that suggest, and could be used to comprise, a relevant vision.

A comprehensive review of program documentation did not result in the finding of a defined vision or vision statement.

A vision can be defined as “a brief description of the ideal state. Due to its idealistic nature, it may in fact never be realized by the organization. Rather, the vision should serve to inspire the organization to move towards the ideal state.”9

A vision statement “outlines what [an organization] wants to be. It focuses on tomorrow; it is inspirational; it provides clear decision-making criteria; and it is timeless.”10

There are themes in program documentation that suggest the state that the program is working towards. These themes include:

• Enhancing national capacity to meet emergencies of all types with a relatively uniform standard of national response;11
• Reducing injuries, loss of human life and property damage costs associated with emergencies;12
• Encouraging and supporting cooperation among federal and territorial governments;13
• Envisaging a series of cooperative ventures with each party assuming its emergency responsibilities through appropriate contributions.14

JEPP has three defined objectives that could be more clearly articulated to define a preferred end-state.

The program has three stated objectives identified in the RMAF/RBAF:

• Make a significant contribution to the achievement of an appropriate increased level of national civil preparedness for emergencies and critical infrastructure protection;
• Build capacity for provincial/territorial civil preparedness and, through the provinces/territories, for community civil preparedness in priority areas;
• Lever provincial/territorial/municipal government financial support for emergency preparedness and critical infrastructure protection activities that have measurable progress points.

The objectives could be better articulated. Usually, an objective relates to a “high-level, enduring benefit towards which effort is directed”16. The three objectives are more akin to identifying a means to an end (i.e. make, build, or lever) versus an end-state itself (e.g. adequately prepared emergency management organization or first-responder that is capable and has the capacity to protect Canadians against emergencies of all types).

The program has clearly identifiable strategies that are being followed.

Examples of strategies the program has implemented to achieve objectives include:

• Cost-sharing with recipients to encourage the development of high-value projects;
• Cost reimbursement based upon a completed project or milestone, with an itemized list of reimbursable expenses that is detailed in an approved application;
• ational priorities to focus investment in high value areas that increase capacity and capability; for example, investment in communication capability that is used in all emergency situations is encouraged before investment in equipment that may only be used in rare circumstances.

4.1.2 Consistency with Government-Wide Priorities

JEPP continues to be consistent with government-wide priorities related to emergency preparedness.

Public Safety Canada “advises, supports and assists the Minister in his responsibilities as they relate to …:

• Developing policies, programs and procedures to protect Canada’s national security and capacity to prevent, prepare for, respond to and recover from natural and human-made disasters”.17

A departmental program priority is “protecting the security of Canada and Canadians”. A key initiative within this priority area is to “strengthen emergency management capacity, critical infrastructure protection, and federal emergency response capability”18. JEPP’s ultimate outcome, “national emergency preparedness capacity is enhanced to meet emergencies of all types”, directly contributes to this area.

The program also contributes to two other expected results within the department. Specifically:

• Federal Emergency Response Plan (FERP), the federal government’s all-hazards approach to emergency response which is expected to achieve “improved support to the provinces and territories” and “increased federal readiness to respond to emergencies”19, which are both areas to which JEPP indirectly contributes;
• The development and implementation of a national approach to critical infrastructure protection which is expected to achieve “enhanced coordination of critical infrastructure protection across all jurisdictions and with the private sector”20, which is also an area to which JEPP contributes.

4.1.3 Need for Contribution

There is evidence that there is a need for the federal government to continue contributions to build capacity and capability for emergency management organizations at the provincial and, especially, the municipal levels.

There is significant evidence that there is a continuing need for JEPP. Table 2 identifies on an annual basis the number of applications/proposals received and the number of projects completed.

Table 2: Project Summary by Year22

The table shows that on an annual basis, the program has been receiving in the range of 600 or more applications annually.

In the latest complete year (2006-2007), the program received 650 applications; only 10 percent were not approved based upon terms and conditions, but almost 21 percent were not approved because of insufficient funds. 318 projects were completed across the country by emergency management organizations in provinces and communities that were improving their emergency preparedness capacity or capabilities.

In the current year, another 650 applications had been received indicating there is still a continued need for the funds.

Emergency management officials across the country agree that emergency preparedness at the community level would be significantly and negatively impacted by the absence of JEPP.

In addition to the statistical evidence of a continuing need, emergency management officials at the provincial level participating in this review stated:

• Many of the JEPP eligible projects at the community level would never be done without JEPP funds;
• Many related projects that are ineligible for JEPP funding but are related to a JEPP-funded project would never be done;
• Most of the projects would not be completed as quickly;
• Emergency preparedness at the community level would be negatively impacted by the absence of JEPP, especially smaller communities across the country.

4.2 Success

This section addresses the following issues:

• Are planned activities actually being implemented and producing the expected outputs?
• Have the expected results/outcomes and reach been achieved through JEPP?
• Have there been any unexpected outcomes?

4.2.1 Activities and Expected Outputs

Table 3 is extracted from the project logic model in the RMAF/RBAF.

Table 3, Program Logic Model Activities and Outputs

Planned activities have substantially been implemented and are largely producing the expected outputs.

There is evidence that each of the planned activities has been implemented and expected outputs have substantially been provided over the past five years.

• The AUI report has been delivered for each of these years.
• National priorities include telecommunications, training and education, and exercises. Exercises replaced prototypical initiatives as a national priority in 2005-2006. Table 4 illustrates that, since 2003-2004, a relatively high percentage (78 percent) of applications for projects related to national priorities are approved. Appendix ‘C’ and ‘D’ provide additional detail. In contrast, only about 20 percent of funds are allocated to projects related to national priorities (see: Appendix ‘F’) suggesting there may be a lack of projects that align with national priorities. This may be because a saturation point has been reached or the program is not reaching or being used by end-beneficiaries that need funds for projects related to national priorities.
  
  

  2003-2004 through November 2007
  Number of Applications	1224
  Number of Approved Applications	949
  Percentage of Total Applications Approved	78%

  Table 4, Applications Related to National Priorities
  
  
• The JEPP Program Manual was available and online.
• 1811 JEPP projects were completed or are in progress (see: Appendix ‘D’).
• Twelve provincial progress reports were received and annual auditing was commenced by Audit Services Canada on behalf of the program office in 2006-2007, the second year that they were required.
• There have been instances of press releases and other media events for specific projects. Provinces and, by extension, municipalities make the attempt to acknowledge federal participation in projects however comments by participants suggest that this does not occur in every instance.

There are improvements that could be made to enhance results.

There is evidence of some concerns or areas for improvement.

• AUIs are perceived by recipients to be delivered at inconsistent times and the value was questioned by participants because of the AUIs’ generalizations and perceived lack of specifics.
• The JEPP Program Manual requires updating to ensure consistency with other official documentation. For example, the JEPP manual refers to different program objectives than the terms and conditions.
• Significant funds are lapsing on an annual basis even though the program is over-subscribed (see: Table 5). To date, over the past five years more than $3 million dollars or approximately 16 percent of the total has lapsed (note: 2007-2008 figures were not available when this report was written). Provinces and territories are expected to provide a status on projects and funds in October and identify any allocated moneys that will not be spent during the fiscal year. These funds can subsequently be re-allocated to other applications. To a large extent the provinces provide reports which reflect their understanding of projects’ status. Unfortunately, as time progresses towards the end of the government fiscal year, projects that are expected to be completed are not. A percentage of projects are protected as of April as PAYE. Subsequently, there is the risk that allocated funds will not be used, or they will simply remain in PAYE for a lengthy period of time. This may reflect a communication issue between the provincial or territorial Emergency Management Organizations (EMOs) and their municipalities. Federal program officials noted that there is no mechanism to ensure provinces or territories receive accurate information from project owners, are not mislead, and are able to present accurate program reports.
  
  

  Year	Total Funds23	Lapsed Funds24	Percentage of Total
  2003-2004	$4,843,400	$839,997	17%
  2004-2005	$5,017,438	$863,248	17%
  2005-2006	$5,112,000	$1,031,045	20%
  2006-2007	$5,071,000	$560,510	11%
  Total	$20,043,838	$3,294,800	16%

  Table 5, Applications Related to National Priorities
  
  
• Annual provincial (and territorial) progress reports are not delivered on a consistent basis in a consistent format. Progress reports have only been expected since terms and conditions were revised in 2004. AUI 11 dated November 2005 “asks” provinces and territories to provide a report that “should” be submitted by April 30th and it provides suggestions about the content but it is not prescriptive.
• There is minimal evidence that the results of provincial/territorial progress reports and audits have been used to date to inform the improvement of the program.

There are barriers that impact recipients’ and beneficiaries’ ability to participate in the program.

Participants in the evaluation identified a number of barriers experienced by recipients and beneficiaries including the following.

• Municipalities, especially smaller ones, have difficulty raising their share of project cost.
• Municipalities have a different fiscal year to the federal government. Local emergency management organizations often receive their budget in late March or early April missing the February deadline for applications to be received by the federal program office and causing the EMO to wait until the following application cycle, making the application process appear very long.
• The claims process can take a long time to complete since it has to go through three levels of government and features the federal government issuing a cheque to the provincial government which in turn issues a cheque to the municipal government. The process can be further lengthened if the federal value is over $50,000 and a provincial/territorial audit is required. Some provinces will issue a cheque to the municipality prior to receiving payment from the federal government provided the province is assured the federal government has approved payment.

The program logic model which should form the basis for the monitoring, analysis and reporting of on-going performance does not reflect all activities and outputs.

The logic model does not reflect all JEPP activities. Conceptually, JEPP has three broad areas of activities – planning, provisioning and delivery – each having a number of sub-activities with related outputs. Appendix ‘J’ provides a summary of program activities observed during the evaluation. It identifies many additional activities that are not included in the existing logic model.

4.2.2 Achievement of Outcomes and Reach

Table 7 and Table 8 are extracted from the project logic model in the RMAF/RBAF.

Table 7, Program Logic Model Immediate Outcomes

Table 8, Program Logic Model Intermediate Outcomes

There is evidence that the program has had a positive effect on ensuring emergency preparedness equipment and personnel are in place to respond to all emergencies (immediate outcome 1).

Appendix ‘D’ lists the number of completed or in progress projects by year and project type. Total projects completed that are related to emergency equipment (i.e. equipment, telecommunications, vehicle/trailer, vessel, and generators/wiring) number 915, and there may be other equipment related projects under the other project types. Provincial and territorial participants said that this equipment has been instrumental in helping first responders respond to emergency situations.

Appendix ‘D’ also identifies the total number of training projects completed or in progress: 141 in total. The majority of these projects resulted in the training of multiple emergency management personnel to better prepare and provide them with the capability to respond to emergencies.

There is evidence that emergency plans are available to facilitate emergency response (immediate outcome 2).

Appendix ‘D’ identifies over the past five years, 110 projects that have been completed or are in progress related to the development of emergency plans. In the same period, there were also 137 exercises completed or in progress. Many, if not all, of these projects would have focused on exercising emergency plans and personnel to ensure equipment and personnel are in place to respond to emergencies.

There is some evidence of public awareness of emergency management (immediate outcome 3).

Upon completion of a project, provincial/territorial and municipal officials try, in varying ways, to promote to the public JEPP projects and more particularly the involvement of the federal government. This is often done through a media release or a cheque presentation to municipal officials. The incidence, by total number of projects, of public promotion is not known. The program office reported concern that public awareness did not appear to be a high priority of recipients and end-beneficiaries, and to a degree this was confirmed by participants. The difficulty may lie in the fact that the purpose of “appropriate public awareness” is not well-defined and is stated as an outcome by itself and by itself, offers no apparent intrinsic value to recipients and end-beneficiaries. The value is derived primarily by the Canadian public that receives assurance their tax-dollars are being spent effectively and they are better protected. This value has not been reflected in the program logic model.

Determining whether the investment in emergency preparedness capacity by GC, provinces, territories and municipalities has been “wise” is difficult to ascertain (intermediate outcome 1).

There is limited data indicating how changes to emergency capacity and capability achieved through the various projects have impacted response and recovery to emergency situations. The only example provided was related to the 1998 ice storm in Ontario. The belief was that small communities in eastern Ontario responded and recovered in a more effective manner because they had emergency plans in place guiding their response. These plans were funded through JEPP and the implication is that a wise investment was made. There may be many examples of wise investment where, for example, a particular piece of equipment funded by JEPP was used to save a life or communication gear allowed a faster or more appropriate response to limit the scope of an emergency situation. Unfortunately, these examples have not been reported back to the JEPP program office and as a result data was not available during the evaluation.

There is evidence that active partnerships exist between levels of government and stakeholders to ensure emergency preparedness response (intermediate outcome 2).

Provinces and territories reported municipalities that are in close proximity with each other are partnering together to raise funds to participate in JEPP and to share in the resulting investment. JEPP has also resulted in the development of closer emergency management relationships between provinces/territories and their municipalities, evidenced by, for example, the application and proposal process at the local level. Another indicator is the amount of money the levels of governments have jointly spent on projects over the past five years, approximately $21 million (see: Appendix ‘G’) by the federal government and similar amount by the provincial/territorial and municipal governments through cost-sharing. Both provinces/territories and the PS Regional Offices report a close, cooperative and positive working relationship due in part to JEPP.

There is no partnership per se between federal and municipal governments. Working in cooperation with provincial officials, the program provides access by the federal government to municipalities. The program does allow the federal government to influence emergency preparedness at the local level and to ensure there is a level of consistency of emergency management capacity and capability across the country

There is considerable evidence of significant program reach and uptake.

All provinces and territories are participating in JEPP. Provincial and territorial participants in the evaluation reported that municipalities are aware and have been informed about the program. The larger provinces report a very high uptake among municipalities while the smaller provinces and territories report a lower uptake but still significant. Part of this lower uptake can be attributable to the small size of many municipalities in the respective province/territory and the difficulty in meeting cost sharing requirements.

4.2.3 Unexpected Outcomes

JEPP projects have resulted in the development of other projects that are not eligible for JEPP funds.

Provincial and territorial participants reported that related projects (not funded by JEPP) to those that had been funded by JEPP were initiated and completed. They stated this never would have happened without the completion of the JEPP funded project.

Recipients and end-beneficiaries have been known to limit their applications to avoid the requirement of a provincial/territorial project audit.

The claims process can be delayed substantially if the federal contribution is $50,000 or greater and a provincial/territorial audit is required. Provinces or territories have on occasion limited applications to less than $50,000 to avoid auditing. Participants suggested that the $50,000 audit threshold is too low and, considering the low-risk nature of the program, should be raised because it imposes needless burden on recipients or end-beneficiaries, and results in many projects being protected as PAYE for a significant period of time beyond the fiscal year.

Project cost-sharing is between the federal and municipal governments more so than between the federal and provincial/territorial governments even though all federal contributions flow through the provinces.

Program documentation lends itself to the perception that the cost-sharing of projects is between the federal and provincial governments (see: for example, the Introduction to the JEPP Manual). Although all JEPP contributions flow to provinces or territories, of the 318 completed projects in 2006-2007, about 12 per cent of projects were provincial/territorial and featured cost sharing between these two levels of government. About 88 percent of projects featured cost sharing between federal and municipal governments. Very few municipal projects receive any provincial funding. The role of provincial and territorial governments is heavily weighed to being administrative, rather than beneficiary.

JEPP provides a funding vehicle for other federal government initiatives to use to reach and affect emergency management capacity and capability at the provincial and local level.

JEPP has been and is being used as a vehicle to deliver other emergency management initiatives. These initiatives currently include the Critical Infrastructure Protection and the Heavy Urban Search and Rescue initiatives, and previously the Chemical, Biological, Radiological and Nuclear initiative. Without JEPP, these initiatives would have had to develop their own set of terms and conditions, management and operational vehicles and procedures. By using JEPP, these initiatives were quickly able to commence delivering services.

4.3 Cost-Effectiveness and Alternatives

This section addresses the following issues:

• Are there appropriate management and decision-making structures in place to meet JEPP objectives?
• Has there been an assessment and strategic use of lessons learned?
• Has there been consideration of options/alternatives to increase cost-effectiveness or efficiency in delivery?

4.3.1 Management and Decision-Making Structures

JEPP has an established governance structure.

The Program Manager of JEPP is a direct report of the Director, Disaster Financial Assistance and Preparedness Programs, who reports to the Director General, Emergency Management and Policy Directorate, who reports to the Senior Assistant Deputy Minister (SADM), Emergency Management and National Security Branch.

Provincial and territorial participants stated that the PS Regional Offices and the JEPP program office have been very responsive and have provided high quality service in support of the program.

Regional Directors are reports of the Director General, Coordination Directorate, who reports to the SADM. RD’s have a major responsibility within the JEPP application and claim processes. They are required to formally recommend, by signing the respective forms, applications for project funding and claims for project expense reimbursement. They are also expected to provide information and advice about the program to provincial and territorial JEPP representatives. They are not, however, vested with any accountability for the results or outcomes of JEPP, or the quality and accuracy of applications and claims. The federal program office stated that they receive a significant number of claims that are incomplete or inaccurate, requiring additional information and delaying reimbursements.

The program has instituted controls to ensure that approved applications/proposals meet appropriate criteria, and terms and conditions are followed.

On an annual basis in the third week of March, the JEPP Review Committee convenes to review applications that have been submitted at the beginning of February by provinces/territories on behalf of municipalities and provincial/territorial EMOs for earmarked and regular funds. The committee is chaired by the Director, Disaster Financial Assistance and Preparedness Programs. It is also comprised of the Ontario and Quebec Regional Directors, one or two additional RDs, representatives of the Canadian Emergency Management College who advise on training and education applications, and the JEPP Program Manager. Input is also solicited from the PS National Exercise Division, other PS organizations and federal government departments and agencies as required.

Concern was expressed by regional offices about the inclusiveness of the JEPP Review Committee since it always includes Quebec and Ontario.

Each proposal is considered against:

• JEPP terms and conditions;
• National priorities;
• Available funding.

The outcome of the JEPP Review Committee meeting is a list of recommended projects to be submitted to the SADM for approval.

Applications that are within the terms and conditions but cannot initially be approved due to insufficient funds are put on a waiting list for funding that may become available during the fiscal year. If the project is still feasible when funding becomes available, the application is recommended for approval.

The SADM directly approves all projects that feature a federal contribution of $75,000 to $250,00025. For projects less than $75,000, the DG Emergency Management and Policy Directorate has delegated authority.

Upon completion of a project, the applicable provincial/territorial government submits a claim, including required documentation, for payment to the Regional Office of Public Safety Canada. If the federal share is $50,000 or more, the claim must be certified by a designated provincial/territorial official with appropriate fiscal accountability under provincial/territorial financial administration acts. The PS RD recommends payment (or not) to the JEPP program office, who in turn, reviews the claim to ensure all conditions have been met. If so, payment is recommended to the SADM or delegate.

The program uses the completed and signed federal application form as the formal funding agreement between provincial/territorial recipient and the federal government.

The program does not create, for each approved project, a contribution agreement that is a formal written legal document setting out the obligations of the federal government and the recipient with respect to the transfer payment. Instead it considers the federal application form as a type of quasi contribution agreement. The form commits the recipient to complying with program terms and conditions and “JEPP guidelines”. The impact of the lack of a formal contribution agreement is difficult to assess given the success of the program, however, federal officials expressed concern about their lack of recourse when provinces/territories act in a way that does conflict with program terms and conditions.

The program is considering the implementation of contribution agreements. Provinces and territories expressed concern that the agreement may include a liability clause they are unable to accept because of their lack of control over projects that originate at the municipal level.

There is limited evidence that a risk-based approach has been taken to the management of projects or the program itself.

JEPP has been identified as a low-risk contribution program26. No evidence was collected during the evaluation process to suggest otherwise. However, there are still threats to program success and there is limited evidence that a risk-based management approach has been taken including the explicit identification of risk, mitigation strategies, and monitoring.

For example, the program lapses on average sixteen percent of funds on an annual basis even though the program is over-subscribed. Funds are committed to projects that are not started or completed. Provinces/territories and municipalities are allowed to retain scarce funding whether they have the actual ability to use it, to the detriment of other projects and ultimately with no consequences. This could represent a threat to current program funding levels and future potential increases.

National priorities are not approved by senior officials and do not include provincial or territorial input.

Although national priorities at one time during the program life cycle involved provinces and territories and were agreed to by Senior Officials Responsible for Emergency Management (SOREM), this is no longer the situation. National priorities are determined by the JEPP program office and rarely include the involvement of senior officials at the various levels of government, making questionable whether they are actual priorities in the broad national interest or criteria and guidance by the program office for the effective investment of limited funds.

The JEPP program office has limited direct contact with provinces or territories, or other organizations that could contribute independent advice, recommendations or guidance.

The JEPP program office is somewhat insular in the sense that its conduit for information is primarily the PS regional offices. There is limited direct contact with recipients meaning that information from these sources can be subject to a regional office filter prior to receipt. The program office has even less contact with municipal end-beneficiaries because the federal government is obligated to communicate to them through or with the cooperation of provincial or territorial governments. The program office has no direct way of communicating with the majority of end-beneficiaries to elicit input on priorities or feedback on funded projects or the program itself.

JEPP also does not have the benefit of an advisory organization that could contribute independent advice, recommendations or guidance that could, for example, help formulate annual priority investment decisions and provide input into the program renewal process.

The provincial/territorial governments have JEPP administrative organizations that carry out tasks and activities that are often equivalent to those completed by the federal JEPP program office.

Provinces and territories, especially the larger ones, have created program infrastructure that is very similar to the federal program office. They are staffed with emergency management personnel who have the responsibility to review, ensure compliance with JEPP terms and conditions, ensure completeness, and recommend applications and claims to their senior management for approval to submit to the PS regional office and ultimately to the JEPP program office. The JEPP program office processes the applications and claims and effectively duplicates many of the tasks and activities completed at the provincial level prior to submission to the SADM for approval.

Program outcomes are not well-formulated making it difficult to align with objectives and to measure performance.

Objectives should be closely aligned with outcomes. However, because objectives and program outcomes are not well-formulated the alignment and relationship is not obvious. For example, intermediate outcomes identify “wise investments” and the existence of “active partnerships”. The relationship between these statements and the objectives can be surmised, but it is open to interpretation and not clear.

Information and data about defined performance indicators is not collected, analyzed or used on an on-going basis.

Much of the data associated with performance indicators identified in the RMAF/RBAF is not collected or used on an on-going basis to manage the program or track actual versus expected results. Examples include:

• Percentage of annual funds allocated to annual JEPP priorities;
• Feedback from recipients on usefulness of JEPP guidance materials;
• Percent utilization of JEPP funds;
• Number of EP personnel trained using JEPP funds;
• Reported accomplishments and changes to emergency preparedness;
• Number of interactions between partners, through JEPP projects.

4.3.2 Use of Lessons Learned

There is minimal evidence that the program makes use of lessons learned.

Some participants suggested that they informally consult with their counterparts in other provinces or territories to learn about lessons learned in other jurisdictions. The JEPP review Committee considers issues and concerns on an annual basis in the context of evaluating projects. The JEPP program office is in the process of documenting all facets of the program which may take into account lessons learned, however it is difficult to determine because the individual who was writing the documentation has since left the program office.

The program has not implemented formal processes or mechanisms for the identification, sharing and use of lessons learned across jurisdictions.

Participants consistently suggested that a formal mechanism to collaborate about JEPP and share lessons learned between federal JEPP program office, PS Regional Offices, and provincial and territorial JEPP organizations would be valuable. Possible mechanisms include an annual conference or regular teleconferences.

There was no consensus on the involvement of municipalities or whether an invitation should be extended for their participation.

4.3.3 Increased Cost Effectiveness or Efficiency

There has been some consideration of options/alternative to increase cost-effectiveness or efficiency in delivery.

Options and alternatives are considered during the creation of the AUI. Specific projects to improve the program are also considered for funding. For example, the program office has contributed to an Ontario project the purpose of which is to automate the application process at the municipal and provincial level.

The application and claims processes from end-beneficiary to federal government can be long, include a large number of approval points, and feature duplicative effort and activities.

The application process can be long and involved, especially those which originate at the municipal level. Municipalities complete an application and proposal with supporting documentation and the necessary approval of senior municipal officials. The application is often completed using a provincial/territorial JEPP form. In the larger provinces, there may be a provincial/territorial emergency management (EM) district office that helps the municipality complete the application and subsequently receives, reviews for program eligibility criteria and completeness, and provides a recommendation. The application is then forwarded, either directly by the municipality or through the EM district office, to the emergency management organization at the provincial/territorial level. The provincial/territorial EMO receives the application, reviews for program eligibility criteria and assesses completeness. If the application is not complete, the municipality will be asked to provide further information. The province/territory will convene its JEPP review committee to assess and prioritize applications that will be forwarded to the federal government. The assessment and prioritization is usually based upon criteria suggested by the federal government in AUIs (i.e. national priorities) but provinces/territories may also set their own priorities providing they are in accordance with program terms and conditions. During the review process, the province/territorial JEPP officials usually work closely with the PS Regional Office. The provincial/territorial EMO will transcribe the provincial/territorial application to a federal application form. Federal applications that are recommended by provincial/territorial JEPP officials are referred for signing by provincial/territorial senior officials. The application forms are subsequently forwarded to the PS Regional Office. The PS Regional Office receives the application, reviews for program eligibility criteria and assesses for completeness. If the application is not complete, the province will be asked to provide further information. The PS Regional Office will then sign each application providing its recommendation that the proposal should be considered for federal funding. Subsequently, the application will be sent to the federal JEPP program office. The JEPP program office receives, reviews for program eligibility criteria and completeness. If the application is not complete, the PS Regional Office will be asked to provide further information. Although no statistics are available and even though there have been at least two reviews of the application prior to its involvement, the federal program office indicates that they still receive a substantial number of applications that are incomplete and require further information, suggesting improvements can be made in the review process at either or both the PS Regional Office or the province/territory EMO. The JEPP program office will convene the JEPP Review Committee to assess and prioritize applications. Recommended applications will be forwarded to the PS SADM for final approval. Unlike previous steps in the process, the JEPP program office is not actually required to formally sign off on the application in a similar manner as the municipalities, provinces/territories or PS Regional Offices.

The claims process is similarly involved. Although it only happens for a small percentage of projects, claims can be delayed substantially if the federal contribution is $50,000 or greater and a provincial/territorial audit is required.

There are variations to the process depending upon the province or territory. Smaller provinces or territories may not have their own form and may not prescribe a proposal format. Provinces/territories may set there own criteria for prioritizing applications during the review process. They may also set contribution limits on specific items. Claims forms are submitted by municipalities directly to the provincial/territorial EMO and not, if it exists, through a provincial EM district office.

There are no statistical figures on the end-to-end cost of applications or claims as a percentage of the federal contribution so it is difficult to assess whether the onerous applications and claims processes are appropriate cost of business for the dollar amount. However, during the four years from 2003-2004 through 2006-2007, approximately 70 percent of completed projects received a federal contribution of less than $10,000 (see: Table 9). Many projects are in the hundreds of dollars.

Table 9, Completed Projects with a Federal Contribution of less than $10,000.

The JEPP program appears to be efficient but, due to lack of personnel, this is accomplished at the expense of effective program monitoring and reporting, and strategic activities.

The JEPP program office has a head-count of four FTE’s. The program also uses a portion of an EX-01 and a portion of an AS-01. At the time this evaluation was completed, all positions were vacant with the exception of the EX-01, AS-01 and one PM-04 who, during the PM-06 vacancy, was acting as a PM-05. The result is that existing staff tend to focus on immediate, tactical, operational tasks, usually processing applications and claims. To the program office’s credit, the large numbers of applications are processed within a two month window. Claims tend to take longer to process; a sampling (see: Appendix ‘H’) suggests claims typically take three months to be processed at the federal level although some take considerably longer. As a result of resource constraints, minimal time is available to delegate to strategic activities such as planning, program review and improvement; performance and risk management; and promotion. The lack of resources also impacts the program office’s ability to advise, mentor and train Regional Offices about JEPP, and to facilitate regular collaborative events involving the program office, Regional officials, provincial EMO officials, and industry experts in which there can be a dialogue about, for example, lessons learned, program improvements, and national priorities.

5. Conclusions

5.1 Rationale and Relevance

Evidence suggests that the Joint Emergency Preparedness Program is still consistent with department and government-wide priorities related to emergency management, and it realistically addresses an actual need (i.e. the need for funds at both the provincial/territorial and municipal levels to enhance Canada’s national capacity to address all types of emergencies). Emergency management officials across the country agree that emergency preparedness at the community level would be significantly and negatively impacted by the absence of JEPP.

The clarity and design of the program could be improved. Articulating a vision and clarifying objectives to define a preferred end-state would be a first step.

5.2 Success

JEPP has achieved positive results:

• Planned activities have substantially been implemented and are largely producing the expected outputs;
• There is evidence that the program has had a positive effect on ensuring emergency preparedness equipment and personnel are in place to respond to all emergencies (immediate outcome 1);
• There is evidence that emergency plans are available to facilitate emergency response (immediate outcome 2);
• There is some evidence of public awareness of emergency management (immediate outcome 3);
• There is evidence that active partnerships exist between levels of government and stakeholders to ensure emergency preparedness response (intermediate outcome 2);
• There is considerable evidence of significant program reach and uptake;
• JEPP projects have resulted in the development of other projects that are not eligible for JEPP funds;
• JEPP provides a funding vehicle for other federal government initiatives to use to reach and affect emergency management capacity and capability at the provincial and local level.

The scope of success is difficult to determine because of a logic model that could be improved to better reflect the program and the limited availability of relevant performance data. There have also been some unexpected or negative consequences:

• There are barriers that impact recipients’ and beneficiaries’ ability to participate in the program;
• Recipients and end-beneficiaries have been known to limit their applications to avoid the requirement of a provincial/territorial project audit;
• Project cost-sharing is between the federal and municipal governments more so than between the federal and provincial/territorial governments even though all federal contributions flow through the provinces.

5.3 Cost-Effectiveness and Alternatives

There is evidence that efficient means are being used to achieve results and there are appropriate management and decision-making structures in place to meet objectives:

• JEPP has an established governance structure;
• The program has instituted controls to ensure that approved applications/proposals meet appropriate criteria, and terms and conditions are followed;
• The program uses the completed and signed federal application form as the formal funding agreement between provincial/territorial recipient and the federal government.

Management and decision-making structures could be strengthened. The lack of a contribution agreement leaves the program office with limited means to manage project risk. It also leaves the program office with minimal recourse when a project is not completed as planned. There is limited evidence that a risk-based approach has been taken to the management of projects or the program itself.

Overall, the JEPP program appears to be efficient but, due to lack of personnel, this is accomplished at the expense of effective program monitoring and reporting, and strategic activities. There is evidence that program design or delivery approaches should be updated and alternatives considered:

• National priorities are not approved by senior officials and do not include provincial or territorial input;
• The JEPP program office has limited direct contact with provinces or territories, or other organizations that could contribute independent advice, recommendations or guidance;
• The provincial/territorial governments have JEPP administrative organizations that carry out tasks and activities that are often equivalent to those completed by the federal JEPP program office;
• Program outcomes are not well-formulated making it difficult to align with objectives and to measure performance;
• There is minimal evidence that the program makes use of lessons learned;
• The program has not implemented formal processes or mechanisms for the identification, sharing and use of lessons learned across jurisdictions;
• The application and claims processes from end-beneficiary to federal government can be long, include a large number of approval points, and feature duplicative effort and activities.

6. Recommendations

Recommendation 1: Fill Open Positions in the JEPP Program Office

A key priority of the Director, Disaster Financial Assistance and Preparedness Programs is to fill the open positions in the JEPP program office and obtain a commitment for their on-going funding. At the time of writing this report, the current JEPP human resources were almost exclusively utilized managing project applications and claims with little time left over for any other activities such as monitoring and improving performance, risk management, reporting, or stakeholder engagement. Until this recommendation is substantially completed, addressing other recommendations will be difficult.

Recommendation 2: Engage Provincial, Municipal and Industry Stakeholders

• Stakeholders should be engaged for a number of reasons including:
• To ensure that the program office is fully informed about the current state of emergency management at the provincial/territorial and municipal levels and the options available to improve emergency preparedness so that it [the program office] can provide the best investment advice and make the best investment decisions;
• To share lessons learned across jurisdictions related to program planning, provisioning and delivery;
• To share the impact and identify the results that completed projects have achieved during specific emergency events including reducing injuries, loss of human life and property damage;
• To build the national emergency management community;
• To inject understanding and knowledge into the program not found in the immediate JEPP community.

Engagement should take the form of an annual conference in which stakeholders can physically meet in the same location rather than a teleconference which can limit participation and the sharing of information.

Engagement may also take the form of an advisory committee that would provide strategic advice to the program on standards, trends and future direction as well as daily operational issues such as governance and process improvements.

Recommendation 3: Review and Update End-to-End Application and Claims Processes

As noted in the findings, the end-to-end application and claims processes can be long and, across jurisdictions, can involve many duplicate tasks and approvals. Also, critical dates defined by the program can conflict with critical fiscal dates at the municipal level. The federal JEPP program office should take the lead in reviewing the processes, identify options for improvement (e.g. reducing duplication, increasing time to completion, mitigating municipal constraints), and facilitate implementation. Consideration should be given to:

• Differences in size and financial capabilities of provinces/territories and municipalities influencing the types of tasks that are executed at provincial/territorial or federal level;
• Differences in the complexity and risk of projects influencing the level of administrative burden as well as where the final authority resides to recommend the project or claim for approval by the PS EMNS SADM;
• Additional support (e.g. training, mentoring, support materials, etc.) enabling greater responsibility and accountability at the front end of the processes (i.e. provinces/territories, Regional Offices) and fewer burdens on the JEPP program office at the back end;
• Automation (e.g. expanding the Ontario application automation initiative);
• Modifying the application or claims cycle to mitigate municipal constraints and/or align with municipal fiscal dates (e.g. consider implementing two application cycles within the federal government fiscal year).

Recommendation 4: Update Program Design

Findings identified a number of issues related to program design suggesting a comprehensive review is required which should ultimately result in an updated program design. During this exercise, consideration should be given to:

• Defining and/or validating end-beneficiary needs;
• Defining and/or validating recipient needs;
• Creation of a program vision;
• Definition of objectives;
• Clarifying/updating activities and outputs;
• Identifying how program activities and outputs provide value to end-beneficiaries or recipients through the affect on needs;
• Defining/updating outcomes specifically identifying how the program will affect change to a target group (e.g. end-beneficiary, recipient, Canadian citizen) and its needs;
• Updating the logic model;
• Improving performance management including the identification, data collection and monitoring, and reporting on key indicators; reporting vertically on the effectiveness and efficiency of the program and horizontally on the quality of activities and outputs;
• Conducting a threat/risk assessment of the program;
• Validating governance and making explicit accountabilities, roles and responsibilities;
• Completing the JEPP Standard Operating Manual and Samples.

The updated program design should also consider the relationship of related initiatives including HUSAR and CIP, and how they should be integrated into the overall design.

The updated design should be used to inform an update of the RMAF/RBAF, the program terms and conditions, and the JEPP Manual.

Recommendation 5: Update Terms and Conditions

The program’s terms and conditions should be reviewed and updated. In particular, consideration should be given to the following.

• Prescribing the use of a formal contribution agreement defining each project. Alternatively, consideration should be given to developing a single contribution agreement with set of general terms and conditions applicable to all recipients, a set of terms and conditions that are specific to a province or territory, and a final set of conditions specific to a particular project and are appended to the agreement as an annex or appendix. Care must be taken that the contribution agreement is acceptable to recipients and does not place undo burden on the administration of the program from all perspectives (program office, recipient and end-beneficiary). A formal contribution agreement may be a requirement of the new TBS policy on transfer payments.
• Strengthening and prescribing the specific reporting requirements expected of recipient provinces and territories. Alternatively, this could be considered within the context of the contribution agreement.
• Reviewing the requirement to have a provincial auditor review projects with a federal contribution of $50,000 or more. $50,000 has been the prescribed level since 1996. It significantly impacts the administrative burden of the province/territory or municipality. In addition it impacts the program office because it must often follow up on projects that have a PAYE status because they are awaiting the completion of an audit. Consideration should be given, based upon an assessment of risk, to increasing the threshold.

Recommendation 6: Adjust the Manner in which Projects are Promoted

The manner in which the program promotes its involvement in projects and why it promotes itself should be reviewed. Consideration should be given to:

• Identifying in the program design the target group(s) of the promotion (e.g. Canadian citizens, senior officials responsible for emergency management, etc.) and the result that is expected to be achieved;
• Setting a threshold on the project contribution levels that require recipients to ensure federal government involvement is effectively communicated to Canadian citizens and other stakeholders;
• Direct promotion by the JEPP program office who should implement alternate means such as an annual (or semi-annual) promotion through, for example, a media release or posting on government web-sites of major completed projects that have been funded through JEPP.

Appendix ‘A’ – Documents Reviewed

Brenda Buchanan, “Joint Emergency Preparedness Program, Evaluation of the Federal Contribution Program”, Final Report, November 25, 2003

Disaster Financial Assistance and Preparedness Programs (DFAPP) Division, “Business Plan 2006-07” March 31, 2006.

Joint Emergency Preparedness Program, “Annual Update Instruction #1”, January 1995.

Joint Emergency Preparedness Program, “Annual Update Instruction #2”, November 1995.

Joint Emergency Preparedness Program, “Annual Update Instruction #3”, July 1996.

Joint Emergency Preparedness Program, “Annual Update Instruction #4”, September 1997.

Joint Emergency Preparedness Program, “Annual Update Instruction #5”, November 1998.

Joint Emergency Preparedness Program, “Annual Update Instruction #6”, December 1999.

Joint Emergency Preparedness Program, “Annual Update Instruction #7”, December 2000.

Joint Emergency Preparedness Program, “Annual Update Instruction #8”, December 2001.

Joint Emergency Preparedness Program, “Annual Update Instruction #9”, January 2004.

Joint Emergency Preparedness Program, “Annual Update Instruction #10”, November 2004.

Joint Emergency Preparedness Program, “Annual Update Instruction #11”, November 2005.

Joint Emergency Preparedness Program, “Annual Update Instruction #12”, January 2007.

Joint Emergency Preparedness Program, “Annual Update Instruction #13”, September 2007.

Joint Emergency Preparedness Program, “Joint Emergency Preparedness Program (JEPP)”.

Joint Emergency Preparedness Program, “Management Action Plan (MAP) – Joint Emergency Preparedness Program (JEPP) Evaluation”, February 2004.

Joint Emergency Preparedness Program, “Results-Based Management and Accountability Framework (RMAF)/Risk-Based Audit Framework (RBAF) for the Joint Emergency Preparedness Program (JEPP)”, November 2004.

Joint Emergency Preparedness Program, “Standard Operating Manual and Samples”, August 2, 2007.

Joint Emergency Preparedness Program, “Terms and Conditions for the Payment of Contributions under the Joint Emergency Preparedness Program (JEPP)”, November 2004.

“Memorandum of Understanding between Public Safety and Emergency Preparedness Canada and Audit Service Canada 2006-2007”, unsigned, undated.

Office of Critical Infrastructure Protection and Emergency Preparedness, “Information Note”, June 20, 2003.

Office of the Auditor General of Canada, “2004 Report of the Auditor General of Canada”, March 2004, chapter 3.

Office of the Auditor General of Canada, “2005 Report of the Auditor General of Canada”, April 2005, chapter 2.

Public Safety and Emergency Preparedness Canada, “Briefing Note for the Honourable Beverley Oda – Funding for Emergency Preparedness through the Joint Emergency Preparedness Program (JEPP)”, March 12, 2007.

Public Safety and Emergency Preparedness Canada, “Information Note”, December 2006.

Public Safety and Emergency Preparedness Canada, “Memorandum for the Minister, Joint Emergency Preparedness Program (JEPP)”, File No./TD No. 7210-1/338506”, undated.

Public Safety and Emergency Preparedness Canada, “Memorandum for the Senior Assistant Deputy Minister, Joint Emergency Preparedness Program (JEPP) Applications for 2005-2006”, File No. 7500-1/TD No. 326954”, undated.

Public Safety and Emergency Preparedness Canada, “Memorandum for the Senior Assistant Deputy Minister, Joint Emergency Preparedness Program (JEPP) Applications Over $75,000 for 2005-2006”, File No. 7500-1/TD No. 327228”, undated.

Public Safety and Emergency Preparedness Canada, Preparedness and Recovery Directorate, “Summative Evaluation for the Heavy Urban Search and Rescue (HUSAR) Initiative”, August 30, 2007.

Public Safety and Emergency Preparedness Canada, “Report on Plans and Priorities 2007 – 2008”.

Public Safety and Emergency Preparedness Canada (PSEPC), “Update on Audit Plans for FY 2005-2006 and Proposed Audit Plan for FY 2006-2007 to FY 2008-2009”, undated.

Public Safety Canada, “Briefing Note, Joint Emergency Preparedness Program”, April 16, 2007.

Public Safety Canada, Disaster Financial Assistance and Preparedness Program Division, “Joint Emergency Preparedness Program (JEPP), Contributions, A Practical Guide to Project Contribution Management”, August 2007.

Statues of Canada 2007, Chapter 15, “Bill C-12, An Act to provide for emergency management and to amend and repeal certain Acts”, 22nd June, 2007.

Appendix ‘B’ – Participants

The following individuals were consulted or interviewed in the preparation of the evaluation and provided information that has been used in this report.

Kay Agelakos, Senior Emergency Management Assistant, EMO, Province of Ontario

Robert Bégin, Regional Director, Quebec Regional Office, Emergency Management and National Security, Public Safety Canada

Nicole Bizai-Lévesque Program Officer Disaster Financial Assistance and Preparedness Programs, Public Safety Canada

Melissa Black, Coordinator, Disaster Recovery/Joint Emergency Preparedness Programs, Alberta Emergency Management Agency

Nadine Blackburn, Senior Communications Advisor, Quebec Regional Office, Emergency Management and National Security, Public Safety Canada

Wayne Brocklehurst, Regional Director, Ontario Regional Office, Emergency Management and National Security, Public Safety Canada

Cameron Buchanan, Acting Regional Director, Manitoba Regional Office, Emergency Management and National Security, Public Safety Canada

Teresa Ferguson, Coordinator, Recovery and Funding Programs, Provincial Emergency Program, Emergency Management British Columbia

Barry Folland, Program Officer, Office of Public Safety, Province of Prince Edward Island

Christine Hick, Senior Emergency Management Officer, Alberta Regional Office, Emergency Management and National Security, Public Safety Canada

Fred Hollett, Fire Commissioner/Director, Fire and Emergency Services Agency, Province of Newfoundland and Labrador

Colin King, Federal and Provincial Programs Supervisor, SaskEMO, Province of Saskatchewan

Colin Lloyd, Director of Community Programs, Alberta Emergency Management Agency,

Johanna Morrow, Manager, Recovery and Funding Programs, Provincial Emergency Program, Emergency Management British Columbia

Dave Neville, Director, Disaster Financial Assistance and Preparedness Programs, Public Safety Canada

Paul Pagotto, Manager, NCIAP, Strategy Implementation and Partnerships, Critical Infrastructure Policy, Public Safety Canada

Paul Peddle, Training Officer, Fire and Emergency Services Agency, Province of Newfoundland and Labrador

Randy Reid, Deputy Chief, Program Support, EMO, Province of Ontario

Adam Rostis, Senior Policy Advisor, Emergency Management Office, Province of Nova Scotia

Murray Sanders, Federal Programs Coordinator, SaskEMO, Province of Saskatchewan

Gord Settle, Emergency Planning Officer, Yukon Territory

Dennis Shea, Manager, Plans and Operations, Fire and Emergency Services Agency, Province of Newfoundland and Labrador

Mona Smith, Program Officer, Recovery and Funding Programs, Provincial Emergency Program, Emergency Management British Columbia

Appendix ‘C’ – Total Number of Applications29

Note: Characters in blue and bolded directly relate to national priorities for the given year.

Appendix ‘D’ – Total Number of Completed Projects
(Including PAYE)30

Note 1: Characters in blue and bolded directly relate to national priorities for the given year.
Note 2: 2007-2008 figures reflect completed, in process or approved

Appendix ‘E’ – Percentage of Applications Approved
(Including PAYE)

Note: Characters in blue and bolded directly relate to national priorities for the given year.

Appendix ‘F’ – Value of National Priority Applications Completed
(Including PAYE)31

Note 1: All figures in $000s, amounts are federal contributions, Sub-Total and Total columns exclude 2007-2008
Note 2: Regular funds include moneys that have been allocated to a project but are unused during the year either because the project has been cancelled or the completed project came in under budget. As a result, the funds are available for re-distribution by the Federal JEPP Program Office to applications that have been kept on a waiting list.

Appendix ‘G’ – Total Federal Funds Contributed or Committed32

Note 1: All figures in $000s, rounding may introduce small variances with actual figures.

Appendix ‘H’ – Timeframe to Process Claims at the Federal Level33

Appendix ‘I’ – Program Logic Model34

Appendix ‘J’ – Observed Activities

1 “Joint Emergency Preparedness Program, Evaluation of the Federal Contribution Program, Final Report”, Brenda Buchanan, November 25, 2003.

2 The CIP Initiative is not within the scope of this evaluation.

3 The HUSAR Initiative is not in the scope of this evaluation.

4 RD 418-80RD (c).

5 P.C. 2003-2086 issued under section 3 of the Public Service Rearrangement and Transfer of Duties Act, R.S.C. 1985, c. p-34.

6 TB Number 775116.

7 TB Number 799188.

8 TB Number 831600

9 http://www.gov.sk.ca/finance/accountability/2006/keyterms.htm

10 http://strategis.ic.gc.ca/epic/site/stco-levc.nsf/en/h_qw00037e.html

11 “Results –Based Management and Accountability Framework (RMAF) / Risk-Based Audit Framework (RBAF) for the Joint Emergency Preparedness Program (JEPP)”, November 2004, section 1.

12 Ibid.

13 JEPP Manual, modified: 2007-9-18.

14 Ibid.

http://www.tbs-sct.gc.ca/eval/pubs/rmaf-cgrr/rmafcgrr05_e.asp

http://www.gov.sk.ca/finance/accountability/2006/keyterms.htm

17 “Report on Plans and Priorities 2007-2008”, Public Safety and Emergency Preparedness Canada, p.11.

18 Ibid, p.15.

19 Ibid, p.24.

20 Ibid, p.25.

21 Year to date (November 2007). Some figures have not been included because they would be misleading or are only relevant once the year is complete.

22 Source: JEPP Program database.

23 Source: Annual Update Instructions for the respective year.

24 Source: JEPP Program Database

25 The Deputy Minister has signing authority up to $500,000, The Minister signs for amounts in excess of $500,000.

26 “Results –Based Management and Accountability Framework (RMAF) / Risk-Based Audit Framework (RBAF) for the Joint Emergency Preparedness Program (JEPP)”, November 2004, Preamble.

27 Source: JEPP Program Database.

28 Ibid.

29 Source: JEPP Program database

30 Source: JEPP Program database

31 Source: JEPP Program database

32 Source: JEPP Program database

33 Source: JEPP Program project files.

34 Source: “Results –Based Management and Accountability Framework (RMAF) / Risk-Based Audit Framework (RBAF) for the Joint Emergency Preparedness Program (JEPP)”, November 2004, section 2.

Date modified:

2015-12-15