2011-2012 Evaluation of the International Association of Fire Fighters/Canada Contribution Program - Final Report

2011-2012 Evaluation of the International Association of Fire Fighters/Canada Contribution Program - Final Report PDF Version (432 KB)

Table of contents

• Executive Summary
• 1. Introduction
• 2. Profile
  • 2.1 Background
  • 2.2 Roles and Responsibilities
  • 2.3 Resources
  • 2.4 Logic Model
• 3. About the Evaluation
  • 3.1 Objective
  • 3.2 Scope
  • 3.3 Methodology
  • 3.4 Limitations
  • 3.5 Protocols
• 4. Findings
  • 4.1 Relevance
  • 4.2 Performance--Effectiveness
  • 4.3 Performance--Efficiency and Economy
• 5. Conclusions
  • 5.1 Relevance
  • 5.2 Performance
• 6. Recommendations
• 7. Management Response and Action Plan
• Annex A: Documents Reviewed
• Annex B: Training organizations
• Annex C: Financial Analysis

Executive Summary

This is the 2011-2012 Evaluation of the International Association of Fire Fighters/Canada Contribution Program. Evaluation supports accountability to Parliament and Canadians by helping the Government of Canada to credibly report on the results achieved with resources invested in programs. Evaluation supports deputy heads in managing for results by informing them about whether their programs are producing the outcomes that they were designed to achieve, at an affordable cost; and supports policy and program improvements by helping to identify lessons learned and best practices.

What we examined

Starting on April 1, 2008, Public Safety Canada entered into a contribution agreement with the International Association of Fire Fighters/Canada to deliver a national hazardous materials training program to first responders. The amount of the contribution agreement was $500,000 per year for five years.

This evaluation was conducted in accordance with the requirement that an evaluation be completed prior to expiration of the five-year contribution agreement on March 31, 2013. The scope of the evaluation includes Public Safety Canada's activities in administering the contribution program.

Why it is important

Professional fire fighters are often first to arrive on the scene of emergencies, including incidents involving hazardous materials. Budget 2007 provided support to implement a hazardous materials training program that would be available to all first responders including fire fighters, police, paramedics and utility workers nation-wide. Interoperability among first responders is important to the safety of Canadians during an event involving hazardous materials.

Under the Emergency Management Act, the Minister of Public Safety is responsible for exercising leadership related to emergency management in Canada. The Minister is also responsible for coordinating, among government institutions and in cooperation with the provinces and other entities, emergency management activities, including training related to emergency management.

What we found

The International Association of Fire Fighters/Canada Contribution Program was originally intended to address a gap in training among fire fighters and contribute to interoperability among first responders. Because training data is not collected nationally it was difficult for the evaluation to determine if these training gaps persist. However, based on available evidence, there is still a need for hazardous materials training for first responders. Systematic collection of data related to hazardous materials training across the country would provide a more accurate understanding of the current level of first responder training and the remaining training gaps.

Federal support for first responder training was evident in Budget 2007 and the years following the development of the first Public Safety Canada strategy related to chemical, biological, radiological, nuclear, or explosive events^{Note 1}. Emergency management training is not part of the stated 2012-2013 Departmental Key Priorities; however it remains a Sub-Sub-Activity in the Program Activity Architecture under 1.4.1.2 Emergency Management Training and Exercises.

Legislation and policy frameworks show that it is the federal government's role, in collaboration with the provinces/territories, to facilitate national coordination and advancement of emergency management training. The International Association of Fire Fighters/Canada Contribution Program appears to be disconnected from the leadership and collaborative role of Public Safety Canada in relation to the provinces/territories. Missed opportunities for synergy and strategic linkages may be facilitated through the Federal/Provincial/Territorial Forum for Senior Officials Responsible for Emergency Management. In addition, the current fiscal and organizational context suggests a change in focus of Public Safety Canada training efforts and movement toward a more strategic role. Alternate ways of supporting emergency management training are being examined, and a new business model is evolving. Public Safety Canada will broaden its partnership base to support and strengthen emergency management training through the development and promotion of national emergency management training standards in consultation with provincial/territorial partners, academia, and emergency management practitioners. In addition, Public Safety Canada will provide emergency management training to federal public servants through a memorandum of understanding with the Canada School of Public Service. Under this memorandum of understanding, Public Safety Canada will provide policy and program guidance.

In terms of duplication, hazardous materials training is available in most provinces through provincial emergency management organizations, colleges and universities, and the private sector; however, the key difference is that this Program delivers training on-site, making it advantageous to small communities that cannot afford the risk of having a portion of their workforce away concurrently.

The funding provided by Public Safety Canada increased the International Association of Fire Fighters/Canada's capacity to develop and deliver the training program. The Program developed bilingual course material, trained qualified instructors and, from April 2009 to December 2011, trained a total of 1312 participants.

In the provinces where training was delivered, there has been increased access for fire fighters in small/isolated communities. The Program has not significantly increased access to hazardous materials training on a national level since training has been mainly concentrated in two provinces. The fact that the International Association of Fire Fighters/Canada courses are not yet accredited has been a significant obstacle to provide training in many provinces. However, the International Association of Fire Fighters/Canada continues to promote the program and has applied for accreditation.

Evidence shows that the training program is of good quality and that it has increased the knowledge and confidence of participants in their ability to manage hazardous materials incidents. The training has improved interoperability in those communities where emergency response personnel, other than fire fighters, also attended the course. There is little evidence to demonstrate the wider impact of the Program in terms of protecting the safety of fire fighters and the general public; however, the training was well received in the communities where it took place, and several examples provided by interviewees indicate that the training had a positive effect on community safety by increasing preparedness.

With some exceptions, the Program is being administered efficiently to produce outputs and progress towards expected outcomes. The Program is considered efficient since the program administration ratio compares favourably to other Public Safety Canada transfer payment programs. However, there are lapses in Program spending; more accurate recipient estimates and closer attention to year-end consolidation of expenditures may improve this result. In terms of Program economy, the cost per training participant is within range of two other similar training offerings that were compared during the evaluation.

Recommendations

The Evaluation Directorate recommends that the Assistant Deputy Minister, Emergency Management and Regional Operations Branch:

1. Develop program alternatives that better reflect Public Safety Canada's leadership and collaboration role.

   Program alternatives should consider:

   • need/gaps in hazardous materials training in provinces/territories, through consultation with the Federal/Provincial/Territorial Forum of Senior Officials Responsible for Emergency Management); and
   • Public Safety Canada's evolving business model in the area of emergency management training.
2. Develop a mechanism that will promote timely updates of recipient estimates and expenditures in order to provide a more accurate financial representation of the program.

1. Introduction

This is the Public Safety Canada (PS) 2011-2012 Evaluation of the International Association of Fire Fighters (IAFF)/ Canada Contribution Program. This evaluation provides Canadians, parliamentarians, Ministers, central agencies, and the Deputy Minister of Public Safety with an evidence-based, neutral assessment of the relevance and performance (effectiveness, efficiency and economy) of this federal government program.

This evaluation was conducted in accordance with the requirement that an evaluation be completed prior to expiration of the five-year contribution agreement on March 31, 2013. The scope of the evaluation includes PS's activities in administering the contribution program.

2. Profile

2.1 Background

Professional fire fighters are often first to arrive on the scene of emergencies, including incidents involving hazardous materials (HazMat). The IAFF/Canada is an organization that represents over 21,700 professional fire fighters in Canada.^{Note 2} The organization provides assistance in collective bargaining, health and safety matters, public relations, crisis management, legislative initiatives and other regulatory issues.^{Note 3} A survey conducted by IAFF/Canada in 2005 indicated that many of their affiliates lacked the ability to respond safely and effectively to basic HazMat incidents.

Budget 2007 provided support to implement a HazMat training program that would be available to all first responders including fire fighters, police, paramedics and utility workers nation-wide. As a result, a contribution agreement was signed with IAFF/Canada for $2.5 million to implement the HazMat training program from 2008-2009 to 2012-2013.

HazMat training offered in Canada and in the United States is based on the same standard from the National Fire Protection Association.^{Note 4} Since 2001, the IAFF/Canada advocated for this program because the course material was already in place in the United States and could be adapted to the Canadian context. Because the program could be delivered via a train-the-trainer approach, the effects of training would be multiplied as participants continued to deliver the course to first responders in their home community.^{Note 5} The training would therefore enhance the number of front-line first responders prepared to respond to a HazMat incident.^{Note 6}

The 2011 Chemical, Biological, Radiological, Nuclear, or Explosive (CBRNE) Resilience Strategy for Canada recognizes the relationship between a HazMat and a CBRNE response. CBRNE events involve potential, perceived, or actual acts with chemical, biological, radiological, nuclear, or explosive materials that are, or are suspected to be, used in a deliberate or intentional way to cause harm. These acts are considered criminal acts and are often acts of terrorism. HazMat events are caused by accidents, negligence or a natural occurrence (e.g. spills, violent weather). These events require a very precise and specialized response. In most instances, in the absence of clear evidence to suggest an intentional event, the response to a CBRNE event starts as a HazMat response. HazMat is a core capability that is required to respond to a CBRNE event.^{Note 7}

The objective of the HazMat training program, delivered under the agreement, is to train first responders about strategies to safeguard their health and safety and the safety of Canadians; and to reduce deaths and injuries related to HazMat incidents. The goal is to provide the highest level of instruction and training by:

• conducting local, customized training that meets the needs of first responders, local jurisdictions and PS;
• collaborating to reach as many first responder populations as possible; and
• ensuring that all training programs continue to meet or exceed applicable professional standards; and that they incorporate lessons learned and best practices.

2.2 Roles and Responsibilities

PS administers the IAFF/Canada contribution agreement through the Emergency Management and Regional Operations Branch. The contribution agreement was managed by the Canadian Emergency Management College from 2008-2009 to 2011-2012. As of April 2012, PS is re-examining the organizational placement of the Program given that the College's activities are being reorganized.

Under the Program, the IAFF/Canada has responsibility for the following activities:

• adapt the training program to the Canadian context;
• ensure adherence to the training standards;
• train instructors;
• promote the program;
• deliver the training across Canada; and
• update the course as required.

2.3 Resources

Table 1 presents the funding profile^{Note 8} for the IAFF/Canada Contribution Program for the first five years.

2.4 Logic Model

Figure 1 - Logic Model of the IAFF/Canada Contribution Program

The logic model presented in Figure 1 is a visual representation that links what the IAFF/Canada Contribution Program is funded to do (activities) with what the program produces (outputs) and what the program intends to achieve (outcomes). It also provides the basis for developing the evaluation matrix, which gave the evaluation team a roadmap for conducting this evaluation.

 PS conducts the following enabling activities:

• Develop Terms and Conditions
• Negotiate, administer and monitor the contribution agreement
• Participate in the Advisory Committee

These activities result in an increased IAFF Canada capacity to deliver HazMat training (Outcome A).

These PS activities and outcome described above contribute to the following IAFF activities and outputs.

1) Training Program Development

• Produce and distribute student work books and other course materials
• Adapt existing course materials for Canadian delivery
• Train the trainers and instructors, and provide ongoing support

This activity leads to the following outputs:

• HazMat course material with Canadian content, in both official languages
• Course material distributed
• Trained instructors

2) Communication

• Market the program

This activity leads to the following output:

• Promotional material

3) Course delivery

• Make logistical arrangements to provide training
• Deliver the courses

This activity leads to the following output:

• Courses delivered

4) Liaison/Collaboration

• Liaise with the Canadian Emergency Management College and Provincial Emergency Management Offices officials, as appropriate, to develop, market and deliver courses

This activity contributes to the achievement of all the IAFF outputs.

These activities and outputs contribute to the achievement of the following immediate, intermediate, and ultimate outcomes:

• Nation-wide access to standardized training for first responders (Immediate outcome B)
• Canadian fire fighters and other first responders have the skills and knowledge to respond to HazMat incidents, that is, improved interoperability (Intermediate outcome C)
• Canadians are protected from the aftermath of HazMat incidents (Ultimate outcome D)
• Canadian fire fighters and other first responders safely respond to HazMat incidents (Ultimate outcome E)

These IAFF outcomes contribute to PS's strategic outcome of building a safe and resilient Canada.

3. About The Evaluation

3.1 Objective

This evaluation supports:

• accountability to Parliament and Canadians by helping the Government to credibly report on the results achieved with resources invested in this program;
• the Deputy Minister of Public Safety in managing for results by informing him about whether this program is producing the outcomes that it was designed to produce, at an affordable cost; and,
• policy and program improvements.

3.2 Scope

The evaluation assessed the relevance and the performance of the Program beginning on April 1, 2008. The scope of the evaluation included the administration and oversight of the contribution agreement by PS.

3.3 Methodology

This evaluation was conducted in accordance with the Treasury Board of Canada Secretariat Policy on Evaluation, the Standard on Evaluation for the Government of Canada and the PS Evaluation Policy. Evaluators took into account the following factors in order to determine the evaluation effort, including the approach, scope, design, and methods, required for this evaluation:

• risks;
• quality of past evaluations;
• soundness of program theory;
• longevity of the program; and
• contextual stability.

Specifically, the evaluation methodology and associated level of effort were calibrated taking into consideration the characteristics of the IAFF/Canada Contribution Program.

1. The Program is of low materiality and dollar value ($500,000 annually).
2. The Program is low risk. The contribution program provides funding to one recipient, the IAFF/Canada, who receives the same amount of funding on an annual basis.
3. As final outcomes of the Program are mostly attributable to IAFF/Canada, less focus was placed on achievement of these outcomes and more focus was placed on PS "enabling" activities and immediate outcomes (capacity building and access to the training).
4. The IAFF/Canada provided course evaluation information; therefore, a small sample of course participants was interviewed.

3.3.1 Evaluation Core Issues and Questions

As required by the Directive on the Evaluation Function, the following issue areas and evaluation questions were addressed in the evaluation:

Relevance

• 1. What need was the IAFF/Canada Contribution Program intended to address? Does the need persist?
• 2. To what extent is the IAFF/Canada Contribution Program aligned with government-wide priorities and supportive of departmental strategic outcomes?
• 3.
  1. To what extent is the IAFF/Canada Contribution Program aligned with federal roles and responsibilities?
  2. To what extent does the program duplicate or complement other similar programs?

Performance - Effectiveness

• 4. To what extent have outcomes been achieved?
  1. To what extent has PS contributed to increased IAFF/Canada capacity to deliver HazMat training?
  2. To what extent do first responders have access to standardized training Nation-wide?
  3. To what extent do Canadian fire fighters and other first responders have the skills and knowledge to respond to HazMat incidents?
  4. To what extent do Canadian fire fighters and other first responders safely respond to HazMat incidents?
  5. To what extent are Canadians protected from the aftermath of HazMat incidents?

Performance - Efficiency and Economy

• 5. Is the IAFF/Canada Contribution Program being administered efficiently to produce outputs and progress towards expected outcomes?

3.3.2 Lines of Evidence

The evaluation team used the following lines of evidence to assess the Program: document review, interviews, and a review of performance and financial data. Each of these methods is described in more detail below.

Document Review

The document review included the following types of documents: corporate documents, accountability and policy documents, program inception and renewal documents, reports on plans and priorities, performance reports, speeches from the Throne and budget documents, legislative documents and program-specific documents, including a Management Review conducted in 2010-2011. A list of documents reviewed is presented at Annex A.

Interviews

Twelve interviews were conducted using interview guides developed for each of the interview groups described in Table 2. The choice of interviews of training participants was based on the provinces where the training was delivered to ensure geographic coverage with a focus on areas where the most training was delivered.

Financial and Quantitative Data Review

An analysis of financial and quantitative data included Program performance documents such as progress reports and course evaluation data. Budget and expenditure data related to the contribution agreement were also reviewed. In addition, for the assessment of program economy, a limited comparative analysis was conducted using web-based, publicly available information from two other training organizations.

3.4 Limitations

• A limited number of course participants were interviewed to gain understanding of the impact of the HazMat training program. Results are not statistically significant due to the small sample size. Where possible, the evaluation addressed this limitation by supplementing interview information with quantitative information and document review. Fire fighters from provinces that did not access the IAFF/Canada Contribution Program were not interviewed.
• No interviews were conducted with provinces/territories or with the private sector (teaching establishments/providers) on provincial course offerings. The evaluation addressed this limitation by including interview questions related to other course offerings and relying on information presented on provincial websites.
• Expenditures by PS program staff were based on the level of effort estimated by staff since this information was not tracked under individual cost centers or as internal orders.
• Besides the baseline study conducted by the IAFF in 2005, no other quantitative information was available regarding the gap in the level of training on a national basis.

3.5 Protocols

During the conduct of the evaluation, PS program representatives assisted in the identification of key stakeholders and provided documentation and data to support the evaluation. Collaborative participation greatly enriched the evaluation process.

This report was submitted to Program representatives and to the responsible Assistant Deputy Minister for review and acceptance. A management response and action plan was prepared in response to the evaluation recommendations. These documents were presented to the PS Departmental Evaluation Committee for consideration and for final approval by the Deputy Minister of Public Safety.

4. Findings

4.1 Relevance

4.1.1 Continuing Need for the Program

In order to assess whether there is a continuing need for the IAFF/Canada Contribution Program, the evaluation examined the rationale for the creation of the Program and the continuing need in terms of gaps in training among first responders^{Note 9}.

In 2002-2003, the federal government initiated a broad Public Safety and Anti-Terrorism program that included targeted emergency response training of first responders. After training thousands through this program, there remained a large number of fire fighters and other first responders that still needed awareness and operational HazMat training.^{Note 10}

In 2005, IAFF/Canada surveyed individual IAFF local associations across Canada regarding their preparedness to respond to Weapons of Mass Destruction and CBRN^{Note 11} events. Of the local associations who responded^{Note 12}, 19% had the equipment and training to respond safely and effectively to CBRN incidents in their city. Furthermore, 75% indicated that they had little or no training and indicated they were not prepared to respond safely and effectively to a CBRNE incident. The 2008 Report by the Standing Senate Committee on National Security and Defense on Emergency Preparedness in Canada stated that 50% of municipalities reported that they did not have the capacity to manage a CBRN incident.^{Note 13} In 2008, the IAFF/Canada Contribution Program was launched to help fill these training gaps.

There are no recent studies assessing the continued need for HazMat training for fire fighters in Canada. The IAFF/Canada reports that they have trained only 5% of fire fighters since the teaching component of the program began in 2009. It is currently unknown what the total percentage of trained fire fighters is because there is no national system to collect this type of data. However, interviewees report that there is still a need for HazMat training, particularity in small communities through which chemicals are frequently transported by train and transport trucks. Fire fighters from these communities are first on the scene when an incident occurs, while military-based teams with specialized skills to handle these incidents can take hours to arrive.^{Note 14} There is also a need in the volunteer fire fighter community as they typically do not have access to the same HazMat training as professional fire fighters.

In terms of continuing gaps related to the IAFF/Canada HazMat training, only two provinces (Ontario and British Columbia) have had full access to the training. Some provinces have had limited training, while others have not yet accessed it. PS program representatives also report that gaps remain in first responder training such as the lack of Canadian training standards, training knowledge networking and outreach, and professional development.

On a larger scale, the consequences of natural and human-induced hazards and disasters have become more widespread due to increasingly urbanized and globally interconnected societies. The threat and risk posed by CBRNE events is a global challenge.^{Note 15}

4.1.2 Alignment with Federal and Departmental Priorities

The evaluation sought to assess the degree of alignment of the Program with federal government and departmental outcomes and priorities.

Federal support for this program was evident in Budget 2007 when the Minister of Finance stated that the "...Government is helping to ensure that fire fighters and other first responders have the training they need to safety and effectively respond to hazardous material emergencies."^{Note 16} In 2009, the Minister of Public Safety declared that this funding would "significantly increase the number of Canadian first responders who have the skills and knowledge necessary to respond to a HazMat incident. Our ongoing commitment to enhancing emergency response will better protect all Canadians."^{Note 17} In recent budgets and speeches from the Throne, the safety and security of Canadians remains a central theme.

Recent initiatives highlight the need to be ready to respond to HazMat and CBRNE incidents, such as the Government of Canada's Counter Terrorism Strategy (2011)^{Note 18}, which sets out Canada's integrated approach to dealing with terrorist threats, and the United States-Canada Joint Border Threat and Risk Assessment (2010)^{Note 19}, which identifies CBRNE as one of the significant threats along the United States/Canada border.

Within PS, the importance of emergency management training of first responders was highlighted in the Department's 2010-2011 Report on Plans and Priorities as a key performance measure for preparedness. More recent departmental reports give prominence to national security issues, emergency management response and mitigation. Emergency management training is not one of the stated 2012-2013 Departmental Key Priorities; however it remains a Sub-Sub-Activity in the Program Activity Architecture under 1.4.1.2 Emergency Management Training and Exercises.

The emergency management training landscape has changed in Canada in the last several years. Traditionally, the Canadian Emergency Management College was the Government of Canada's focal point for cross-disciplinary emergency management training and learning. Its mission was to enhance the capability of Canada's emergency management community through educational means so that the impact of disasters on the Canadian public is minimized. One of the priorities of the College was to work to support and augment programs with complementary courses and services, in close collaboration with provincial/territorial emergency management training programs.^{Note 20}

Current fiscal realities have meant a change in focus of PS's training efforts. As a result of renewal efforts, which started in 2010, and following the announcements of Budget 2012, PS deemed that some programs offered by the College were outside the core business of the federal government and were no longer sustainable. Thus, the federal learning facility and others across the country will no longer provide course delivery. Alternate ways of supporting emergency management training are being are being examined and a new business model is evolving. Public Safety Canada will broaden its partnership base to support and strengthen emergency management training through the development and promotion of national emergency management training standards in consultation with provincial/territorial partners, academia, and emergency management practitioners. In addition, Public Safety Canada will provide emergency management training to federal public servants through a memorandum of understanding with the Canada School of Public Service. Under this memorandum of understanding, Public Safety Canada will provide policy and program guidance.

4.1.3 Alignment with Federal Roles and Responsibilities

The evaluation examined accountability documents to determine Program alignment with federal roles and responsibilities. Interviewee perceptions, supplemented with document review, were used to assess possible duplication and synergy.

The Minister of Public Safety is responsible, under the Emergency Management Act, for exercising leadership relating to emergency management in Canada by coordinating, among government institutions and in cooperation with the provinces and other entities, emergency management activities, which includes [...] providing education and training related to emergency management.^{Note 21}

The Strategy for Emergency Management Training in Canada 2010-2015 states that provinces have jurisdictional responsibility for emergency management training and consequently exercise control over their training initiatives. In collaboration with the provinces and territories, the federal government also facilitates national coordination and advancement of emergency management training and supports provincial and territorial training efforts through the sharing of information, materials and other resources.^{Note 22}

The Program appears to be disconnected from the leadership and collaborative role of PS in relation to the provinces/territories. The evaluation notes that there may be missed opportunities for synergy and strategic linkages between the IAFF/Canada training and other training offerings across Canada. Given that HazMat training is closely linked to CBRNE, it appears that the Federal/Provincial/Territorial Forum of Senior Officials Responsible for Emergency Management--more specifically its National Emergency Management Training Committee and/or CBRNE Sub-working Groups--could be an appropriate venue for discussions. Issues that could potentially be addressed include: duplication and synergy; provincial/territorial intent with respect to HazMat training; and course accreditation. The CBRNE Sub-working Group is responsible to consult such groups as fire services, emergency medical services, military, intelligence and law enforcement.^{Note 23} The Sub-working Group contributed to the 2011 CBRNE Action Plan for Canada. One of the objectives of this action plan is to build an effective and interoperable workforce through the review of CBRNE training and the establishment of training curriculum and standards in the next one to three years.

In 2012, emergency management training is widely available through provincial/territorial governments, some municipal governments, community colleges, universities, and private sector organizations. Most provinces in Canada have an accredited organization that offers HazMat training to first responders. (Refer to Annex B for a list of accredited training organizations). Interviewees stated that they are aware that HazMat training is available at some community colleges, universities and in the private sector. Some interviewees mentioned that they had taken HazMat training previously and the IAFF/Canada training was a good "refresher". However, others mentioned that were it not for the IAFF/Canada course, they would not have received HazMat training. They noted a key difference in that the IAFF/Canada program is delivered on-site in large and small communities. In addition to fiscal constraints, for small communities, this is a crucial difference; they cannot afford to have a portion of their small workforce away concurrently, leaving limited capacity to respond to an emergency.

4.2 Performance--Effectiveness

Performance was assessed using data collected by the IAFF/Canada (i.e. course evaluation sheets, testimonials) and annual performance reports provided by IAFF/Canada to PS. Performance information was also gathered through interviews.

4.2.1 Capacity to Deliver the Training

IAFF/Canada was allocated $500,000 annually from PS to implement the HazMat training program. The IAFF/Canada does not receive other funding to deliver the training.

The contribution agreement was meant to increase capacity in the following categories^{Note 24}:

• production, distribution and modification of course materials in both official languages;
• logistical arrangements for the provision of training ;
• delivery of courses in both official languages;
• liaison with federal and provincial points of contact for course delivery; and
• selection, coordination and management of consultants and subject matter experts for training.

In the first year of the program, 2008-2009, the funding was received late in the fiscal year. Despite this, the funding enabled the IAFF/Canada to customize the training program to meet the needs of first responders in Canada, and select and train 10 Canadian Master Instructors. Administrative and logistical policies and processes were also developed during this time.

The funding enabled the IAFF/Canada to make logistical arrangements to deliver the training in multiple provinces. In 2009-2010, the IAFF/Canada started to deliver the training. The course consists of three, eight-hour sessions, totalling 24 hours of instruction. As of December 2011, IAFF/Canada had held 73 training sessions and trained 1,312 participants. IAFF/Canada progress reports state that the target was to train about 550 participants per year. IAFF/Canada did not reach this goal, noting that class size was below the estimated 25 students per class.

The IAFF/Canada continues to develop Master Instructors through an ongoing training program and have developed a Canadian Instructor Policy. They continuously revise the course curriculum to ensure that the training program meets the standards.

A review of the IAFF/Canada by PS determined that availability of bilingual courses was lacking. This was not in keeping with the Treasury Board Policy on Transfer Payments and did not meet the requirements of the agreement. The training manual was translated to French in 2011, and bilingual instructors are now available. The IAFF/Canada now has the capacity to provide bilingual training.

The IAFF/Canada continues to develop partnerships to deliver the training within provinces.

4.2.2 Level of Access to Training

The IAFF/Canada training program was meant to be a national program; however, the training has been concentrated in two provinces, Ontario and British Colombia, as shown in Figure 2. A few courses have been provided in the Northwest Territories, Yukon, Alberta, Saskatchewan and Nova Scotia. The other provinces have not yet accessed the training.

Figure 2 - Number of course participants from April 2009 to December 2011

Figure 2 illustrates the number of course participants by province from April 2009 to December 2011 (714 in British Columbia, 454 in Ontario, 51 in Saskatchewan, 27 in Alberta, 29 in Yukon, 19 in Nova Scotia, and 18 in Northwest Territories).

Most interviewees noted that there are other HazMat course providers in their province. However, they indicated that there are challenges accessing these courses due to distance/isolation of certain communities. Some communities cannot afford the associated training and travel costs, as well as the travel time and the concurrent absence of employees. Some provinces/territories do not have training organizations that provide HazMat training. For example, fire fighters from the Northwest Territories would have to go to Alberta to attend training.

The IAFF/Canada interviewees noted that they have conducted promotional activities within provinces. They participated in several conferences (e.g. Canadian Association of Fire Chiefs Conferences), and distributed brochures and pamphlets at the Canadian Emergency Management College courses and other seminars. They have also advertised the training in magazines and participated in several meetings at the international, provincial and local levels.

Despite promotional activities, the IAFF/Canada has encountered some obstacles in providing the training on a national basis (i.e. within every province). Accreditation is one of these obstacles. Provincial authorities set the framework for fire services and related training. Although the IAFF/Canada HazMat training course is designed to be compliant with the National Fire Protection Association's Standard 472^{Note 25}, it is not recognized by most provinces.

Provincial training is usually recognized and accredited by two organizations, the ProBoard or the International Fire Service Accreditation Congress. The absence of IAFF/Canada accreditation limits uptake of the course by participants in most provinces. Progress reports note that, in the absence of accreditation, the IAFF/Canada has tried to make agreements with provinces, but progress has been slow. A 2011 Management Review conducted by PS identified accreditation as the main obstacle to increased course delivery. One exception to this is the province of British Columbia where the IAFF/Canada has trained many fire fighters. The province has made arrangements with the Justice Institute of British Columbia to allow the fire fighters to write the necessary examination. The IAFF/Canada has applied for accreditation through the ProBoard and they anticipate that they will receive it in the fall of 2012.

4.2.3 Skills and Knowledge to Respond to HazMat Incidents

The IAFF/Canada measures results of the courses using pre-course and post-course testing^{Note 26}. For 2009-2010 and 2010-2011 combined, the average pre-test result was 56% while the post-test result was 79%; this suggests a 22% increase in knowledge as a result of the course.

Based on the course evaluations from 2009-2010 and 2010-2011, participants overwhelmingly agreed that the training:

• was valuable in helping them safely do their job;
• that it covered the skills most emergency responders need;
• that the training will help them respond and train more effectively; and
• that they will practice the skills on the job.

Participants also felt that the quality of the instructor and the overall quality of the course was good.

One testimonial sent to the IAFF/Canada states that the IAFF/course was "one of the most informative and professionally delivered courses they have taken in this field and have gained much confidence as a result". Participants also reported having changed their way of thinking when responding to calls.

All course participants interviewed were satisfied with the IAFF/Canada course noting that the information was at the right level and that the instructors were knowledgeable. One participant mentioned that his group would have liked to take the course in French but that the training was not available at the time. Interviewees agreed that the training increased their skills and knowledge of how to respond during a HazMat incident, and that the training had increased their confidence in their ability to respond.

The target audience for training is mainly fire fighters (career, volunteers and paid on-call), with a goal to include 15% of participants from other emergency services across Canada (emergency medical services, police, etc.). Since the beginning of the training program, fire fighters represent 89% of course participants and other first responders represent 11%. The results are presented in Table 3.

IAFF/Canada progress reports indicate that interoperability between first responders was improved in the classes where other emergency response personnel participated. The training allows all first responders to have the same vision and help them to respond safely and effectively to an incident. One fire department testimonial states that "the opportunity for fire, emergency management services and police to work and learn together provided for a more in-depth understanding of all our roles at any scene". All interviewees agree that the training would help them to work more effectively with first responders during a HazMat training exercise or event. Some noted that the IAFF/Canada training is not entirely responsible for this as they had received previous training courses or because they work with other first responders in their community on a daily basis.

4.2.4 Protecting the Safety of Fire Fighters and the Public

Through interviews, the evaluation examined whether fire fighters used the skills acquired during the training to respond to an actual HazMat event and whether the training has had an impact on community safety.

There are no national statistics on the number of incidents per year and related fire fighter deaths/injuries as each province has its own protocols on recording this data. However, course participants reported being better prepared for a HazMat incident even though the majority had not experienced such an event. They noted several examples of impacts of the training: that incidents can now be appropriately identified and handled as HazMat (e.g. carbon monoxide incident); that protocols have been re-evaluated for specific incidents; and that they now carry a guidance manual in the fire trucks (i.e. Canutec manual). One course participant stated that the course had been instrumental in responding to a gas leak in their community.

The majority of interviewees felt that the course had a positive impact on safety in the communities where the training took place. Community preparedness has increased as a result of having better trained first responders. One course participant mentioned that the course enabled them to be proactive. They now visit specific buildings in their community (i.e. schools, hospitals) to make an inventory of potential HazMat threats. Firefighters know what they are dealing with when an incident happens and this makes the community safer.

The IAFF/Canada has received several letters of encouragement and testimonials from the fire departments and participants that show specific examples of the positive impact training has had in communities. For example, one fire department states that they are "much better positioned to respond to the many very real HazMat threats that face (their) community every day". Other communities commented that they are "in a better operation capable position in responding to and mitigating these serious incidents" and that "as a result of the excellent course material and instructional delivery, the citizens in our community and this region of Canada, are better protected for hazardous materials incident".

4.3 Performance--Efficiency and Economy

The evaluation examined whether the IAFF/Canada contribution program being administered efficiently to produce outputs and progress towards expected outcomes. Program efficiency was assessed by examining how PS has minimized the use of resources in managing the contribution agreement. Program economy was assessed by conducting a limited cost comparison.

4.3.1 Program Efficiency

The Program is managed under a single contribution agreement with one recipient; thus, it is expected that the level of PS effort required to manage the agreement will be minimal. Program interviewees indicate that the contribution agreement was monitored through a review of budget submissions and progress reports from the IAFF/Canada. An Advisory Committee has met at least once a year to review progress; to provide advice on meeting objectives; to review training material; and discuss any outstanding issues and opportunities. Representation at the meetings included the Canadian Emergency Management College and the IAFF/Canada. PS officials also attended the course twice to monitor course delivery, at periodic intervals.^{Note 27} A Management Review was conducted by PS in 2010-2011. The Review identified opportunities and expectations for the training program, such as the provision of bilingual courses and the issue of accreditation.

The evaluation examined PS resource utilization by comparing the administration ratio^{Note 28} of the Program to other PS transfer payment programs. The ratio is, as expected, generally low, ranging from 2.2% to 3.1% for three of the four years. In contrast, in 2010-2011, when the Management Review was conducted, the ratio was high at 17.3%, due to additional staff time that was spent during this year. The average ratio at 5.6% compares favourably to other PS contribution programs. Details of the calculations are contained in Annex C.

Further analysis of expenditure data reveals lapses^{Note 29} in Program spending. PS financial management advisors indicate that the lack of accuracy in year-end estimates, for the amounts set aside for Payables at Year-end (PAYEs) claims, contributed to financial management challenges. The over-estimation of PAYEs, in the first two-years, caused the Department to reduce the expenditures in the financial statements in the year that the PAYEs were written-off, once it was known that IAFF/Canada would not submit a claim for those years. Further analysis of financial data reveals lapses in contributions of $322,600 in 2008-2009; $193,500 in 2009-2010; $184,300 in 2010-2011; and $89,200 in 2011-2012. The lapses are primarily due to late issuance of funding in 2008-2009 and the accreditation issue that inhibited planned training delivery.

4.3.2 Program Economy

Recognizing that course costs for HazMat, and emergency management training in general, vary across Canada, the evaluation conducted a limited comparison of the cost per training participant for two other courses that are similar to the IAFF/Canada training. The first comparator was the HazMat Materials Emergency Response - Operations course offered by the Justice Institute of British Columbia^{Note 30}. The tuition price for the course is about $990 per participant. The training audience includes employees of industries that manufacture chemicals, transport or store other hazardous materials; and fire personnel and police who respond to incidents involving HazMat. The students learn techniques to recognize and assess the degree of risk involved with HazMat and their containers; safe defensive control techniques to protect people; and the steps involved in managing an incident. The second comparator was NFPA 472 Hazardous Materials Operations Plus course at the Fire and Emergency Services Training Institute in Toronto^{Note 31}. The tuition price for the course is about $920 per participant. During this course, the students learn techniques to protect people, the environment, and property from the effects of an unplanned HazMat release. These two courses comply with the laws and standards for HazMat emergency response at the operations level as per National Fire Protection Association Standard 472 and they are accredited by the International Fire Service Accreditation Congress and ProBoard. The IAFF/Canada course is delivered at a cost to the federal government of $835 per participant.

A comparison of the tuition price of these other courses to the cost per participant of the Program provides a limited comparison, since there are a number of unknowns in the tuition price. For example, if the analysis assumes that the tuition price includes developmental costs, overhead, and is offered on a break-even basis, the costs are comparable to the federal Program. However, these other courses could be subsidized by partnerships or volunteer efforts that contribute "in-kind" and/or are not reflected in the tuition price; and profit may be included in the price if the courses are not offered on a "break-even" basis. Despite these unknowns, it appears that the cost per participant of IAFF/Canada course, at $835, is within range of these comparators.

5. Conclusions

5.1 Relevance

The IAFF/Canada Contribution Program was originally intended to address a gap in training among fire fighters and contribute to interoperability among first responders. Because training data is not collected nationally it was difficult for the evaluation to determine if these training gaps persist. However, based on available evidence, there is still a need for HazMat training for first responders. Systematic collection of data related to HazMat training across the country would provide a more accurate understanding of the current level of first responder training and the remaining training gaps.

Federal support for first responder training was evident in Budget 2007 and the years following the development of the first PS strategy related to CBRNE events^{Note 32}. Emergency management training is not part of the stated 2012-2013 Departmental Key Priorities; however it remains a Sub-Sub-Activity in the Program Activity Architecture under 1.4.1.2 Emergency Management Training and Exercises.

Legislation and policy frameworks show that it is the federal government's role, in collaboration with the provinces/territories, to facilitate national coordination and advancement of emergency management training. The IAFF/Canada Contribution Program appears to be disconnected from the leadership and collaborative role of PS in relation to the provinces/territories. Missed opportunities for synergy and strategic linkages may be facilitated through the Federal/Provincial/Territorial Forum for Senior Officials Responsible for Emergency Management. In addition, the current fiscal and organizational context suggests a change in focus of PS training efforts and movement toward a more strategic role. Alternate ways of supporting emergency management training are being examined, and a new business model is evolving. PS will broaden its partnership base to support and strengthen emergency management training through the development and promotion of national emergency management training standards in consultation with provincial/territorial partners, academia, and emergency management practitioners. In addition, PS will provide emergency management training to federal public servants through a memorandum of understanding with the Canada School of Public Service. Under this memorandum of understanding, PS will provide policy and program guidance.

In terms of duplication, HazMat training is available in most provinces through provincial emergency management organizations, colleges and universities, and the private sector; however, the key difference is that this Program delivers training on-site, making it advantageous to small communities that cannot afford the risk of having a portion of their workforce away concurrently.

5.2 Performance

The funding provided by PS increased the IAFF/Canada's capacity to develop and deliver the training program. The Program developed bilingual course material, trained qualified instructors and, from April 2009 to December 2011, trained a total of 1312 participants.

In the provinces where training was delivered, there has been increased access for fire fighters in small/isolated communities. The Program has not significantly increased access to HazMat training on a national level since training has been mainly concentrated in two provinces. The fact that the IAFF/Canada courses are not yet accredited has been a significant obstacle to provide training in many provinces. However, the IAFF/Canada continues to promote the program and has applied for accreditation.

Evidence shows that the training program is of good quality and that it has increased the knowledge and confidence of participants in their ability to manage HazMat incidents. The training has improved interoperability in those communities where emergency response personnel, other than fire fighters, also attended the course. There is little evidence to demonstrate the wider impact of the Program in terms of protecting the safety of fire fighters and the general public; however, the training was well received in the communities where it took place, and several examples provided by interviewees indicate that the training had a positive effect on community safety by increasing preparedness.

With some exceptions, the Program is being administered efficiently to produce outputs and progress towards expected outcomes. The Program is considered efficient since the program administration ratio compares favourably to other PS transfer payment programs. However, there are lapses in Program spending; more accurate recipient estimates and closer attention to year-end consolidation of expenditures may improve this result. In terms of Program economy, the cost per training participant is within range of two other similar training offerings that were compared during the evaluation.

6. Recommendations

The Evaluation Directorate recommends that the ADM, Emergency Management and Regional Operations Branch:

1. Develop program alternatives that better reflect PS's leadership and collaboration role.

   Program alternatives should consider:

   • need/gaps in HazMat training in provinces/territories, through consultation with the Federal/Provincial/Territorial Forum of Senior Officials Responsible for Emergency Management); and
   • PS's evolving business model in the area of emergency management training.
2. Develop a mechanism that will promote timely updates of recipient estimates and expenditures in order to provide a more accurate financial representation of the program.

7. Management Response and Action Plan

Annex A: Documents Reviewed

Department of Finance Canada. News Releases, April 10, 2007, Canada's new government investing in safe, secure communities.

Department of Finance Canada. The Budget Plan 2007, A Safer Canada: Building a stronger Canada in a Modern World. Hazardous Materials Training for Canada's First Responders.

Department of Justice Canada. Emergency Management Act (S.C. 2007, c.15)

International Association of Fire Fighters (2005). Submission for Emergency Preparedness Act Review.

International Association of Fire Fighters (2005). Submission to the Standing Committee on Finance - Pre-budget Consultation.

International Association of Fire Fighters (2011). Stage set for 18^th Canadian Legislative Conference.

International Association of Fire Fighters (2011), Submission to Standing Committee on Finance 2012 Pre-budget Consultations.

International Fire Service Accreditation Congress (2012). Accredited Entities and Levels.

Parliament of Canada (2008). Report of the Senate Committee on National Security and Defence, Emergency Preparedness in Canada.

Parliament of Canada (2011).Report of the Standing Committee on Finance, Staying focussed on Canadian Jobs and Growth.

Public Safety Canada (2008). Results-Based Management Framework and Risk-Based Audit Framework, For International Association of Fire Fighters, Canada, Hazardous Materials Training Initiative.

Public Safety Canada (2010). A Strategy for Emergency Management Training in Canada 2010-2015.

Public Safety Canada (2011). Building Resilience Against Terrorism: Canada's Counter-terrorism Strategy

Public Safety Canada (2011). Chemical, Biological, Radiological, Nuclear, or Explosive Resilience Strategy for Canada.

Public Safety Canada (2011). Management Review of the Contribution Agreement with the International Association of Fire Fighters, Canada

Public Safety Canada. News Releases, April 27, 2009, Minister Van Loan announces $ 2.5 million for hazardous materials (HazMat) training.

Public Safety Canada. 2010-2011 Report on Plans and Priorities.

Public Safety Canada. 2011-2012 Report on Plans and Priorities.

United States Department of Homeland Security (2010). United States-Canada Joint Border Threat and Risk Assessment.

Websites

IAFF/Canada: http://www.iaff.org/canada/index.htm

Public Safety Canada Website: http://www.publicsafety.gc.ca

Annex B: Training organizations

The following table provides examples of HazMat training providers within provinces that are accredited through the ProBoard and/or the International Fire Service Accreditation Congress.^{Note 33}

HazMat Training Providers in Provinces

British Columbia

• College of The Rockies
• District of North Vancouver Fire & Rescue Services Training Center
• Justice Institute of British Columbia, Fire and Safety division
• Vancouver Island Emergency Response Academy

Alberta

• Alberta Emergency Management Agency
• Office of the Fire Commissioner - Alberta Municipal Affairs

Saskatchewan

• Office of the Fire Commissioner

Manitoba

• Manitoba Emergency Services College, Office of the Fire Commissioner

Ontario

• Office of the Ontario Fire Marshal

Quebec

• École nationale des pompiers du Québec

Nova Scotia

• Nova Scotia Fire Service Professional Qualifications Board

Prince Edward Island

• Atlantic Police Academy

Newfoundland

• Office of the Fire Commissioner, Newfoundland and Labrador

Annex C: Financial Analysis

Values in the table are in dollars and have been rounded to the nearest hundred.

Note 1: PS Program Staff salaries have been estimated based on one employee working on the program full time for one week twice per year (4%). In fiscal year 2010-2011, PS undertook a Management Review. Thus, more staff was expended (estimated at an additional 30%). For the Executive Director's office, salary estimates are based on the assumption that 1% of two employees' salaries are dedicated to supporting the program.

Note 2: For fiscal year 2008-2009, an expense of $300,000 was incurred through the creation of a PAYE. The actual payments in that year totaled $177,400. As a result, the remainder of the PAYE ($122,600) was written off. The same situation occurred in fiscal year 2009-2010 where an expense of $500,000 was incurred through the creation of a PAYE. In this scenario, the actual payments totaled $306,500 and the remainder was subsequently written off.

Date modified:

2022-07-26