2016-2017 Evaluation of the Regional Resilience Assessment Program and the Virtual Risk Analysis Cell

2016-2017 Evaluation of the Regional Resilience Assessment Program and the Virtual Risk Analysis Cell - PDF Version (526 KB)

Table of contents

• Executive Summary
• 1. Introduction
• 2. Profile
  • 2.1 Background
  • 2.2 Program Objectives
  • 2.3 Resources
• 3. About The Evaluation
  • 3.1 Objective
  • 3.2 Scope
  • 3.3 Methodology
  • 3.4 Limitations
• 4. Findings
  • 4.1 Relevance
  • 4.2 Performance - Effectiveness
  • 4.3 Performance - Efficiency and Economy
• 5. Conclusions
  • 5.1 Relevance
  • 5.2 Performance – Effectiveness
  • 5.3 Performance - Efficiency and Economy
• 6. Recommendations
• 7. Management Response And Action Plan
• Appendix A: Evaluation Questions
• Appendix B: Alternative Service Delivery Options
• Appendix C: Bibliography
• Footnotes

Executive Summary

Evaluation supports accountability to Parliament and Canadians by helping the Government of Canada to credibly report on the results achieved with resources invested in programs. Evaluation supports deputy heads in managing for results by informing them about whether their programs are producing the outcomes that they were designed to achieve, at an affordable cost; and supports policy and program improvements by helping to identify lessons learned and best practices.

What we examined

This evaluation covers Public Safety Canada’s (PS) Regional Resilience Assessment Program (RRAP) and Virtual Risk Analysis Cell (VRAC), two programs launched under the 2011 Canada-U.S. Beyond the Border Action Plan. The evaluation covers the period from 2011-2012 to 2016-2017 with a primary focus on the programs’ domestic Canadian work. The RRAP is an all hazards resilience program featuring site assessments designed to help critical infrastructure (CI) owners and operators identify and measure vulnerabilities and interdependencies, as well as improve their resilience. The VRAC provides risk-based analysis and analytical support for CI sectors, identifying potential impacts of disruptions. This evaluation follows the Treasury Board Policy on Results and thus assesses the continued need for the programs, their alignment with Government of Canada and PS priorities, alignment with federal roles and responsibilities, achievement of expected outcomes, and demonstration of efficiency and economy.

Why it is important

The Beyond the Border Action Plan jointly committed Canada and the U.S. to implementing measures to improve national security and emergency management (EM) cooperation, including measures to improve CI resilience and cyber security. This has been articulated through a commitment between PS and the Department of Homeland Security to work with provincial, territorial, and state authorities, as well as with CI sectors to identify and assess CI risks, evaluate owner and operator capabilities, and introduce tools and training to enhance resilience across all CI sectors.

The rationale for the programs has been to strengthen the resilience of CI assets and networks (e.g., water treatments plants, manufacturing plants, electrical power grid), bolster the resilience of supply chain interdependencies, and minimize risks to cross-border economic stability and national security. The programs also provide mechanisms for sharing information and strengthening collaboration to improve CI resilience. Thus, the programs build on bilateral efforts to conduct site assessments, analyze risks, and develop mitigation measures in collaboration with regional officials and private sector stakeholders on both sides of the border.

What we found

Relevance

The underlying objectives of both RRAP and VRAC remain relevant today. They have responded to the initial needs and expectations to address CI vulnerabilities by assisting owners, operators, and stakeholders involved in ten critical infrastructure sectors. RRAP site assessments are proving to be useful in helping to identify CI strengths and weaknesses. RRAP assessments are leveraging new investments in CI by owners and operators; evidence suggests that these investments could be even greater given additional incentives (e.g., in the form of grants and contributions, tax credits, etc.). Regarding VRAC, the continued need is underlined by its analytical outputs in the form of impact assessments and action reports, its support during events such as the Fort McMurray Wildfires, as well as through its liaison role on exercises and simulations.

Both programs are key components of the federal government’s CI agenda and are identified as a priority in multiple fundamental government documents including the originating 2011 Canada-U.S. Beyond the Border declaration. However, although CI is a key federal government role and RRAP and VRAC contribute to the role by supporting the objectives the National Strategy for Critical Infrastructure, this responsibility is shared with numerous stakeholders, such as provincial/territorial governments, local authorities, and owners and operators, due to the interdependent nature and properties of physical and security-related assets.

Performance

With respect to performance, both programs have contributed to CI stakeholders’ understanding of risks and threats to their assets and organizations. Interviewees stated that RRAP reports provide specific and actionable all-hazards risk information to support CI resiliency. Owners and operators stated that they took tangible action to mitigate the key risks identified during their site assessment, typically in the form of business continuity planning and investments to address deficiencies. However, faced with increasing demand for site assessments, RRAP will need to take steps to prioritize its efforts. Greater prioritization is particularly important given the evaluation finding that site selection in RRAP’s initial phase was demand-driven and concentrated in areas where specialized PS resources were located to promote the program, resulting in regional and sector imbalances in coverage. 

Regarding VRAC, interviewees indicated that this program has facilitated the Government Operations Centre and stakeholder understanding of CI supply chain interdependencies, risks and threats during events and in exercises. VRAC is positioned to provide timely CI analysis, including impact assessment reports, suspicious incident and threat reports, and after-action reports. However, the CI Gateway is an underused resource that could benefit from greater promotion.

In terms of CI community collaboration and partnerships, both RRAP and VRAC play a significant role in developing and sharing best practices, with VRAC having a more visible role in this regard, in large part due to its management of the CI Gateway. Both programs have contributed to enhancing PS and stakeholder understanding of cross-sector dependencies and linkages. Evidence of cross-sector initiatives exists in the form of assessments, exercises, networks, conferences and other fora. With respect to VRAC, while the evidence from interviews and documentation (e.g., CI Performance Reports) indicates that many collaboration and outreach activities have occurred over the course of the five-year period covered by this evaluation, the attribution of impacts and results (i.e., the extent to which these activities have contributed to mitigate impacts from CI disruptions) is challenging, especially given the absence of a specific and defined collaboration and outreach strategy.

With respect to efficiency and economy, the RRAP and VRAC have operated on a relatively modest budget as components of a portfolio of CI resilience initiatives. Program expenditures have exceeded the forecasted budget in each of the five years of operation due primarily to the additional resources needed to meet demand and workload. While the number of program staff has increased, results have been overly dependent on a few key experts, which imposes potential limits on program scalability.

Nevertheless, recent governance changes have reinforced coordination and resource sharing between the two programs. These are important steps towards optimizing the use of resources especially considering the potential benefits to be gained from closer integration of efforts between the programs. Furthermore, evidence from the examination of alternative service delivery approaches suggests that, except for closer collaboration and partnership with provinces and territories, there is little to be gained and likely more to lose by changing the service delivery model at this time.

In sum, these two programs do not constitute the sole solution to mitigate impacts from CI disruptions, but they remain relevant, have performed effectively and efficiently in their inaugural phase, and have proven to be an important complement to efforts of other CI and EM programs, initiatives and assets, whether they be at the national, provincial or international level.

Recommendations

The following recommendations are being provided in the spirit of continuous improvement.

The Senior Associate Deputy Minister of the National and Cyber Security Branch should consider:

1. Developing RRAP site assessment selection processes, and VRAC products, that consider risks and priorities.
2. Leveraging CI community engagement and targeted outreach activities to support achievement of RRAP and VRAC program objectives.
3. Ensuring appropriate resources to support the scope of activities outlined in annual RRAP and VRAC workplans.
4. Exploring options to support owners and operators to address improvements identified through site assessments that will increase the resilience of CI sites across Canada.

Management Response and Action Plan

Management accepts all recommendations and will implement an action plan.

1. Introduction

This report presents the results of the Public Safety Canada (PS) 2016-2017 Evaluation of the Regional Resilience Assessment Program (RRAP) and the Virtual Risk Analysis Cell (VRAC), which were introduced as part of a package of initiatives announced under the 2011 Canada-U.S. Beyond the Border Action Plan. While the original focus of the Action Plan was on enhancing cross-border security and the legitimate flow of people, goods and services, the primary focus of this evaluation was on PS’ domestic RRAP and VRAC efforts over the past five years, from 2011-2012 (program inception) to 2016-2017.

Critical Infrastructure (CI) refers to processes, systems, facilities, technologies, networks, assets and services essential to the health, safety, security or economic well-being of Canadians and the effective functioning of government. Currently, the number of CI sites of national interest in Canada has been estimated by PS at approximately 900. It is possible, and indeed highly likely, that the total number of sites capable of being assessed under RRAP is well beyond the current program capacity. Thus, an important adjunct of this evaluation was to consider the efficacy of potential alternate service delivery options or approaches to this program. 

CI protection is a shared responsibility between the federal government and the provinces and territories. The federal government may also provide financial assistance to a province or territory in the case of a natural disaster or emergency to help mitigate future risk and improve resilience.

2. Profile

2.1 Background

RRAP

The RRAP is a comprehensive risk assessment program designed to help owners and operators of Canadian CI improve their organizations’ resilience to hazards such as cyber threats, accidental or intentional man-made events, and natural catastrophes. The RRAP conducts site assessments of owner/operator facilities and broader regional-scale CI assessments. It also produces reports and guidance tools to help organizations understand their points of vulnerability and ways in which they can improve their CI resilience. RRAP site assessments are voluntary, non-regulatory, free-of-charge and confidential.

RRAP assessments are conducted using tools developed by Argonne National Laboratory, managed by University of Chicago Argonne, LLC, for the U.S. Department of Energy's Office of Science. Assessment tools have been adapted for use by PS under a Memorandum of Understanding (MOU) with the Department of Homeland Security (DHS). The three main tools used to deliver RRAP assessments are the following:

1. Critical Infrastructure Resilience Tool (CIRT): an on-site, survey based tool that measures the resilience and protective measures of a facility from an all-hazards perspective;
2. Critical Infrastructure Multimedia Tool (CIMT): a multiplatform software tool that generates an interactive visual guide of a CI facility, featuring spherical photography; and
3. Canadian Cyber Resilience Review (CCRR): an on-site, survey-based tool that measures the cyber security posture of an organization.

VRAC

The VRAC analyzes supply chain processes and vulnerabilities, builds understanding of cross-sector dependencies, and fosters communication and collaboration with CI communities nationally and internationally. VRAC identifies potential impacts of disruptions and produces impact assessments and geospatial maps of CI to support federal and provincial response efforts in cases of emergencies or CI disruptions.

The VRAC supports simulation exercises and conducts joint analyses of cross-border CI with the DHS and leads engagement on CI issues within international fora, including the Critical Five, the EU-U.S.-Canada CI Experts Group and the Canada-Israel CI Protection Working Group. The VRAC also maintains the CI Gateway, a web portal designed to facilitate access to CI knowledge and information sharing with CI stakeholders.

The VRAC operates in both an event (e.g., providing direct information through the CI Gateway during events) and steady state mode (e.g., through ongoing activities and projects to identify and assess vulnerabilities and cyber dependencies).

Partners and stakeholders

As one of the two signatories to the Beyond the Border Agreement, the U.S. government is a key partner and stakeholder in the RRAP and VRAC. PS and DHS collaborate closely on their respective assessment programs especially with respect to cross-border assessments and supply chain interdependencies.

The responsibility for public safety, CI resilience and emergency management is shared between all levels of government. As such, the provinces, territories, and municipalities are key CI stakeholders, as are owners and operators. The roles and responsibilities of these actors are outlined in Canada’s Action Plan for Critical Infrastructure (2014-2017).^Footnote1 Other key stakeholders include the ten CI sector leads at the federal level and the respective sector associations and networks. The sector leads, shown in Table 1, are particularly important to RRAP and VRAC in their community collaboration and information sharing efforts. For VRAC, the Government Operations Centre (GOC) is also a key partner, especially so during events where VRAC resources are integrated with the GOC to provide CI expertise and response support.

Law enforcement and intelligence agencies, such as the RCMP and CSIS, also have a stake in RRAP and VRAC. Both provide intelligence products and services that complement VRAC analysis, especially with respect to organized criminal or terrorist related activity directed at CI sites or with CI implications.

Governance Structure

The RRAP and VRAC are part of the Critical Infrastructure and Strategic Coordination Directorate (CISCD) of PS, which is within the National and Cyber Security Branch. Both RRAP and VRAC play a role in delivering Canada’s National Strategy for Critical Infrastructure (the Strategy), which defines the overall approach to enhancing CI resilience in Canada and supports coherence and coordination of planning and activity across all levels of governments and the private sector. With a primary focus on the latter, the RRAP and VRAC contribute to all three objectives of the Strategy: partnership building, information sharing, and all-hazards risk management. The National Cross Sector Forum and the Federal-Provincial-Territorial Critical Infrastructure (FPT CI) Working Group are examples of two bodies dedicated to facilitating the Strategy’s implementation. The National Cross Sector Forum includes representatives from each of the ten CI sectors. The annual meeting is co-chaired by the Deputy Minister of Public Safety and a provincial/territorial representative. The FPT CI Working Group is the standing forum and primary conduit for federal/provincial/territorial government collaboration on CI matters. Both fora have been used to raise awareness of the RRAP and VRAC and to support program development and improvement.

Within this context, the RRAP and VRAC have undergone recent (2015) organizational changes aimed at improving structural integration and coherence, including alignment of activities (e.g., leveraging VRAC expertise to support RRAP site selection, using information collected from site assessments to support VRAC event-state impact assessments).

2.2 Program Objectives

The December 2011 Beyond the Border Action Plan committed Canada and the U.S. to implementing measures to improve national security and emergency management cooperation, including measures aimed at improving CI resilience and cyber security.  

The Action Plan recognized that building CI resilience could contribute to cross border economic stability and national security. As such, the Action Plan called for PS and DHS to work with their respective jurisdictions (e.g., provinces/territories, states) and CI sectors to identify and assess risks facing CI, evaluate owner/operator capabilities, and introduce tools and training to enhance resilience.

The Action Plan provided the practical roadmap for these efforts towards protecting vital CI assets, such as the North American electricity grid, transnational pipelines, and international bridges. The Plan also contained measures for enhancing cyber CI (e.g., oil, gas, water, pipeline assets) increasingly managed by automated industrial control systems with their own inherent vulnerabilities (see also Canada’s Cyber Security Strategy – Private Sector Engagement pillar).

2.3 Resources

As shown in Table 2 below, total expenditures over the 5-year evaluation period amounted to $7.6 million compared to the total budgeted estimates of $4.9 million. Salaries accounted for a significant portion of the difference. The Beyond the Border initiative funded 8 full-time equivalent (FTE) staff (9.2 FTEs including corporate services). The programs now have a total of 20 positions (18 are currently filled). New positions were created due to program demand and were funded from two sources: (1) internal reallocations and available existing funding and (2) funds to permit more cyber-focused work to develop (e.g., the Canadian Cyber Resilience Review), which only began in 2016-2017.

One year of additional funding of $1.37 million was announced in the most recent budget (March 2017) for the continuance of RRAP and VRAC operations in 2017-18. Table 2 illustrates the budget and expenditures for RRAP and VRAC for the period covered by the evaluation (2012-2013 to 2016-2017).

*Note on Table 2: Budget for FY2012-2013 to FY2015-2016 includes only funding received under the Beyond the Border (BTB) initial allocation for both programs (RRAP & VRAC)
*Note on Table 2: A breakdown by program (RRAP and VRAC) was not provided in the original Beyond the Border costing tables, so only total figures are available.
*Note on Table 2: Figures for 2012-2013 to 2016-2017 do not include corporate overhead (e.g., employee benefit plans, accommodations, etc.).
*Note on Table 2: Budget for FY2016-17 includes funding under the BTB initial allocation and part of Cyber Phase II.

2.4 Logic Model

The logic model is a visual representation that links what the program is funded to do (activities) with what it produces (outputs) and what it intends to achieve (outcomes). It also provides the basis for developing the evaluation matrix, which gave the evaluation team a roadmap for conducting this evaluation. The following is the CISCD logic model for CI. RRAP and VRAC operations are integrated within this logic model.

Image Description

The Logic Model works as a communication tool that summarizes the key elements of the program, explains the rationale behind program activities, and clarifies all intended outcomes.

The inputs are human and financial resources, and information from partners and stakeholders.

These support the five activity areas:

• Delivering risk analysis
• Conducting Vulnerability and Dependency assessments
• Raising Awareness, Delivering Exercises, Coordinating Learning and Community Building Events
• Conducting research and developing policy
• Engaging, Consulting, and Creating Networks/Partnerships

The activities lead to the outputs of:

• vulnerability reports (CI resilience tool)
• Canadian Cyber Resilience Review reports
• Cross-border RRAP final reports
• Threat reports/briefs
• VRAC impact assessments
• Dependency-interdependency modelling and analysis reports
• VRAC daily infrastructure reports
• VRAC disruption scenarios
• Policies, guidance documents and awareness products
• Awareness activities (workshops, learning sessions)
• Exercises (TTX, drills, full-scale, etc.)
• CI multimedia Tool
• CI Resilience Tool scenario dashboards
• Sector networks and OGDs (e.g. NCSF, FPT CI WG and MSN)
• CI and Global CI Information Gateways
• International engagement activities (C5, CAN-EU-US)
• Sustained and commonly beneficial relationships.

The Outputs lead to the following immediate outcomes:

• CI Stakeholders have an understanding of risks and threats to their assets and organizations.
• CI owners and operators have access to timely and actionable cyber and physical risk information.
• CI stakeholders and partners understand effective risk management principles (includes international partners)
• CI stakeholders are aware of existing enabling collaboration infrastructure

The immediate outcomes then lead to the following intermediate outcomes:

• CI stakeholders take all-hazards risk management action
• CI Stakeholders take preventative measures to avoid disruptions
• CI Stakeholders respond to disruptions rapidly and effectively and cascading impacts are mitigated
• CI community collaborates in sharing information and adopting best practices
• Partnerships are revitalized and sustained
• Information is shared and protected

The intermediate outcomes leads to the following final outcome: Canada’s Critical Infrastructure is Secure and Resilient and Impacts from Disruptions are Mitigated.

The final outcome leads to the ultimate outcome: A safe and resilient Canada

About The Evaluation

3.1 Objective

Conducted in accordance with the Treasury Board Policy on Results, this evaluation examined the following five primary evaluation issues:

1. The continued need for RRAP and VRAC
2. The alignment with PS priorities and government direction
3. The alignment with government roles and responsibilities
4. The achievement of expected outcomes (effectiveness)
5. The efficiency and economy of both programs

This evaluation also examined the extent to which the current delivery model is the most optimal and if there are other delivery models that would provide the Government of Canada with better value-for-money.

3.2 Scope

The scope of this evaluation covers the period from the inception of both programs (late 2011) to 2016-2017. While alignment with DHS is an important feature of the programs considering the cross-border interdependencies related to CI resilience, it should be noted that the primary focus of this evaluation effort is on the domestic aspect of these programs.

3.3 Methodology

3.3.1 Evaluation Questions

In total, 15 evaluation questions and 48 indicators were identified in the framework. The evaluation questions are linked to five evaluation issues noted earlier (section 3.1). Please refer to Appendix A for evaluation questions and the full evaluation framework which outlines the supporting indicators and data sources required to conduct an objective, thorough, and relevant evaluation.

3.3.2 Lines of Evidence

Document and Literature Review

The document and literature review provided the evaluators with a full coverage of the work undertaken by the RRAP and VRAC, as well as an understanding of the context, the environment and the evolution over time. It also provided reliable key data for many of the indicators. Policy documents, program files, and other appropriate literature and print material were reviewed to support analysis, Findings and conclusions. The full list of the documents reviewed is presented in Appendix C (bibliography).

Document and Administrative Data Review

Key data files and repositories, including performance data, survey data, financial data, the Criticality Index data, and CI Gateway data and repository were reviewed.

Key Informant Interviews

The evaluation team conducted semi-structured interviews (individual and group) with 51 key representatives from the Government of Canada, the provinces, as well as with clients (e.g., owners and operators) from both programs.

Site Visits

Two site visit were conducted to provide a boots-on-the-ground assessment of the programs, the first one in Halifax (NS) and Saint John (NB) in February 2017, and the second one in Regina and Moose Jaw (SK) in March 2017. The site visits allowed for direct observation of program impacts and interviews with stakeholders who have had first-hand experience with RRAP and VRAC products, people, processes and systems.

Case Examples, Impact Assessments, Reports

Case examples of RRAP assessments and VRAC analysis (e.g., impact assessments, threat assessments, risk Profiles, geospatial analysis, etc.) were selectively identified and examined to assess the quality, nature and type of interventions undertaken by the programs.

3.4 Limitations

Interview sample

Qualitative information from interviews represents the views of a sample of interviewees selected with the assistance of program staff from among PS representatives, owners and operators, and other stakeholders familiar with the programs. Thus, there is a risk that this sample contains an inherent bias in favour of the programs. The evaluators addressed this risk by stressing confidentiality and corroborating interview input from other lines of evidence.      

Performance data

In some cases, lack of performance data for certain indicators or gaps in the data for certain time periods were noted. Thus, the evaluators exercised their professional judgement in extrapolating and interpolating from existing data or in compensating with qualitative sources of evidence.   

Causality between activities and outcomes

The nature and the context of both programs make it difficult to prove the causality between certain activities and outcomes, especially for the intermediate and ultimate outcomes. For instance, some of the outcomes measured could be a result of multiple factors, either internal or external to the programs. Some caution must be exercised in attributing results directly and solely to the programs.

4. Findings

4.1 Relevance

The evaluation examined program relevance from two perspectives: 1) whether there is a continued need for the programs, taking into account the extent to which they have evolved since inception and 2) the extent to which the programs are aligned with government and departmental priorities, roles and responsibilities.

4.1.1 Continued Need

There is a continued need for both programs. However, greater prioritization is needed to address capacity and outreach.

RRAP

Assessment requests from owners and operators are increasing and could soon outpace RRAP’s capacity to meet demand. For the five-year period covered by this evaluation, RRAP had completed a total of 131 assessments or an average of 16 per year.^Footnote2 Site selection to date has been somewhat arbitrary, i.e., primarily based on requests received from CI owners and operators who learned about the program from events and conferences. The program is currently exploring a risk-based approach to prioritizing site assessments. Based on the most recent program estimates of the number of priority CI sites, the total number of sites assessed to date represents about 15% of all CI sites of national interest in Canada. As currently resourced and structured, the program has the capacity to conduct approximately 60-80 assessments per year (vs. estimated 900 sites of national interest).

Evidence from program documentation and interviewees indicates that the RRAP has responded to the initial needs and expectations to address CI vulnerabilities identified in the Beyond the Border Action Plan. Feedback from owner and operator interviewees indicates that site assessments are proving to be useful in helping to identify their CI strengths and weaknesses and in leveraging new CI investments. Although based on a very limited sample, survey feedback also indicates that the program is meeting their needs. All of the respondents indicated that the RRAP assessment products were informative and nearly all said that RRAP guidance is both informative and actionable. Similarly, feedback from other stakeholders, including PS officials interviewed for this evaluation, suggests that information and insights gained from conducting the site assessments are making an important contribution to public knowledge required to promote improved CI resiliency. 

VRAC

The evidence from this evaluation points to a continued need for VRAC to address potential CI vulnerabilities. Evidence of this need is highlighted, in part, by VRAC’s recent role (past two years) in assisting the GOC, which has no resident CI capacity or expertise of its own during emergencies such as the 2016 Fort McMurray Wildfires and 2017 Nova Scotia Snowstorm, as well as through its liaison role on exercises and simulations and its analytical outputs in the form of impact assessments and risk-related products/reports.

Evidence from stakeholder interviews indicates that VRAC involvement and analysis during events such as the Fort McMurray Wildfires is useful and that impact assessments support CI community awareness through communication of event analysis and lessons learned. It was noted, however, that VRAC’s direct support to the GOC in a response coordination capacity began around 2015. This was attributed to an increasing realization among VRAC and GOC management regarding the potential benefits of closer integration. Prior to this time, GOC had relied on the ten CI sector leads to play this role. Interviewees also see VRAC as playing an important liaison and support role in exercises such as Pacific Quake and Staunch Maple. While interviewees generally acknowledge the need for and usefulness of VRAC assessments (e.g., 14 impact assessments in 2016), there is some uncertainty regarding the timeliness and intended audience for these products. 

Interview and document evidence strongly indicate a need for intra and inter-sectoral collaboration and information sharing (e.g., tools and best practices) among CI community stakeholders to foster CI resiliency. Through its management of the CI Gateway (and Global CI Gateway), VRAC has contributed to meeting this need by serving as a repository for CI knowledge and best practices. However, membership and user data indicate that the Gateway is an underused tool that has yet to meet its potential. Many interviewees for this evaluation were unaware of this resource. Portal data indicate that active membership has grown somewhat in recent years from 302 members in 2014 to 406 members in 2016; however, usage remains relatively modest and member growth has occurred mainly from the government sector in Ontario. Many interviewees look to other sources to meet their CI information and collaboration needs. For example, several interviewees referred to the International Association of CI Professionals;^Footnote3 others referred to their sector associations, as well as sector sponsored conferences and networks as sources, many of which engage in timely and easily accessible push notifications to alert them on new items or content.       

4.1.2 Program Evolution to Meet Changing Needs

Both programs were launched in accordance with the Beyond the Border Action Plan in early 2012 with an initial focus on assessing regional cross-border interdependencies. Early regional assessments included assessing risks and vulnerabilities in the New Brunswick - Maine (2011-2013) and BC - Alaska (2013) corridors. Over the evaluation period, which was described by some program interviewees as the pilot or proof-of-concept phase, RRAP and VRAC efforts evolved from the original cross-border focus to a more domestic focus on Canadian CI resiliency. The evaluators were unable to identify a specific trigger or tipping point for this change. Rather, it appears to reflect the implicit assumption that there are important benefits to be gained from building CI resiliency among Canadian owners and operators generally and that providing CI community stakeholders with tools, information and resources to foster resiliency is not only consistent with the intent of the Action Plan, but also good public policy.

Evidence from interviewees and program files also points to at least three other possible explanations for the programs’ evolution: 1) limited capacity to conduct regional cross-border assessments due to the fact that they are resource intensive; 2) increasing demand for assessments from domestic owners and operators as awareness of the programs grew; and 3) greater maturity as CI resilience became increasingly seen by both governments as important national goals. This evolution can be seen as a natural extension of the programs’ original design.

However, the shift to a more domestic focus has not replaced the cross-border approach. Evidence from interviews, documents and files demonstrates that there remains a need for assessing cross-border resilience in line with the expectations of the Beyond the Border Action Plan. It was noted, for example, that a new regional assessment with US counterparts is planned for the Quebec-New England energy sector (electricity grid) and that close collaboration between PS and DHS is ongoing at both the national and regional  (e.g., meetings, conferences) levels.

4.1.3 Impact if Programs were No Longer Provided

The evaluators assessed the potential impact should the programs no longer be provided. Most interviewees stated that the loss of both programs would have various consequences. PS program staff highlighted the view that the department and its CI stakeholders would experience diminished capacity to support the building blocks of the Beyond the Border Action Plan and a coherent national CI-EM Strategy. These interviewees also stressed that discontinuation of the programs would be a blow to PS’ credibility among stakeholders (owners and operators, PTs, municipalities, U.S. counterparts) and would result in a lost opportunity to monitor and track progress on CI resilience (sector comparison index, performance measurement). With respect to VRAC specifically, it was noted that in addition to the loss of CI intelligence and knowledge sharing, there would be a diminished capacity to support GOC with CI expertise during events.

More generally, interviewees and program documentation indicate that in the absence of these programs, vulnerability to risks and disruptions would likely increase. Risk was seen to be associated with loss of awareness of CI supply chain interdependencies (in the absence of compensating programs) and a coherent national vision for bolstering CI resilience.

Despite perceived setbacks, some interviewees suggested that the products, services and functions provided by RRAP and VRAC (e.g., assessments, analysis, and support) could be delivered (in whole or in part) by organizations other than PS, such as the sector lead organizations, PTs, private organizations or by a self-assessment approach. These potential alternative approaches are considered elsewhere in this report (4.3 – Demonstration of Efficiency and Economy).

4.1.4 Alignment with PS Priorities and Government Direction

Key Government of Canada documents, including the 2017 Budget, PS’ Ministerial Mandate Letter (2015), the Beyond the Border Action Plan (2011), among others^Footnote4, cite CI protection and resilience as priorities of past and current governments. These documents refer to the need to safeguard Canadian CI, make progress on critical infrastructure and cyber-security priorities, and work collaboratively on cross-border CI efforts and priorities. 

These documents not only outline the necessity to prioritize and work on CI, but also the importance of partnerships, ways of collaborating (e.g., through information sharing) and enhancing resilience across borders and across CI sectors (interdependencies); the role of VRAC and RRAP in fostering resilience and enabling all stakeholders, including owners and operators; and the need for assessments and knowledge-building.^Footnote5

4.1.5 Alignment with Government Roles and Responsibilities

PS is responsible for national security and for the safety of Canadians; its mandate is to keep Canadians safe from a range of risks, including natural disasters, crime, and terrorism. In this respect, protecting and bolstering the resilience of CI falls under PS’ role and responsibilities, as well as the responsibilities of the federal government as a whole, to ensure safety.^Footnote6

Several documents define federal roles and responsibilities regarding CI, including the National Strategy for Critical Infrastructure (2009), the Action Plan for Critical Infrastructure (2009), Departmental Planning documents, and the Cyber Security Action Plan between PS and DHS. Furthermore, the Ministerial Mandate Letter to the Minister of Public Safety (2015) outlines PS’ role to ensure that Canadians are safe, including protecting against harm to public security (e.g., CI) from various sources.

Although it is the federal government/PS’ role to ensure public safety in Canada, responsibility for CI in Canada is shared by federal and PT governments^Footnote7 and key stakeholders, including local authorities and owners and operators, due to the nature of physical and security-related assets. For instance, a large number of assets are controlled privately, by owners and operators who bear the primary responsibility for protecting them. In this respect, several interviewees suggested that the PTs should be more involved in the delivery of both programs. This idea is further explored in Section 4.3 - Demonstration of Efficiency and Economy.

4.2 Performance - Effectiveness

4.2.1 Achievement of Expected Outcomes

The following section covers the eight outcome themes, which are aligned with the evaluation questions (please refer to Appendix A: Evaluation Framework and Questions).

Understanding of risks and threats

Evidence from this evaluation effort indicates that the RRAP and VRAC have contributed to CI stakeholders’ understanding of risks and threats to their assets and organizations. The three main RRAP assessments tools (CIRT, CIMT, and CCRR) are specifically designed to assess resilience and contain detailed questions aimed at identifying facility deficiencies and vulnerabilities. All owners and operators interviewed for this evaluation that have undergone RRAP assessments and received RRAP reports indicated that the process was useful and contributed to or validated their understanding of risks and threats. As previously noted, this finding is supported by RRAP survey respondents, all of whom indicated that the assessments contributed to or validated their existing knowledge/awareness about their vulnerabilities.

Internal (PS) stakeholders, including members of the GOC, mentioned that VRAC demonstrated that it could provide the CI analytical support function required during events. They also indicated that VRAC supported GOC with analysis of CI supply chain interdependencies, risks, threats, and vulnerabilities. However, interviewees were less certain about the contribution of VRAC products to broader stakeholder understanding of CI risks and threats. Some interviewees found it difficult to assess the contributions of VRAC products (attribution limitation) and questioned whether the VRAC outputs were optimally positioned or communicated to reach target audiences. Those interviewees who were aware of the CI Gateway and who had used it identified it as a source of useful information on CI cases and best practices. However, the evaluators noted that many interviewees were either not aware of the CI Gateway or had not used it and therefore had not had the opportunity to benefit from it. 

Access to timely and actionable cyber and physical risk information

The production of RRAP reports following CI owner/operator site visits is done by RRAP staff in Ottawa. Evidence from interviews and survey data indicates that the reports provide specific and actionable physical risk information to support CI resilience. For example, assessment reports provide various recommendations to owners and operators in a dedicated section of the RRAP final report. Recommendations can cover such things as deficiencies in site perimeter security (e.g., fencing), in closed circuit television (CCTV) monitoring, or in guard facilities.  Survey data indicate that recommended options presented in RRAP reports were, for the most part, relevant (69%) and actionable (89%).

With respect to the turnaround time for the production and delivery of RRAP reports, data for 2016-2017, the most recent period for which data are available, indicate that average report production turnaround time for the preliminary report is 75 days. This compares to a service standard of six (6) weeks, although this is an internally published standard only. RRAP managers pointed out that timeliness is often affected by delays in obtaining follow-up clarifications from the site staff. They also noted that production turnaround times have improved since the program’s inception and that they expect to meet the service standard by fiscal year 2018-19 or earlier.

VRAC analysis and support is situationally dependent and much more time sensitive during an event (as events occur 24/7 in real time) than during a steady state situation. Through its integration with the GOC during events, VRAC is positioned to provide timely CI analysis, including geospatial mapping of CI infrastructure and impact assessments, to responders as situations unfold. VRAC supply chain modeling, analytical reports and other CI risk-related products are less time sensitive. These reports and products, as well as those developed by other departments/agencies (e.g., suspicious incident and threat reports), are shared with stakeholders and, if unclassified, can normally be accessed via the CI Gateway. Currently, there are more than 600 different reports published on the CI Gateway (e.g., impact assessments, vulnerability assessments, threat assessments, tools, strategies, situation reports, reference guides, plans, daily operations briefs). While a potentially timely and actionable repository of physical risk information, input from interviewees as well as usage data suggests that the CI Gateway is an underused resource that could benefit from greater promotion. Some interviewees suggested that push notifications, news feeds and alerts could be used to improve both the awareness and the timeliness of information on the Gateway. 

Owners and operators take action to manage risks as suggested by programs

The evaluation examined evidence regarding the extent to which owners and operators, as well as other stakeholders, acted to mitigate CI risks and vulnerabilities. Most owner and operator interviewees stated that, following their RRAP site assessments, they took at least some action, typically in the form of investments to address deficiencies (e.g., site security, cameras, bollards, fencing etc.). Of those who provided estimates, the dollar amounts ranged from a low of $10,000 to a high of $10 million. Survey data, though limited, tends to reinforce the finding that RRAP assessments led to new investments. Among those who responded, investments amounted to $7 million or an average of $600,000 per site assessment, with a median investment figure of $116,000. At an average cost of $9,228 per assessment, and using the median investment value, this yields an estimated return on investment (ROI) ratio of 13:1. 

Notably, owners and operators mentioned that, by independently identifying CI weaknesses and vulnerabilities, the RRAP assessment reports helped them make the business case for new investments. While these are positive indications of program performance, the Findings are not conclusive and should be viewed with caution due to the limited sample size. More systematic follow-up by RRAP regarding action taken following site assessments would contribute to a more conclusive assessment regarding this performance metric.

VRAC does not deal directly with owners and operators. However, it does deal directly with the GOC and other government departments and agencies, coordinates with RRAP, and contributes to EM exercises. Other VRAC outputs take the form of stakeholder collaboration, analysis and support. Those interviewees who are aware of VRAC and who deal with the program directly speak favourably of its contributions. However, some feel that VRAC’s efforts and outputs need to be better positioned and focused to determine where it can provide the most value. The evaluation found that, at this time, there is insufficient data to determine the extent to which stakeholder action or decision-making was influenced as a direct result of VRAC.

Programs contribute to the owner and operators’ and the Government of Canada’s capacities to plan and respond to disruptions

Evidence from interviews indicates that information obtained by stakeholders from both RRAP and VRAC outputs contribute to PS’ understanding of risks and criticalities, while helping to support CI policy and decision making processes.

With respect to RRAP, interviewees affirmed that the program helps to identify risks and vulnerabilities through the assessments and that it contributes to owner/operator risk mitigation and business continuity planning. However, RRAP site selection during this initial phase has been, for the most part, reactive and demand-driven (i.e., responding to those who learn about the program through various fora rather than strategically planned and selected by RRAP based on defined CI prioritization criteria). The exception to this approach has been with the regional cross-border initiatives (NB-Maine, BC-Alaska) where significant planning and coordination was required.  

As has been noted elsewhere in this report, VRAC’s contributions to the Government of Canada’s capacities to plan and respond to disruptions are evident from its support to GOC during events, from its contributions to steady-state planning and preparation (e.g., through exercises), from its contribution via the National Risk Profile to identify risks on a national scale, and from its analysis including geo-mapping and supply chain analysis (e.g., fuel distribution model). 

CI community collaboration (sharing of information and best practices) and partnerships

In addition to their primary objective of supporting CI risk management, the RRAP and VRAC were meant to contribute to CI resilience through enhanced CI community collaboration (e.g., sharing of information and best practices) and partnerships. Evidence from documentation and interviews indicates that cross-sectoral collaboration took place in the form of various fora. For example, CISCD Performance Measurement Reports indicate that Canada-U.S. CI Working Group meetings focusing on the launch of RRAP began in 2011-2012 and were followed by ongoing bilateral meetings regarding RRAP and VRAC implementation in 2012-2013.

Evidence of additional collaborative activity in the form of stakeholder presentations, briefings, meetings and workshops is demonstrated in the various performance reports. For example, briefings to the provinces of Ontario, Alberta, BC and Yukon in 2013-2014 were reported as well as presentations to sector networks and the National Cross Sector Forum (June 27, 2013). That same year, RRAP and VRAC were also presented at three workshops hosted by the Conference Board of Canada. These selected examples of collaborative outreach gleaned from a review of several sources are representative of the broader scope of activity conducted by the programs operating within the PS CI context. In the absence of a defined outreach strategy specifically for RRAP and VRAC (as opposed to CI generally), it is not possible to determine conclusively the extent to which community collaboration objectives have been met.     

Several interviewees consider that both programs hold a significant role in developing and sharing best practices, with VRAC (through the CI Gateway) having a more visible role in this respect. The CI Gateway has also served as a CI community collaboration tool as well as a portal and repository for information access and best practice.

Progress has been made consistent with the Programs’ inaugural phase. However, activity data indicate that program reach and coverage have been geographically skewed to areas and regions where resources have been most prevalent and that more balanced coverage is required to reflect an appropriate distribution of effort based on risk criteria (for further detail, please refer to Tables 5 and 6). 

In the same vein, respondents stressed the importance of various networks (e.g., community, sector or industry led) to forge partnerships and to share best practices. It is worth noting that network activity is similarly resource dependent. For example, achievements of the Critical Infrastructure Advisory Network, one of the more mature CI networks in Canada, are largely due to the collaborative initiative of its local resources. By contrast, CI network collaboration in other regions and provinces is almost non-existent.

As noted above, with respect to VRAC, the CI Gateway^Footnote8 is an important collaborative tool. The CI Gateway repository currently contains more than 600 documents including reports (e.g., infrastructure, exercise, cyber security, situation), frameworks (e.g., critical five, cyber incident, CI info sharing), support materials (e.g., CI planning guide, fact sheets, best practice guides), geospatial material (e.g., sector maps), guides (e.g., cyber security, risk assessment methods, reference guides), records (e.g., meeting and conference minutes, briefing and consultation materials), and threat and impact assessments (e.g., Fort McMurray, Nova Scotia Drought, New Brunswick Winter Storm, Saint-Luc-de-Vincenne Landslide). The CI Gateway portal also contains downloadable self-serve resource (toolbox) material and archived records of 20 cross-fora CI sector meetings, workshops and networking events.

Data indicate that this tool is used disproportionately by members from Ontario, with more than three-quarters of all hits originating from Ottawa. Many interviewees use sector-specific sources for their CI information and suggested that the CI Gateway might benefit from more content targeted to the specific interests of the 10 sectors. Membership has increased from 333 in 2014 to 546 in 2016. In the same time period, active members increased from 302 to 406.  

Enhancing understanding of cross-sector dependencies and linkages

Both RRAP and VRAC have played a role in enhancing PS and stakeholder understanding of cross-sector dependencies and linkages. Evidence of cross-sector initiatives exists in the form of assessments, exercises, networks, conferences and other fora. 

Regional assessments were part of the originating vision for RRAP. To date, two have been completed (NB-Maine and Alaska-Yukon-BC) and a third is under consideration (Quebec-New England). However, experience has proven that while key to highlighting cross-dependencies, these cross-border assessments are time and resource intensive to conduct, a factor which, given limited capacity, may have accounted for a shift in program focus towards domestic CI. As noted earlier, this shift can be seen as a natural extension of the programs’ originating mandate given the growing concerns in both countries regarding the significant impacts of domestic disruptions to CI. 

Through VRAC’s support to exercises and its analytical outputs, it has played a role in enhancing understanding of sector interdependencies. Because it is not site specific, VRAC has the ability to examine and capture CI linkages and supply chain dependencies that may not be immediately evident from a site-specific perspective.  VRAC’s CI Gateway links with the CI Global Gateway; impact assessments and analytical reports (Soo Locks Report), exercises (Pacific Quake and Staunch Maple tabletop exercises), and a compendium of the cyber dependencies of each critical Canadian CI asset are examples. However, VRAC has been constrained by limited resources and outreach. Interviewees familiar with VRAC suggest that it requires greater focus for its full potential to be realized.

Programs’ contribution to mitigation of impacts from disruptions

At a broad level, the RRAP and VRAC are expected to contribute to the mitigation of impacts from disruptions (all hazards approach) to CI. Interviewees indicated that both programs do contribute to a certain extent to the mitigation of impacts from disruption, despite their limited capacity and reach. As previously noted, however, program coverage is unbalanced and localized to areas where experienced resources (i.e., RRAP assessor teams) are in place. CI protective measures and resiliency scores/rankings are a means by which sites can be compared to other sites within a sector. These comparability scores are seen by owners and operators as useful measures to assess their sector ranking with respect to CI. However, the Canadian database currently suffers from a lack of suitable comparability data (131 sites in total) overall and within each sector. Uneven assessment coverage (regionally and across sectors) contributes to this limitation. To date, U.S. comparative data has been used when presenting owners and operators with their RRAP results. However, some owners and operators interviewed for this evaluation point out that this data is not always directly comparable due to its emphasis on different variables. 

Data on the distribution of RRAP assessments versus the current total of CI sites of national importance by province/territory are shown in Table 5. Notably, this table indicates a significant regional imbalance vis-à-vis number of assessments conducted in each PT versus number of nationally significant CI sites identified by VRAC. To illustrate an example of uneven coverage, of the 131 RRAP site assessments conducted to date, 34% have occurred in the Atlantic Region and 25% in Saskatchewan; RRAP has highly experienced assessors located in both regions. By comparison, program data indicate that the Atlantic Region accounts for 17% of total CI sites of national importance and Saskatchewan accounts for 6%. However, the RRAP reports that it is in the process of developing a criticality exposure index to enable a more strategic, risk-based site selection approach.

Similarly, Table 6 below shows data on the distribution of RRAP assessments versus the distribution of total CI sites of national importance by sector, as recently identified by PS VRAC staff. As with Table 6 above, Table 7 indicates a significant sectoral imbalance vis-à-vis number of assessments conducted versus number of CI sites identified for each sector. For instance, 18 event facility assessments (14%) have been conducted; although event facilities (e.g., Grey Cup facility in Toronto) are not categorized as CI sites of national importance (please refer to Table 1 in section 2 of this report).^Footnote9  

Programs’ contribution to making Canada’s CI secure and resilient

The evidence from this evaluation supports the contention that the Programs contribute to Canada’s CI security and resiliency. This is evident through program activities and outputs, new investments by owners and operators, and by CI community collaboration, information sharing, and best practices. Many interviewees stated that RRAP assessments helped them to make a business case for new CI investments or prioritization. In one specific example, the operator estimated that the RRAP assessment helped define an initial short-term investment of approximately $50K and a similar longer-term investment. These investments helped to improve the resilience of a single water treatment facility servicing two large communities with a combined population of more than a quarter of a million residents.

The three RRAP assessment tools are unique in their ability to support CI interventions. There is no known existing private market equivalent and no other tool or approach capable of providing resilience scores, rankings or indexing for purposes of comparability among owner and operator facilities, sectors or jurisdictions. Index data generated by these unique tools indicate that the U.S. average resilience score is slightly higher than in Canada; however, some U.S. criteria contained in the scoring grid do not apply in Canada (e.g., armed guards), which could account for the lower score. DHS and its partners assessed more than 6,000 sites versus 131 for Canada. The data on national and sectoral resilience are important quantitative indicators that enable both government and CI owners and operators to assess, respond to and monitor the state of Canadian CI security and resilience.

4.3 Performance - Efficiency and Economy

Since inception, the RRAP and VRAC have operated on a relatively modest budget as components of a portfolio of CI resilience initiatives. Investments in these initiatives amounted to just over $5 million for the period of 2012-2013 through 2016-2017. Additional funding of $1.37 million for 2017-2018 was announced in the March 2017 budget for the continuance of RRAP and VRAC operations. Over their formative period, both programs have established a baseline of operations. For the most recent year (2016-2017), RRAP assessment costs totaled $710,531 with cost per assessment averaging a little over $9,000.

4.3.1 Steps Taken to Optimize Resources

The RRAP and VRAC have complementary, mutually-reinforcing goals. However, evidence from some interviewees suggests that the two programs had not always coordinated or integrated their efforts to their potential (i.e., operating in silos). Recent changes have been aimed at addressing this issue.    

The RRAP and VRAC have exceeded their budget in each of the five years covered by this evaluation (see Table 2). This can be attributed largely to greater than expected demand and increased workload as the programs evolved from their original cross-border focus to a more domestic focus, as well as to the enhanced cyber resiliency component of the programs. The programs currently operate with a total of 20 FTEs (18 currently staffed), a significant increase over the eight resources originally planned. To reduce travel costs and enhance capacity, the RRAP has begun using regional resources to support assessments. The program has also benefitted from a seconded RCMP resource.

Due to the specialized nature of CI assessment work, both programs require experienced resources to function efficiently. In the case of the RRAP, the program has benefitted from a small cadre of highly trained and experienced individuals with the result that program performance tends to be highest in areas where these resources are located. A more systematic approach to resource (assessor) recruitment, training, and deployment could help to address this imbalance. 

Requests for RRAP assessments have so far not been subjected to rigorous risk-based criteria. This demand-driven approach has been acceptable so long as the program was looking to gain experience and establish itself. However, as awareness of and demand for assessments increase and as the program evolves to its next phase, resource optimization will require greater scrutiny of site assessment requests.

With respect to VRAC, the program has benefited from collaboration with other CI and EM sector partners and agencies, including the GOC. This collaboration is required in part due to the analytical nature of the work it performs, its liaison role in exercises and event support, and through the custodianship of the CI Gateway. As an information provider and CI enabler with limited resources, it is important that VRAC’s outputs (products and services) be carefully targeted to the needs of its audiences. Evidence from some interviewees suggests that VRAC could do more to ensure that optimal value is being derived from its outputs and that VRAC might achieve economy of scale by promoting a more broad-based approach to CI information sharing (e.g., by building on its existing CI Gateway membership community).

4.3.2 Alternative Delivery Approaches

The assessment of alternative service delivery options was meant to complement and support the evaluation of the RRAP and VRAC to determine the extent to which the current delivery model is the most optimal and if there are other delivery models that would provide the Government of Canada with better value-for-money.  

Several themes emerged from this analysis, especially so with respect to the advantages of the current approach, existing and potential obstacles, and the role and capacity of PS to deliver the RRAP and VRAC. Advantages of the current approach include PS’ national leadership role with respect to CI; its holistic perspective regarding supply chain interdependencies, resilience, and vulnerabilities; opportunities for building new and fostering existing relationships with the CI community and in particular with owners and operators; the CI knowledge and information garnered through VRAC and RRAP work; and the credibility and expertise of RRAP and VRAC professionals. There are also opportunities for PS to work more closely with other jurisdictions as well as with CI sector leads and experts to arrive at a more holistic and strategic coverage of CI assessments and analysis, as well as to leverage knowledge from a variety of centres of expertise.

In contrast, certain needs and challenges are evident with the existing service delivery model, including limited resource capacity (human, financial, and technology-related) to meet the potential demand for assessments; a need for greater clarification with respect to VRAC’s role and collaboration as a partner in emergency management; a need for prioritization and strategic selection of CI sites; and a need for enhancing collaboration and opportunities for working more closely with other jurisdictions/regions. 

Table 7: Assessment of Alternative Service Delivery Options presents the ranking of each of the seven proposed options based on key benefit and risk scores.^Footnote10 Of the benefits assessed for each option, four (4) were deemed critical for PS:

1. Maximizing the reach of programs;
2. Access to expertise and information from multiple sources;
3. Access to; and
4. Facilitating access to the existing methodology.

These benefits have been weighted accordingly in the analysis. Additionally, loss of trust from owners and operators has been weighted to reflect its importance as a risk factor. The full assessment table is provided in Appendix B.

The alternative model that would provide the most additional benefits is an increased partnership with other levels of government, with PS maintaining the national leadership role. The status quo could also increase its benefits by addressing some of the challenges identified.

5. Conclusions

5.1 Relevance

The underlying objectives of both RRAP and VRAC remain relevant today. They have responded to the initial needs and expectations to address CI vulnerabilities by assisting owners, operators, and stakeholders involved in ten critical infrastructure sectors. RRAP site assessments are proving to be useful in helping to identify CI strengths and weaknesses. RRAP assessments are leveraging new investments in CI by owners and operators; evidence suggests that these investments could be even greater given additional incentives (e.g., in the form of grants and contributions, tax credits). Regarding VRAC, the continued need is underlined by its analytical outputs in the form of impact assessments and action reports, its support during events such as the Fort McMurray Wildfires, as well as through its liaison role on exercises and simulations.

Both programs are key components of the federal government’s CI agenda and are identified as a priority in multiple fundamental government documents including the originating 2011 Canada-U.S. Beyond the Border declaration. However, although CI is a key federal government role and RRAP and VRAC contribute to the role by supporting the objectives the National Strategy for Critical Infrastructure, this responsibility is shared with numerous stakeholders, such as provincial/territorial governments, local authorities, and owners and operators, due to the interdependent nature and properties of physical and security-related assets.

5.2 Performance – Effectiveness

With respect to performance, both programs have contributed to CI stakeholders’ understanding of risks and threats to their assets and organizations. Interviewees stated that RRAP reports provide specific and actionable all-hazards risk information to support CI resiliency. Owners and operators stated that they took tangible action to mitigate the key risks identified during their site assessment, typically in the form of business continuity planning and investments to address deficiencies. However, faced with increasing demand for site assessments, RRAP will need to take steps to prioritize its efforts. Greater prioritization is particularly important given the evaluation finding that site selection in RRAP’s initial phase was demand-driven and concentrated in areas where specialized PS resources were located to promote the program, resulting in regional and sector imbalances in coverage. 

Regarding VRAC, interviewees indicated that this program has facilitated the GOC and stakeholder understanding of CI supply chain interdependencies, risks and threats during events and in exercises. VRAC is positioned to provide timely CI analysis, including impact assessments, supply chain modeling and risk-related products. However, the CI Gateway is an underused resource that could benefit from greater promotion.

In terms of CI community collaboration and partnerships, both RRAP and VRAC play a significant role in developing and sharing best practices, with VRAC having a more visible role in this regard, in large part due to its management of the CI Gateway. Both programs have contributed to enhancing PS and stakeholder understanding of cross-sector dependencies and linkages. Evidence of cross-sector initiatives exists in the form of assessments, exercises, networks, conferences and other fora. With respect to VRAC, while the evidence from interviews and documentation (e.g., CI Performance Reports) indicates that many collaboration and outreach activities have occurred over the course of the five-year period covered by this evaluation, the attribution of impacts and results (i.e., the extent to which these activities have contributed to mitigate impacts from CI disruptions) is challenging, especially given the absence of a specific and defined collaboration and outreach strategy.

5.3 Performance - Efficiency and Economy

With respect to efficiency and economy, the RRAP and VRAC have operated on a relatively modest budget as components of a portfolio of CI resilience initiatives. Program expenditures have exceeded the forecasted budget in each of the five years of operation due primarily to the additional resources needed to meet demand and workload. While the number of program FTEs has increased, results have been overly dependent on a few key experts, which imposes potential limits on program scalability.

Nevertheless, recent governance changes have reinforced coordination and resource sharing between the two programs. These are important steps towards optimizing the use of resources especially considering the potential benefits to be gained from closer integration of efforts between the programs. Furthermore, evidence from the examination of alternative service delivery approaches suggests that, except for closer collaboration and partnership with PTs, there is little to be gained and likely more to lose by changing the service delivery model at this time.

In sum, these two programs do not constitute the sole solution to mitigate impacts from CI disruptions, but they remain relevant, have performed effectively and efficiently in their inaugural phase, and have proven to be an important complement to efforts of other CI and EM programs, initiatives and assets, whether they be at the national, provincial or international level.

6. Recommendations

The following recommendations are being provided in the spirit of continuous improvement.

The Senior ADM of the National and Cyber Security Branch should consider:

1. Developing RRAP site assessment selection processes, and VRAC products, that consider risks and priorities.
2. Leveraging CI community engagement and targeted outreach activities to support achievement of RRAP and VRAC program objectives.
3. Ensuring appropriate resources to support the scope of activities outlined in annual RRAP and VRAC workplans.
4. Exploring options to support owners and operators to address improvements identified through site assessments that will increase the resilience of CI sites across Canada.

7. Management Response and Action Plan

*Date dependent on when policy approval and renewed funding for these programs is received.

Appendix A: Evaluation Questions

Evaluation Questions

Relevance

1. Need for RRAP and VRAC
1. Is there a continued need for these programs?
2. What would be the impact if the RRAP and VRAC were no longer provided?
2. Alignment with Government Priorities
1. Are the RRAP and VRAC aligned with Government of Canada priorities?
3. Alignment with Government roles and responsibilities
1. Are the RRAP and VRAC aligned with Government of Canada roles and responsibilities, i.e., is delivering these programs consistent with the roles and responsibilities of federal government?
2. Could this role be better served in partnership with or by another level of government or a different organization?

Performance—Effectiveness 

1. Achievement of expected outcomes
1. To what extent have the programs contributed to CI stakeholders having an under-standing of risks and threats to their assets and organizations? To what extent have the programs contributed to CI owners and operators having access to timely and actionable cyber and physical risk information?
2. To what extent have CI owners and operators taken action to manage risks as identified/suggested by the programs?
3. To what extent do the programs contribute to the owners and operators and the Government of Canada’s capacities to plan and respond to disruptions? To what extent do the programs contribute to the CI community collaboration (sharing of information and best practices) and partnerships?
4. To what extent do the programs contribute to enhancing understanding of cross-sector dependencies and linkages? To what extent have these programs contributed to the mitigation of impacts from disruptions?
5. To what extent have these programs contributed to making Canada’s CI Secure and Resilient?

Performance — Efficiency and Economy

1. Demonstration of efficiency
1. What steps have the programs taken in order to optimize the use of resources in the achievement of results?
2. Could these programs be delivered differently to achieve a better reach and/or better value-for-money?

Appendix B: Alternative Service Delivery Options

Appendix C: Bibliography

Department of Homeland Security (2016). Regional Resiliency Assessment Program.
Dziadyk, W. (BD Pro Inc.) (2011). Harmonized TRA (HTRA) Methodology – Limitations.
Government of Canada (2013). 2012-13 Report on the Beyond the Border Action Plan Horizontal Initiative.
Government of Canada (2014). 2013-14 Report on the Beyond the Border Action Plan Horizontal Initiative.
Government of Canada (2015). 2014-15 Report on the Beyond the Border Action Plan Horizontal Initiative.
Government of Canada (2017). Canadian Critical Infrastructure Information Gateway.
Hardenbrook, B. J. (2005). The Need for a Policy Framework to Develop Disaster Resilient Regions. Journal of Homeland Security and Emergency Management, 2(3), 1-23.
House of Commons (2017). Building a Strong Middle Class - #Budget2017.
Leuprecht, C. et Hataley, T. (2013). Sûreté, sécurité civile et mesures d’urgence au sein du système canadien de gouvernance multiniveau. Télescope : Revue d’analyse comparée en administration publique 19(1), 176-193
Petit, F.D. et al. (Argonne National Laboratory) (2013). Protective Measures Index and Vulnerability Index: Indicators of Critical Infrastructure Protection and Vulnerability.
Petit, F.D. et al. (Argonne National Laboratory) (2013). Resilience Measurement Index: An Indicator of Critical Infrastructure Resilience.
Public Safety Canada (2008). Canada’s National Disaster Mitigation Strategy.
Public Safety Canada (2009). National Strategy for Critical Infrastructure.
Public Safety Canada (2009). Regional Resilience Assessment Program (RRAP) Overview.
Public Safety Canada (2009). Regional Resilience Assessment Program and Critical Infrastructure Assessment Tools.
Public Safety Canada (2011). CISCD - Performance Measurement Strategy, December 12, 2011.
Public Safety Canada (2012). Critical Infrastructure Information Gateway Membership Application Form.
Public Safety Canada (2012). Canada and United States Release Joint Plan for Emergency Border Traffic Management, News Releases 2012.
Public Safety Canada (2012). Canadian Critical Infrastructure Information Gateway – Terms and Conditions of Service.
Public Safety Canada (2012). Considerations for United States – Canada Border Traffic Disruption Management.
Public Safety Canada (2012). Regional Resilience Assessment Program and Critical Infrastructure Assessments Tools.
Public Safety Canada (2013). A Guideline for Enhancing Canada's Critical Infrastructure Resilience to a Catastrophic Earthquake.
Public Safety Canada (2013). CISCD - Annual Performance Report 2012-13.
Public Safety Canada (2013). CISCD - Performance Measurement Strategy 2013-14.
Public Safety Canada (2014). Action Plan for Critical Infrastructure.
Public Safety Canada (2014). CISCD - Performance Measurement Strategy 2014-15.
Public Safety Canada (2014). CISCD Annual Performance Report 2013-14.
Public Safety Canada (2015). Beyond the Border: A Shared Vision for Perimeter Security and Economic Competitiveness.
Public Safety Canada (2015). Critical Infrastructure – 360 Report (Issue 1).
Public Safety Canada (2015). Cybersecurity Action Plan Between Public Safety Canada and the Department of Homeland Security. 
Public Safety Canada (2016). 2016-2017 Evaluation of the Disaster Financial Assistance Arrangements.
Public Safety Canada (2016). CISCD Performance Measurement Strategy 2016-17.
Public Safety Canada (2016). CISCD PMS Data Collection.
Public Safety Canada (2016). Critical Infrastructure Impact Assessment: Fort McMurray Wildfires, May 8, 2016, 4:00PM.
Public Safety Canada (2016). Critical Infrastructure Impact Assessment - Fort McMurray Wildfires.
Public Safety Canada (2016). Critical Infrastructure Impact Assessment: Fort McMurray Wildfires, May 8, 2016, 4:00pm.
Public Safety Canada (2016). Critical Infrastructure Impact Assessment: Fort McMurray Wildfires, May 8, 2016, 4:00pm – Discussion Document.
Public Safety Canada (2016). Critical Infrastructure Resilience: Modernizing the RRAP and VRAC programs.
Public Safety Canada (2016). Executive Dashboard – Canadian Critical Infrastructure Information Gateway.
Public Safety Canada (2016). Regional Resilience Assessment Program – Survey Results – December 2016.
Public Safety Canada (2016). Regional Resilience Assessment Program Strategic Vision.
Public Safety Canada (2016). RRAP - Assessment Process Evaluation Survey.
Public Safety Canada (2016). RRAP - Final Report Evaluation (Survey).
Public Safety Canada (2016). RRAP - One-Year Follow Up (Survey).
Public Safety Canada (2016). RRAP Survey Results Analysis.
Public Safety Canada (2016). Standing Senate Committee on National Security and Defence.
Public Safety Canada (2016). Virtual Risk Analysis Cell (VRAC) - Overview.
Public Safety Canada (2016). Virtual Risk Analysis Cell.
Public Safety Canada (2016). VRAC 48 Hour Fire Growth Model for Fort McMurray.
Public Safety Canada (2016). VRAC Map - 8 Hour Fire Growth (Fort McMurray).
Public Safety Canada (2017). Beyond the Border Action Plan.
Public Safety Canada (2017). Current CI Totals.
Public Safety Canada (2017). Direct Program Costs.
Public Safety Canada (2017). Overview of Canadian Critical Infrastructure Sectors.
Public Safety Canada (2017). Preliminary Report Turnaround Time, 2016-2017.
Public Safety Canada (2017). Regional Resilience Assessment Program (RRAP) and Virtual Risk Analysis Cell (VRAC): Results.
Public Safety Canada (2017). Reports for CI Gateway (Quarterly).
Public Safety Canada (2017). RRAP and VRAC Governance Structure and Budget and Expenditures 2012-2017.
Public Safety Canada (2017). RRAP Team Performance Survey Results.
Public Safety Canada (2017). Summary Table by Estimates Vote Structure.
Public Safety Canada (2017). The Regional Resilience Assessment Program. Retrieved from: https://www.publicsafety.gc.ca/cnt/ntnl-scrt/crtcl-nfrstrctr/crtcl-nfrstrtr-rrap-en.aspx.
Public Safety Canada and Department of Homeland Security (2010). Canada-United States Action Plan for Critical Infrastructure.
Quigley, K. (2013). “Man Plans, God Laughs”: Canada’s National Strategy for Protecting Critical Infrastructure. Canadian Public Administration, 56(1). 142-164.

Date modified:

2022-07-21