Canadian Police Board Views on the Use of Police Performance Metrics

Canadian Police Board Views on the Use of Police Performance Metrics PDF Version (1,775 KB)

John Kiedrowski
Michael Petrunik
Todd Macdonald
Ron Melchers

Report No. 31, 2013
ISBN No.: 978-1-100-21885-4

Table of contents

• Executive Summary
• Introduction
• Analytic Approach
  • Content Analysis
  • Interviews
• Overview of Performance Measures
• Police Performance
• Statutory Requirements for Police Performance
• Overview of Police Oversight Boards
• Content Analysis
  • Findings from the Content Analysis
    • Operational versus Strategic Performance Measurements
    • Performance Indicator Numerator and Denominator Statements
    • A Critique of the Police Performance Measures Reviewed
• Interviews
  • Findings from the Interviews
    • General Information on Police Performance
    • Overview of Performance Metrics
    • Board Members' Involvement with Performance Metrics
    • Interviews with Respondents from Communities Not Using Performance Metrics
    • Interviews with Government Officials and Stakeholders
• Conclusions
• Bibliography
• Appendix A: Additional Tables
• Appendix B: Methodological Approach

List of Tables and Figure

• Table 1: Definitions of Key Terms used in Performance Measurement Models
• Table 2: Overview of MPMP Requirements for Police Services
• Table 3: Overview of OMBI Performance Measures for Police Services
• Table 4: Frequencies of Performance Dimensions
• Table 5: The Seven Dimensions of a Balanced Scorecard and Selected Key Performance Indicators
• Table 6: Overview of Selected Performance Indicators in Relation to a Law Enforcement Scorecard Framework
• Table 7: Oveview of Performance Measurements Goals that were Operational and Strategic
• Table 8: Selected Performance Measures Used by Police Servicesas Indicated in the Content Analysis
• Table 9: Overview of Performance Models and Relevance of Performance Indicators
• Figure 1: Evolution of Police Performance Metrics

Executive Summary

Background

Cost-of-service trends are a major concern for the Canadian policing community and the public sector in general. In calendar year 2011 alone, total operating expenditures for Canadian local policing was roughly $12.9 billion. In response, innovative Canadian police services have begun to adopt a new management paradigm focusing on the use of performance indicators to help to: 1) rethink what services are of highest priority; 2) generate improved outcomes; and 3) better control service delivery costs.

Methodology

This paper set out to provide an evidence-based snapshot of publicly-available police performance metrics and how Canadian civilian police boards deal interpret and apply them. A content analysis of documents collected from police service websites or provided by police boards and structured interviews with police board members across Canada were carried out to assess the extent to which, and the ways in which, police boards use police performance measurement tools.

Findings

Governmental Standards

In Canada, there are no legislative requirements that control or dictate the use of specific performance measures or standards in policing. Some provinces, such as Ontario, do require that police report on specific indicators of performance under municipal reporting regulations. Even scanning internationally, only a handful of American police services apply International Organization for Standardization (ISO) standards to their management of organizational performance. Overall, there are no evidence-based performance standards or existing "industry accepted" measurement frameworks available for police services to follow in Canada.

Police Services

The overall assessment of the selection of police performance metrics outlined by police services reveals that the police services that have the most advanced performance measurement frameworks are generally larger, urban or regional policing services. This study defined performance measurement frameworks as being "balanced" when they were characterized by being comprehensive and sophisticated. Those few police services meeting the criteria for a "balanced" performance framework tended to apply their indicators in a relevant manner. The seven dimensions of a "balanced" framework included tracking performance metrics in the following categories: 1) reduce criminal victimization; 2) call adult and youth offenders to account in appropriate ways; 3) reduce fear of crime and enhance personal security; 4) increase safety in public spaces; 5) use financial resources fairly, efficiently, and effectively; 6) use of force and authority legitimately, fairly, and effectively; and 7) satisfy citizen demands for prompt, effective and fair service.

The research also showed that some individual police services are developing their own performance metrics and business planning structures; however, they are often based on ad hoc peer-to-peer consultations within the policing community. While such peer consultation is useful, adapting new indicators into a coherent performance metrics framework can be a challenge. Although there is some sharing of information on performance metrics between police services, and this sharing is laudable, some of the information on indicator design and selection being shared can be inaccurate. Too often police services have set up their performance metrics in a catch-all fashion trying to measure "everything and anything" related to police work. In fact, much of what is put forward under the rubric of "performance measurement" is more in the nature of broad environmental scanning indicators than those related to effectiveness or efficiency of police services. It is suggested that measurement frameworks consequently require a clearer focus on core policing outcomes.

It is important to note that those police services that scored in the low relevance category in their use of performance metrics were not clear with regard to what was being measured. Further, whole categories of important performance indicators were often absent, most significantly indicators that measured effectiveness, efficiency and quality of service.

Performance Measures

Most police services not using performance metrics or lacking an effective measurement capacity were from smaller municipalities and rural areas. However, there were also a few examples from large urban jurisdictions of police services that demonstrated little capacity to apply performance metrics. In both instances, there were no meaningful regional variations in terms of the size of the police services. Four of the five police services considered to have the best developed performance measurement frameworks represented medium-sized jurisdictions; the fifth was a large urban police service.

The content analysis of police performance metrics indicates that there was no common model or framework for measuring the performance of police organizations. Among the police services studied, each emphasized different dimensions of service delivery performance. A troubling finding was that key dimensions of performance such as effectiveness and efficiency, which are essential to any well-designed performance management model, were rarely applied. When valid and reliable quantitative performance metrics were included they were most often operational indictors. The validity of many of the measurement outputs and outcomes of police performance were difficult to assess in the content analysis, because police publications were unclear as to what type of data were being measured in many of the output, outcome or result metrics.

On the other hand, a positive observation was that many police services used comprehensive methodologies to determine community policing priorities. These priorities were used to develop aspects of most performance measurement frameworks. Common methods for prioritizing policing needs included conducting environmental scans, conducting SWOT analyses (Strengths, Weaknesses, Opportunities, and Threats) and applying SMART criteria (Specific, Measureable, Achievable, Relevant, Timely).

Of the performance measures that were reported by police services many could be classified as being strategic in some way. In fact, many police services do appear to be attempting to focus more on measuring progress towards strategic goals than measurement of operational, tactical objectives.

In some instance, as police improve their performance measurement frameworks, there is an opportunity for tactical or operational performance metrics to be converted into more sophisticated, strategic performance metrics by adding another dimension to the measure, such as reporting the business activity as a rate or proportion of a another relevant measure.

The analysis documented that the use of police performance measures is evolving in Canada. At the bottom of the "evolution curve" there is a cluster of police services that have no performance measures in place, while further along the curve some services have implemented rudimentary frameworks, and at the top of the curve a few services are making good progress in developing frameworks that meet appropriate standards. Police services with less experience in the development of performance measures tend to have a large gap between those measurement tools considered to be ideal and the ones that are actually used.

A number of police services appear to track a significant amount of performance data and create performance ratios, but it was unclear as to what, if any, analysis is used to determine whether goals are being achieved. Where ratios or descriptive data were presented, further data analysis and explanation appears to be required to tie the measures to organizational goals. A relatively small portion of police services had a well-designed portfolio of performance indicators that shed light on core policing outcomes, service delivery productivity and overall value. Not surprisingly, where they were present, these indicators were used in a relevant fashion, extending well beyond simple public reporting, in the police service's annual reports.

Police Boards

While many of the boards representing larger urban police services have full- or part-time professional staff, not all the large urban services in the sample had well-developed performance measurement expertise. A lack of understanding of indicators or the structure of decision making processes means that performance metrics often have little or no impact on police board decision making. However, it is evident that members of boards representing larger policing services, with more board members and benefit from full- or part-time professional staff to provide research support, appear to be more aware of the uses and limitations of various performance measurement tools.

The role of some police service boards, and their rapport with police chiefs, does not allow their review to directly lead to reallocations within police budgets or between municipal services. However, the application of more sophisticated police performance metrics could provide useful information to help direct and allocate costs once a budget has been completed and implemented.

The research found that police service board members had no specific training on police performance measurement. However, it is important to note that the individual board members who know the most about police performance measurement tools usually had the benefit of serving more than one term. Moreover, it is notable that the use of performance information is nonetheless becoming an important part of the planning and budgeting process in many jurisdictions. As such, the lack of adequate training can leave board members vulnerable, especially when they are expected to make decisions based on information they do not feel they know how to assess. Although the level of knowledge and understanding of performance measurement is low amongst most members of police service boards, once information is provided on performance metrics, and it is explained how they can be incorporated into planning processes, board members often appreciated the utility of applying such metrics in their oversight and management role.

Moving Forward

Many police boards and police services will make design changes to bring their performance measurement tools into alignment with their outcome based service delivery goals, service level standards, and budgetary decision making. The improvement of police performance measurement will be an ongoing, iterative process occurring over several years and will likely require guidance and support. As police performance measurement tools evolve via real life successes, there will be a better understanding of what kinds of indicator design improvements can be achieved.

Police performance measurement is not an end unto itself. Even the most sophisticated and perfectly considered performance metrics will not improve organizational performance unless they are applied in decision-making. All the data and performance measures are of little use to police boards if these boards lack a clear idea of how to use them to ensure accountability, improve quality of service and reduce costs. There is no one magic performance measure, nor is there a need to be excessively broad in trying to measure all activities. Performance measures will continue to evolve, and board members will decide on the managerial purposes to which performance measurements may contribute to their organization. To achieve a positive evolution and accomplish cost management results while maintaining policing quality, there appears to be a pressing need to build "best practice" measurement system design and implementation capacity within both the police boards and policing services.

Introduction

Over the past decade, there have been extensive changes in the performance management systems used in various public service sectors. Public management systems based on bureaucratic hierarchic control are being replaced with market-oriented managerial systems based on accountability for performance (OECD, 1997). As this trend has grown, more forward-thinking representatives of the policing community have realized that traditional policing management approaches based on guesswork, imitation, and intuition can no longer be regarded as an acceptable rationale for police organizational management (Davis, 2012; Maillarda & Savage, 2012; Coleman, 2008). Consequently, there has been a call for a more rational and transparent approach that combines technical and experiential knowledge in order to improve police accountability (Murphy and McKenna, 2007).

This new police management approach emphasizes improving efficiency and cost effectiveness in the delivery of policing services to meet the needs and demands of the community. The public, as the primary client of police services, is recognized as a major stakeholder in determining how to best assess police performance and make better use of police performance data (Maguire & Uchida, 2000; ISIS, 2012).^{Footnote 1} As part of the new approach, accountability has become intertwined with performance measurement at the organizational level to ensure that the police "deliver the best possible service to the public" (Home Office, 2004).

Performance measures, such as service effectiveness, efficiency and quality, are the most important factors in determining how well police are spending public monies. Since 2000, Canadian police expenditures have increased yearly between three and seven percent. In 2010, the operating expenditures of Canadian police services amounted to approximately $12.6 billion (Statistics Canada, 2011). The year 2010 also represented the 14^th consecutive year of growth in constant dollar spending on policing services. With costs expected to continue to grow, pressures have been mounting to develop better ways to assess police performance and meet organizational objectives while holding down costs.

The primary objectives of this paper are:

1. to describe the variety of performance metrics used by Canadian policing agencies;
2. to assess how well these performance metrics are understood and how they are viewed by police oversight bodies, such as Police Service Boards; and
3. to determine which types of police performance metrics police oversight bodies find most useful.

Analytic Approach

To conduct a review on the use of police performance metrics by oversight boards, a mixed qualitative and quantitative analytic approach was taken. This included conducting a content analysis, reviewing performance indicators used by a sample of police services, and conducting interviews on police performance metrics with members of selected police boards.

Content Analysis

A content analysis was conducted of the documents provided by police boards in order to assess trends in the use of police performance metrics. Among the 48 police services identified, strategic and business planning documents were identified for only 19 police services. These documents were mainly found on the police board or services websites, but additional documents were sometimes provided following the interviews. An overview of the coding and categorizing of the data and some limitations of the approach are discussed in Appendix B.

Interviews

Following initial contact with 48 police boards, semi-structured interviews were conducted with 30 individuals representing the 15 police boards that agreed to be interviewed. The research team also received correspondence from another 15 representatives of police boards who indicated they either did not have any performance measurement systems in place or could not meaningfully contribute to the interviews for other reasons. This resulted in a response rate of 62 percent. In addition, eight individuals acknowledged receiving correspondence and stated they had forwarded the information to the Chair of the Board for consideration of a possible response. Ten police boards contacted did not provide any response. For a breakdown of the sample see Appendix B.

The major objectives of the interviews were to identify the performance measurement metrics used by police organizations and to evaluate the utilization of performance data in accountability reporting, business planning, operational decision making, and budgeting. The interviews were conducted with those police board members (e.g., chair or vice chair) determined to have a particular understanding of, and responsibility for, the use of police performance metrics. There were some practical and sociopolitical considerations addressed while conducting this research, which are noted in Appendix B.

Overview of Performance Measures

While there are multiple definitions of performance measurement, simply put, it is the development, application, and use of performance measures (also referred to as indicators) to assess the degree to which organizational objectives are being met. Performance measures are an essential component of core management processes, such as budgeting and strategic planning, total quality management, benchmarking, scorecard metrics and organizational learning (Talbot, 2010). Any particular performance measure is composed of a unit of measurement and a number expressing magnitude or intensity. The unit gives content or meaning (a standardized definition of what is being measured) and the number gives a statement of degree of magnitude (how much there is of the unit being measured and what is its strength or intensity). Performance measures need to be tied to a target or objective; their components can be represented in isolation (e.g., in terms of units such as hours, number of accidents, length of time, dollars, or number of errors) or in combination as the ratios of two or more units (Neely et al., 1995).

Valid and productive performance measurement approaches rely on some type of input-output-outcome model.^{Footnote 2}  Table 1 gives an overview of definitions of key terms used in such models.

Police Performance

The application of performance measures to policing is highly challenging given that policing is a complex area of public service with a diversity of objectives and delivery activities. Moreover, much of what drives demand for policing (including broad societal factors, such as income inequality, employment levels, and demographic changes) is not within the scope of policing service to effect change (Vollaard, 2003; Alpert & Moore, 1993; Moore et al., 2003). Dadds and Scheide (2000), Faull (2010), and Davis (2012) point out that data on reported crime rates and crime clearance rates have traditionally been used in both instrumental and symbolic ways to justify the types, and costs of different types, of police services and to enhance public relations. Such data have also been used to evaluate practices based on internal policies (Alpert and Moore, 1993; Moore and Braga, 2004). The logic behind using crime rates as performance measures requires demonstrating the relationship between increases or decreases in crime with the increases or decreases in the nature, quantity, and quality of particular types of police services.

Research has shown, however, that the use of traditional measures such as clearance rates, arrests rates and response times are not always the most appropriate measures to understand the scope of policing activities from solving crimes to maintaining public order (Collier, 2006; Goldstein,1990; Moore, 2002), making it difficult to determine the true nature and extent of the impact of police activity on the intended result. Furthermore, there are issues of consistency (in terms of both validity and reliability) in measures of police performance using these traditional indicators (Alach & Crous, 2012). Such challenges have been a major impetus for police to rethink traditional approaches to performance measurement and to design performance measures that focus on the daily activities of policing and take into account the various contextual circumstances shaping policing and the legislative and practical constraints under which the police operate (Alpert & Moore, 1993).

Statutory Requirements for Police Performance

Currently, Canadian provincial police acts have no specific statutory requirements for specific performance measures. While these acts vary from province to province, they typically set uniform standards for municipal policing services, provide for the creation of police boards and commissions, and establish criteria for the management and operation of police services. Under these criteria, some police boards have approved performance metrics but there are no provincial guidelines or policies on the development of specific performance metrics. The Ontario Police Act, section 30 (1) of the Adequacy and Effectiveness of Police Services Regulations, mandates the existence of qualitative and quantitative performance measures in specific areas of the Board's business plan relating to: (i) the police force's provision of community-based crime prevention initiatives, community patrol and criminal investigation services; (ii) community satisfaction with police services; (iii) emergency calls for service; (iv) violent crime and clearance rates for violent crime; (v) property crime and clearance rates for property crime; (vi) youth crime and clearance rates for youth crime; (vii) police assistance to victims of crime and re-victimization rates; and (viii) road safety.

In Ontario, the Municipal Act, 2001 requires municipalities to measure and report to the taxpayers on their service delivery performance under the Municipal Performance Measurement Program (MPMP). Under the MPMP, municipalities report on efficiency and effectiveness measures for 13 service areas that include policing (Ontario, Ministry of Municipal Affairs and Housing, 2007). Table 2 provides an overview of reported MPMP measures.^{Footnote 3}

While not a statutory reporting framework, the Ontario Municipal Benchmarking Initiative (OMBI) is an established performance measurement initiative. The OMBI was established by a consortium of Chief Administrative Officers and City Managers to collect data on more than 850 measures across 37 municipal service areas. Currently, there are also 13 large Ontario municipalities and two cities located outside Ontario^{Footnote 4} covered under this initiative. Under the OMBI process, each service delivery area (including policing) has established an expert panel to review submitted data and assess its consistency and relevance to the OMBI process. The OMBI Expert Panel has identified 41 policing measures, classified under the categories of Community Impact, Efficiency and Customer Service. A cautionary note to consider is that a consensus of "experts" is not the same thing as expertise, particularly if the experts in question all represent the same sector. A description of these measures and the type of data to demonstrate their outcomes is presented in Table 3 (Appendix A). These performance measures, which are mainly used by municipal managers, may also be reported to the relevant police board. Many of these performance measures may also be incorporated into the police service's overall performance measurement framework.

Overview of Police Oversight Boards

In Canada, provincial police acts give the police boards their powers to oversee the administration of police services. According to the Canadian Association of Police Boards (2006), the mandate of police boards typically covers the following general areas: (1) determining adequate personnel levels; (2) budgeting for the needs of police services; (3) monitoring the budget; (4) reviewing police performance; (5) hiring the chief of police; (6) assisting with labour relations; (7) dealing with discipline issues; and (8) helping to develop policy.

Terminology referring to police boards varies across Canada. They are referred to as police services boards in Ontario, committees of council in Alberta, and boards of commissioners of police in British Columbia and Saskatchewan. Most police boards consist of between three and nine members. The police boards of the largest municipalities typically have the most members. The composition of boards tends to be drawn from the ranks of citizens, often drawn from among volunteers working in the communities and elected officials. In British Columbia, for example, the chair of the police board is typically a mayor of a municipality. ^{Footnote 5} Board members are politically-appointed and are "chosen to reflect the demographics of the community." Ideally, they are "persons who have demonstrated that they can act in the best interest of the community" (BC Ministry of Public Safety and Solicitor General, 2005, section 3.0:11). Members of police boards are generally not paid salaries, but some municipalities may cover specific kinds of expenses incurred by board members while performing board duties.

The role of police boards and their relationship with the police and the overall societal governance process has been extensively debated (LaLonde & Kean, 2003; Stenning, 2004; Law Reform Commission, 2006; Martin, 2004; Pidd, 2007). However, when it comes to assessing police performance, most provincial statutes focus mainly or exclusively on assessing the police chief's performance. For example, under the Ontario Police Services Act, the boards are responsible for policies on effective management of the police and can direct and monitor the performance of the police chief. However, the Police Services Act, (1990) Regulation 421/97, states that: "Board members shall not interfere with the police force's operational decisions and responsibilities or with the day-to-day operation of the police force, including the recruitment and promotion of police officers."

A general challenge facing many police boards is the need to make a paradigm shift away from traditional "individual" policing performance (often based on the perceived success of the police chief) towards a new evidence-based policing management model where organizational performance is measured in terms of a clear definition of responsibilities, increased transparency and accountability, measurements of effectiveness and efficiency, and an expanded role for citizen input (Davis, 2012; Milligan & Fridell, 2006; Kiedrowski, et al., 2009).

One document reviewed illustrates some of the specific challenges facing police boards in the area of performance measures. The report's author outlines that: (1) police management lacked the capacity to evaluate the performance of their patrol functions; (2) although the Board's own governance manual required that annual business plans be prepared, this had not been done; (3) the Board's performance expectations for the Chief were general and not tied to specific policing objectives or outcomes in an annual business plan; (4) the Board's budget was not tied to an annual business plan, strategic objectives, or to performance measures for evaluating budget allocations and there was little or no analysis of draft budgets; (5) the members of the Board had been informed their role was not to question specific costs as that was deemed to be the activities of the local finance committee; (6) the Board did not see its role to be to direct police services to manage expectations in order to help keep costs in line with the proposed budgetary items. Finally, the author noted that an outside review had indicated that the police service needed to develop key performance indicators for departmental operations. Many of these findings could apply to other police boards interviewed for this paper, especially those that have no meaningful performance measures in place.

Content Analysis

In order to get an objective quantitative description of the content of police performance metrics, a content analysis of documents provided by police boards was conducted. More documents were added to the sample as a result of the interviews. This analysis enabled us to sift through a large volume of information on police performance metrics in a systematic fashion in order to make inferences on how police performance measures were implemented in police services. The research team was later able to compare information obtained in the content analysis with information that was gathered from interviews with police board members. There were several limitations with the written materials that were examined in the content analysis. First, most of the documents were strategic plans or business plans. These seldom presented data showing whether or not performance targets were being achieved. One explanation for this was that because many of the police boards and services were in the early stages of performance measure activity no meaningful results were yet available. Second, the sample may have left out some police services with documents showing the impact of their performance measures. Third, while all the websites of the police boards selected for the sample were reviewed, few police boards or services had posted any information on police performance metrics. Consequently, it can either be assumed that either no performance measurements existed or that the results of existing performance metrics were not easily accessible to the public.

Findings from the Content Analysis

The content analysis focused on 20 police services and boards. This included 16 large or regional municipalities, three smaller or rural communities and two First Nations community.^{Footnote 6} The analysis found no commonly accepted model or framework for measuring the performance of police organizations. Some of the services stated they were following the "balanced scorecard" model, others referred to their approach as an "excellence" or "public value" model, while still others claimed to use some type of home-grown model.

Table 4 gives an overview of what dimensions were most frequently considered in the performance measurement systems of individual police services. ^{Footnote 7} There is no single performance dimension that all police services use in their performance measurement framework, and there are large differences in the frequencies with which different dimensions of performance are considered. Some police services treat performance measures and performance indicators as equivalent while others consider the two as entirely different. Some police services use the terms objectives, goals and targets interchangeably; others do not. The police have designed their own performance management system to reflect their own priorities and the demands within the communities they serve. While one can appreciate that each police service uses the performance measurement terminology they consider to best fit their organization, this does make it difficult to draw comparisons with other police organizations.

In terms of the categories of measures, goals (N=907), measurement/indicators (N=652) and objectives (N=471) are the most frequently mentioned terms. Dimensions like effectiveness and efficiency, although essential to performance management frameworks because they indicate how successful organizations are in attaining their objectives, had a lower count. Effectiveness was only used 88 times and efficiency 63 times. This finding is in line with that of other studies (e.g., Julnes & Holzer, 2001; Strieb & Poister, 1999) showing that efficiency and effectiveness measures tend to be used with less frequency than other kinds of output measures in strategic business planning reports.

The content analysis showed that police services considered over 2,000 indicators focusing on categories such as crime and criminal justice, youth and crime, traffic, response to calls, elder safety, crime prevention, community consultation, and administrative issues such as IT/Web, organizational excellence or capacity, and human resources. To organize the information, obtained, the research team used the "balanced scorecard" model for law enforcement (Moore et al., 2002; Moore & Braga, 2003) which outlines seven key dimensions of policing performance. Selected police performance measures were then reviewed and placed under one or another of these seven dimensions. Table 5 below provides an overview of the seven dimensions, as well as suggested measureable performance indicators for each of these dimensions. While the measures used by larger urban police services and those used by the police services of smaller municipalities may vary, the idea here is to show how various performance indicators can be aligned with key performance dimensions to produce a reliable performance measurement system.

Table 6 (Appendix A) provides a more detailed and comprehensive listing of possible indicators used by some of the larger urban police services that were reviewed. Many of the performance indicators used are standard policing or crime measures compiled in federal criminal justice statistics. These include the following: the violent, property and other criminal code rates per 100,000 population; violent and non-violent crime severity indices; information from various surveys (such as those administered to  law enforcement personnel, victims of crime, partner agencies or the broad community); and information from other sources such as provincial highway traffic agencies, and liquor and gaming authorities. The results of such measures are typically stated as follows: in terms of percentages, ratios or counts; in terms of units of time (e.g., travel time in hours); in terms of dollar values; or in terms of trends. Other performance indicators (or statements of goals or objectives) focus on the delivery of programs, implementing or upgrading computer software, and streamlining administrative processes. Finally, there were also instances of the use of performance indicators where the type of measurement and how it was analyzed were unclear. The important point here is that police services are making a serious effort to produce a performance metrics system, but the degree to which these indicators can produce reliable measures vary among the police services.

Tactical versus Strategic Performance Measurements

To further understand the nature of the performance measures identified by the police, the goals or headings were categorized as either tactical or strategic performance measures. An example of a tactical measure is to describe the average time it takes to respond to a call for service and list the number of calls for service. This type of performance metric measures the volume of operational activity. An example of a strategic measure is to describe reductions in the time it takes to respond to calls for service of different types and identify the human and financial resources required by call, for service of a particular type. With this type of performance metric a police organization would attempt to align its business activities to its strategic plans and to monitor progress in achieving strategic goals. Table 7 shows performance indicator goals classified as tactical or strategic.

Tactical performance metrics, that measure operational activity, and strategic performance metrics form a continuum. A tactical or operational performance metric can often be converted into a more sophisticated, strategic performance metric by adding another dimension to the measure, such as reporting the business activity as a rate or proportion of a another relevant measure. Among goals examined in the content analysis, the majority of the performance measures identified can be classified as being strategic in some way. The concept of "effectiveness" was used 54 times to describe a strategic goal (desired outcome) as opposed to an actual outcome for a tactical goal. The data indicate that police services are currently focused more on measuring progress towards strategic goals than measurement of operational, tactical objectives. To better track operational performance measures, including cost effectiveness and efficiency, it has been consistently suggested that police services need to improve their computer systems and data analysis capacity while continuing to focus on achieving a clearer sense of strategic direction (Davis, 2012; Bayley, 1996; Alpert et al., 2001; Alpert & Moore, 1993; Moore, 2002; Moore & Poethig, 1999).

Performance Indicator Numerator and Denominator Statements

Quality performance measures are constructed in a variety of ways, including the use of ratios, percentages or proportions, counts, annual percentage rates of change, or averages. The most basic measures are numerator/denominator equations. The denominator represents the population evaluated by the performance measure that is the number of total units of service or resource, whereas the numerator represents the number of units in the denominator (universe) that receive some deliverable or experience a specific outcome. For example, a fisherman may catch a number of fish (numerator); but to assess how skilled a fisherman is, one must also know the approximately number of fish available to be caught (denominator). If the performance measure is reported as a rate (proportion or ratio), the numerator and denominator statement are completed, though there may yet be validity concerns arising from the logical relevance of a nominator to a denominator. For example, calculating residential break-ins (nominator) as a number per total population (denominator) would expose the result to distortion over time resulting from changes in numbers of occupants per residence. As an illustration, the Ouimet Commission in the 1960s reported a massive increase in motor vehicle theft over the first half of the century, without noting changes in prevalence of vehicle ownership. Had vehicles available to be stolen, rather than population, been used as denominator the trend would have appeared dramatically different.

As part of the content analysis, an attempt was made to examine various performance indicators and identify the numerators and denominators that were used. Unfortunately, the documents reviewed often did not provide operational definitions of the type of data to be collected, the numerator, and the denominator. In some cases where a performance indicator was stated by a particular police service, it was unclear as to what type of data would be collected (and how it would be analyzed) and whether there was, for each indicator, a rationale for the use of that measure and for definitions of the numerator and denominator. It was also unclear whether exclusions, notes, or comments for each variable were provided elsewhere in police services operational or policy manuals. Almost all denominators referenced were either population numbers (e.g., total or youth population, total number of members) or simply a gross volume number for a previous year (annual rates of change). In some instances total police expenditures was used as a denominator with various line items of spending used as numerators. Many of the most unclear measures focused on human resources or the operation of the police organization. The data that was reviewed showed that police services are using some of the commonly acceptable criminal justice indicators developed by the Canadian Centre of Justice Statistics (CCJS) and the OMBI as their main performance measures. Table 8 provides an overview of these indicators as well as operational definitions of the numerators and denominators (population) used.^{Footnote 8}

For a general review of the methodological challenges involved in using these measures, see the following sources: Dadds & Scheide, 2000; Alpert & Moore, 1993; Alpert et al., 2001; Moore & Braga, 2004; Melchers, 2001. However, as police services move to indicators that are identical to the statistics produced by the Canadian Centre of Justice Statistics, the following questions might be raised with regard to the appropriateness, validity and reliability of the performance measures for individual police services:^{Footnote 9}

1. How reliable is the method/process for collecting data? Is it repeatable and consistent with low error rates? Is the number a standardized countable unit of work or population? Are units of work (e.g., policing hours) reliably counted and compiled? Do data definitions remain consistent across time intervals? Is data collection audited or periodically reviewed for accuracy?
2. How valid is a single number? For instance, is the "cost of service" numerator in efficiency measures accurate for the unit of service in question? Are patrol or incident response times consistently tracked against a standard consistent process?
3. How valid (i.e., logically linked to the numerator) is the denominator, where there is a denominator? Is the reported policing population appropriate (e.g., with regard to changes in household occupancy, motor vehicle ownership and operation, demographics, geographical unit boundaries, ecological fallacies, day-time activity shifts)?
4. How valid are the time intervals for time series measurement? Is an annual or a five year interval appropriate to show a trend in results? How reliable are trend data? Do data definitions remain consistent across time intervals?

A Critique of the Police Performance Measures Reviewed

Based on the performance measures that were reviewed as part of the content analysis, two stages of review were executed by the research team.^{Footnote 10} First, the research team reviewed the overall portfolio of performance measures for each policing service in the sample using a "top down" approach. This involved assessing each police services' overall performance measurement framework and the nature of the measures used.^{Footnote 11} The approach was to review the performance measures framework and determine how the metrics relate to the wider context in which they are used, whether that be an internal organizational context or an external (i.e., community) one. Similar approaches to evaluate performance frameworks have been used by others (Johnsen, 2006; Neely & Adams, 2001; Bourne et al., 2003; Caplice & Sheffi, 1995). In this case, the information was then categorized into one of the following four "Models."

No Model: A performance measurement framework could not be identified or the police board or police service indicated they have no performance measurements in place.

Beginners Model: This model is characterized by an emphasis on workload indicators that simply note work completed or to be completed. Simple existing volume-based statistics are reported rather than meaningful results-based indicators. The focus is exclusively on reporting volume of activity, as opposed to providing evidence-based support for budgeting and other types of higher order decision making. The only management question the Beginners Model can answer is: "How busy are we?" Indicators used here reveal virtually nothing about results; their selection is driven by their easy availability, rather than utility.

Efficiency Model: This model is a clear step forward from the Beginners Model. It includes useful indicators such as input-output measures, unit cost measures, and the reporting of time-series data trends, but the focus is primarily on public reporting. There is, however, little attention given to issues of police service outcomes, effectiveness or measurements of quality of service.

Balanced Model: This higher order performance model builds on the efficiency model by adding effectiveness and quality measures that properly consider cost versus results and quality trade-offs. Cost and quality trade-offs are at the core of decision making. This model is the most useful for public reporting, operational planning and evidence-based budget decision making. Time series data trends are usually considered. Meaningful peer benchmarking is an important "value added" component of this model.

In addition, each portfolio of performance indicators was subjected to a review based on the relevance of performance measures. Relevance is considered to be the degree to which the performance measures balance the overall performance measurement framework in relation to efficiency and effectiveness of those measures. Each of the indicators was classed according to the following degree of relevance:

High relevance:^{Footnote 12} This assignment was given if the portfolio of measures purported to address policing effectiveness and quality (e.g., patrol response times), as well as efficiency, in supporting decision making.

Medium relevance: This assignment was given if the measures used only addressed outputs, productivity and efficiency (e.g., cost per capita). Quality and effectiveness measurement is absent in supporting decision making.

Low relevance: This assignment was given if only volume measures of workload activities were used (e.g., number of traffic tickets processed).

Zero relevance: This was given if no actual performance measures were used.

Table 9 shows the number of police services that were assessed as having relevant performance indicators based on the evolving model criteria, and their corresponding indicator relevance.

Police services meeting the criteria of the Balanced Model generally came from those among the larger urban and regional policing services in the sample. Many of the boards for these police services have full- or part-time professional staff. Police services with no performance metrics included a few from large urban jurisdictions, but were mainly services from smaller municipalities and rural areas. In both these cases there were no meaningful regional variations across Canada. The four police services rated as Beginners were mainly from medium-sized jurisdictions, except for one which is a large urban police service. The table further shows those police services that meet the criteria for a Balanced Model were also ranked in the high relevance category. Most of the police services had a score of "low relevance" in their use of performance metrics because of imprecision in their units of analysis (e.g., a lack of clarity as to what is being measured). Whole categories of indicators such as effectiveness (i.e., outcome) and quality indicators are absent.

The analysis indicates that the use of performance measures is likely evolving in Canada. This evolution is the result of police services gaining more experience in the development and implementation of performance metrics, perhaps, when the resources and pressures to evolve are present. This is depicted in Figure 1 which shows a cluster of police services at the bottom of the evolution curve that have no performance measures in place, compared to other forces that have implemented a framework. Many police services appear to be gaining experience with performance metrics. As a consequence, there has been substantial improvement in the quality of performance measurement frameworks. As police services implement a performance measurement framework, there may be variance between the current portfolio of performance measures used and the objective of high relevance in their use. During these periods of variance between the actual measurement tools currently used and the more ideal measurement tools that will hopefully be developed sometime in the future, police services need to make changes to bring the process in line with their outcome-based service delivery goals and service level standards. This will be an ongoing and iterative process occurring across several fiscal years. As the police performance measures further evolve, success can be achieved in using measurement tools to improve results, or a better understanding of what indicator design improvements can do for them. Consequently, the variance between the indicator framework and relevance of use will decrease. This variance is smallest when the Balanced Portfolio is adopted to support budget decisions, police establish quantified operational targets, and report actual results against assigned targets.

Figure 1: Evolution of Police Performance Measures

Image Description

Figure one is a graph entitled "Evolution of Police Performance Measures," which depicts the relationship between the understanding of performance measures (along the Y-axis) and experience in performance metrics (along the X-axis). Both axes are labelled by the researchers using a qualitative scale from low to high understanding or experience. Data from police boards and police services are plotted on the chart. The points on the chart fall into several groupings: ten or so police services applying no performance measures (clustered at a near zero understanding of performance metrics and near zero experience in performance metrics); five beginner police services (clustered at a lower understanding of performance metrics and lower experience in performance metrics); two police services categorized as "efficiency" services (clustered at a mid-level understanding of performance metrics and mid-level experience in performance metrics); about nine police services clustered at a mid-to-high level of understanding of performance metrics and mid-to-high level experience in performance metrics). A curve was drawn between the points that described a positive correlation slope. The curve was not quite a linear curve, but indicated a more balanced correlation between understanding and experience at higher levels, while at lower levels experience in performance metrics increased marginally more rapidly than understanding of performance metrics. Another curve is drawn on the chart to represent the variance between performance indicators and indicator relevance. That curve describes an arc of a more steeply curvilinear form where the same relationship between understanding of performance metrics and experience in performance metrics is even more clearly depicted.

Interviews

Police board members were selected for interview through a purposive sample (see Appendix B). To determine police board membership, an Internet search of the identified police board or their policing service was conducted. In some cases, an e-mail address and contact information was found for some of the police boards that operate in the larger municipalities in Canada. Where specific e-mail addresses were not identified for particular police boards the following approaches were used:

1. e-mails were sent to the "contact us" address that some municipalities and towns use to direct questions to police boards;
2. an on line "contact us" feature was used to leave a message in a municipality's message box to request contact information for the police board or local policing commission;
3. e-mails were sent to the general mailbox of particular police services;
4. e-mails were sent directly to the Chief of Police asking for contact information for the local police board;
5. e-mails were sent to the mayor of the community asking for police board contact information; and
6. where no e-mails addresses were found a Fax was sent to the police services and City Hall's general Fax number.

In addition, the Canadian Association of Police Boards (CAPB) sent an e-mail to their members notifying them about the project and the possibility that they might be selected for an interview. Reminder notices were sent out to encourage participation. To increase the number of possible participants, a "snowball" (or more properly, referential) sample technique was used in which those individuals interviewed were asked to send the description of the project to colleagues sitting on similarly sized police boards or oversight committees.

While a variety of practical and sociopolitical considerations affecting this study are noted in Appendix B, it should be noted that two factors in particular may have significantly influenced the interview response rate.

First, financial budgets for municipalities in Canada are based on the calendar year. Consequently, it was a challenge to schedule interviews in December when many key board members are involved in the final approval of municipal budgets. In addition, some of the board members are mayors or elected councillors sitting on other agencies, boards, or commissions where budgets are being discussed and approved during the period of this study.

Second, a major logistical challenge was encountered when identifying the members of many police boards and contacting them directly. Some police boards have their own websites, and a few others had contact information posted on their police service's website. In most cases, however, the names of members on the police boards, or instructions on how to directly make contact were not available. While there is a national association representing police boards, this organization only provides contact information for its own members. It was pointed out to the research team that because police boards and commissions frequently change composition, with members serving different types and durations of appointments and terms, it is virtually impossible to keep membership information current.^{Footnote 13}

Besides the interviews with police board members, interviews were conducted with the three provincial government agencies responsible for provincial police acts and the setting of provincial standards. Interviews were also conducted with a small number of other selected stakeholders.

Findings from the Interviews

Among the 30 board members that participated in the semi-structured interviews, three of the members interviewed reported they did not use performance measures. The results from these interviews will be discussed later. The interview guide focused on 22 questions that elicited information on how police boards view performance measures. These questions were aggregated into three areas: (1) general information on police performance metrics; (2) overview of performance metrics; and (3) police board involvement in performance measures.

General Information on Police Performance

Under the rubric of general information on police performance, respondents were questioned about their understanding of the nature of their mandate and the process involved in implementing police performance metrics. Many respondents noted that their provincial police acts do not specifically mandate the carrying out of police performance measurement. However, under the various police acts, the police boards are responsible for the governance of their police services. Under this requirement, the police boards can incorporate police performance measures as part of their strategic planning and budgeting processes. Members of one police board noted that while they have attempted to implement some form of police performance measurement since 1995, a process for doing so was only formalized in 2006 as part of municipal amalgamation. Three other interviewees stated that their Boards had moved to implement performance measures around 2009, and that they were still in their first three years of a strategic plan incorporating performance measures.

Several individuals whose police boards fall under the jurisdiction of the province of Ontario pointed to the Ministry of Municipal Affairs and Housing's requirement that the police must report data and other measures of services under the Municipal Performance Measurement Program (MPMP). These individuals also indicated that their particular police services were participating in the Ontario Municipal Benchmarking Initiative (OMBI). A few respondents pointed out that the reports for the MPMP and the OMBI are presented to the board for information purposes only and not as part of their strategic planning process. When prompted to clarify this statement, these respondents suggested the MPMP requirements fall under the Ontario Financial Information Return and do not necessarily reflect the board's requirements. Similarly, they expressed the view that the OMBI information is used more by municipal administrators than by the police boards. Some police boards that do receive such information, stated they view it as "nice to know" but not essential for their work. One particular board member argued that while the OMBI provides a good framework for comparison among participating police services, the information collected has little relation to those police performance measures they consider most important. They consider the performance measures that were identified by the community through public consultation and those identified by the police services as most important. Some of these performance variables used by the police services, however, may include some OMBI measures.

Only one board member talked about how their Board had moved to implement performance measures as part of the overall governance directions on achieving greater accountability and improving the delivery of police services to the community. They questioned "what value is the public getting from their police?" and noted several challenges in the implementation of performance measures.^{Footnote 14}

The first challenge noted was the need to convince fellow board members of the value of adopting a performance measurement framework. This respondent suggested that this challenge was based on the perception of board members that their role involved matters of overall governance and that performance measures were the responsibility of the chief in overseeing operational functions. The respondent noted, however, that once information was provided on performance metrics, and how they can be incorporated into planning processes, the board members usually accepted this direction.

The second challenge noted by the board member was the need to secure a "buy-in" from the Chief of Police and police service members which was not simply symbolic, but meaningful in terms of outcomes that could be identified, with specific concrete actions pointing to changes in the ways in which the police operate. The respondent stated that it was not such a difficult task to gain the Chief's support, but that the greater challenge was to get other members of the police service to "rethink" their operational functions in terms of performance measurement-driven outcome targets. The respondent further noted that because the police had to retool their information systems and gather more and different types of information, there was a concern there would be a resulting increase in costs and required effort.

Another respondent noted that their police service had introduced performance measures as part of a new neighbourhood deployment model. Because of amalgamation and the requirement to police both urban and rural areas, many policing activities were found to overlap. Performance indicators were developed to help assess the impact of this change on policing strategies.

The remaining individuals interviewed were unable to provide any historical context regarding the implementation of police performance measures in their respective boards.

Overview of Performance Metrics

The next series of questions focused on specific measures the police service present to the Board to demonstrate results or outcomes achieved. Unfortunately, the respondents were unable to identify the services or functions that are the focus of performance measures, and could only make reference to either their business or strategic plans. These documents were reviewed under the section on content analysis.

All respondents noted that the performance measures currently used in their police services are based on a variety of historic data collection processes that pre-date the selection of the measures themselves. These data sources include the police services or independent firms conducting community surveys, consultation with partner agencies, or the execution of priority-setting strategic plan exercises. Some respondents noted their police services conducted an environmental scan to help to identify community priorities, while others used a SWOT analysis (Strengths, Weaknesses, Opportunities, and Threats) or SMART criteria (Specific, Measureable, Achievable, Relevant, Timely) to help select their performance measures. These data collection activities were carried out by the police services, and the information was presented to the police boards. When asked if any of the board members participated in this data selection process, the respondents typically expressed that they were not personally involved, and did not know if any of their fellow board members were involved in any of these processes.

In response to questions focusing on the type of performance measures collected and their intended purpose, a few respondents expressed the view that the police need to adopt "outside the box thinking" when generating performance measures targeting activities of particular relevance to their specific communities. These "outside the box" performance measures refer to the view that the police cannot achieve outcomes without relying on other agencies within the community. The problem, as identified by one respondent, is that the police are accustomed to only think about measures that exist within data sets they already generate directly themselves.^{Footnote 15} Consequently, the board member noted that police need to reach out to various municipal service groups, and link their performance metrics to the operations of those agencies that have similar or overlapping objectives. Two respondents suggested that the police need to turn over "social work" related performance measures to those agencies that have special mandates to deal with them. In this particular case, the individuals noted police need to re-think how they develop their performance indicators that they need to incorporate measurements of key outcomes stemming at least in part from community-based interventions.

One respondent argued that a challenge for the police is that these types of performance measures are interrelated with the activities of other agencies and consequently, there is a need to work with those agencies and share data and information. In these instances, the police need to recognize the work that other municipal agencies and non-government organizations do, and not take exclusive credit for outcomes that reflect to varying degrees the efforts of other organizations. To overcome this multi-agency "interdependence challenge," one respondent suggested that there is a need to better share information and data among community partners, by establishing protocols governing their relationship and holding regular meetings to address the issues of performance and how the goals for all organizations can best be achieved. A critique raised by some respondents was that some police performance measures are serving a more symbolic (i.e., considerations of image or public relations) than instrumental purpose.

One respondent noted that some police services list many pages of performance measures "measuring virtually everything in their organization," but questioned how useful these measures actually are in terms of achieving core organizational objectives. This respondent indicated that if the major purpose of performance measures is to assess the effectiveness of the organization to address crime issues, and if crime is dropping because of other factors, such as shifting demographics and community-based prevention efforts, then police budgets should be reduced, as there is no longer a need for the current number of police officers. The respondent further argued that factors such as union agreements and "empire building" or "turf protection" on the part of some police services makes it a challenge to determine the true use, and value, of individual officers to organizational performance. The respondent made reference to a particular public debate in which a police board voted to freeze the police budget without triggering any layoffs, and the chief responded with a proposal to close police stations. The respondent asserted that if performance measures were truly being used in a more instrumental, as opposed to symbolic, fashion, there should be a reduction in the number of police officers, as well as a shift in budget allocations to address where policing demands are greatest. In other words, according to the respondent, the police board may get a change in budget allocation, but will not be able to reduce the number of police officers in order to reduce the police budget. This is significant as salaries are roughly 80 percent of the budget in many police services. (This comment, of course, assumes that the number of police officers directly corresponds with organizational ability in meeting its strategic objectives.)

A few board members were unsure as to whether the police services are the only municipal agency implementing performance measures. Board members located in Ontario noted that the fire, emergency and other services have implemented meaningful performance measures as required by the MPMP.

A few respondents provided opinions on ways to improve existing measures. A respondent noted that their police service hired outside consultants to implement a data-driven patrol staffing analysis to enhance their existing information technology. This patrol deployment analysis incorporated patrol performance measures that took into account variables such as number of patrol units on duty, average travel time per patrol unit, and average travel time for calls by priority type. The respondent was of the opinion that such information was useful in board decision making with regard to the police budget. 

Another respondent stated that their police service uses an online survey that focuses on those individuals or victims that have some contact with the police. For example, if the police respond to a traffic accident, the individuals or victims involved will be asked to go online to complete a questionnaire with items on the timeliness and helpfulness of the response. This approach is used to enhance community engagement by specifically focusing on those individuals that have had an encounter with the police. This approach can also be used to collect data to assess specific organizational activities. The respondent also contended that this was a more economical way to get feedback from the community than through community-wide surveys that do not explicitly focus on those individuals that have had contact with the police.

A third respondent remarked that the Phoenix Police Department was the first law enforcement organization in the United States to certify its quality management system to the standards of the International Organization for Standardization (ISO 9001:2000). The Phoenix Police Department introduced ISO to shift its organization from a workload or production based organization to a knowledge-based one, resulting in a more systematic assessment of the effectiveness of their activities over time.^{Footnote 16}

Board Members' Involvement with Performance Metrics

A series of interview questions concentrated on police board involvement in performance measures.

Those respondents interviewed noted that police performance measures are presented in strategic and business plans. These measures are generally reported quarterly along with budgetary reporting requirements.

Several respondents noted that some individuals appointed to the board may have had previous experiences in dealing with some types of performance measures. However, the respondents gave the impression that the longer the member serves on the board the better understanding they gain of police performance measures. All respondents stated that no training is provided specifically on police performance measures. Those individuals sitting on police boards that have full- or part-time professional staff are more likely to have a good understanding of performance measures. The professional staff can take the time to research and report these issues to the board. Many respondents complimented their respective police chief and their staff for taking the time to explain how performance indicators are designed and used.^{Footnote 17} One respondent suggested that boards can secure additional expert resources to assist them, to conduct studies, or to get advice from outside consulting experts. To date, none of the respondents have acknowledged doing so.

When interview respondents were asked about their input in the development of performance measures, a few mentioned that they occasionally made suggestions to revise the performance metric framework. The rationale given was that, as elected officials, they receive specific queries or complaints from their constituents regarding policing matters and accordingly are entitled to make suggestions about how to deal with them. However, many of the other respondents asserted that the development of performance metrics was strictly an operational function, and therefore was the sole responsibility of the chief of police. These respondents contended that it was the responsibility of the police chief to develop the performance measurement plan and to present this document to the board for approval.^{Footnote 18}

On a similar note, one respondent indicated that many board members, who are not typically exposed to thinking about policing as a service delivery "system," have a difficult time understanding such terms as inputs, outputs, outcomes, benchmarks, efficiency and effectiveness.^{Footnote 19} As a consequence, this respondent claimed that many police boards simply "rubber stamp" the proposed performance measures. While the respondent appeared to understand the importance of having a systematic framework for measuring police performance and using the data derived from it, the usefulness of some of those measures to demonstrate success or failure were still unclear to them. They also noted that the data may be presented in misleading ways that exaggerate performance levels. The respondent gave the example of police reporting an outcome measure indicating that 95% of the public view the police as doing a good job. This respondent questioned the 95% figure because the information was not provided in a context allowing it to be properly interpreted. The respondent raised the following questions: 95% of what sample size?; Who was sampled and how representative are they of the population?; and, what descriptive or analytical statistics were used? The respondent argued that police forces may lack the capacity to present meaningful data that truly relate to the performance measures. The respondent further noted that although members of police boards do not receive training on performance measures, such information is nonetheless becoming part and parcel of the planning and budgeting process. The respondent expressed that they were "very uncomfortable" with that situation, as board members are expected to make decisions on information about which they have little knowledge.

There were a few interview questions that focused on whether performance measures have any impact on board decision making processes with respect to the annual budget. Most of the respondents indicated that the performance measures presented by the police chief and their staff are reviewed and incorporated into the decision making process for the budgets. However, a few respondents stated that although performance measures may not lead to reallocations when police budgets are being developed, they do provide useful information to help direct and allocate costs once a budget has been completed and implemented. One respondent suggested that performance measures also help them to better understand planning processes with regard to how money can be spent. Another respondent argued that while many police boards have implemented some type of police performance metrics, they seem to have little impact on the planning and budgeting process. They qualified this view, however, by adding that performance measurement may have an indirect influence on police spending over time. In this case, police performance measures are developed and implemented as part of the three year planning cycle. In some cases, more time is required to determine the impact of some measures. Once an impact is determined the performance measurement can help to make budget allocations.

Over time, board members will become more aware and understand the use of performance measures. Where performance measures have shown that a specific activity is not meeting its desired outcomes then the funds can be relocated to another function. While the performance measurements may be in place for 3 to 4 years, it may take longer to determine impact.

The last question asked the respondents to provide any other additional comments. One respondent noted that police boards have a very high attrition rate, and consequently, for many members, it is very difficult for them to understand their legal responsibility. There is training for the members on board governance but no training on performance measures. In a similar vein, a respondent mentioned that there is a great need to build the capacity of board members (and police chiefs) to understand and use police performance metrics. Another respondent commented that there is a need to establish standards for police performance metrics. As it is now, police boards and services are often working in a vacuum. While information on performance metric selection and design is shared at conferences and other meetings, there is a concern that sometimes the "wrong" type of information is being shared. An example given was how various police services have set up their performance metrics in a shotgun fashion, measuring "everything and anything" related to police work. Measurement systems are failing to discriminate between what data is really needed, and what data simply clouds the snapshot. A comment was made that in some cases, union agreements can be a major challenge to effectively implementing performance metrics or making changes based on performance outcomes (e.g., more use of civilians). Finally, a comment was made that the use of performance metrics can become very politicized, especially in smaller communities. In these cases, the elected mayor or councillor is responsible for appointing board members. These individuals may also chair the police board or committee. If the mayor or councillor or the chief of police is not supportive of performance measures then they will not be implemented. It was suggested, in this type of case, that there needs to be provincial regulated standards and guidelines so that smaller communities do not have an option on whether to implement performance measures.

Interviews with Respondents from Communities Not Using Performance Metrics

Three respondents noted their respective communities did not use performance metrics. A respondent indicated their community is patrolled by the RCMP, and presently has no police board or committee. This respondent noted that attempts are being made to establish a police advisory board with the approval of the provincial Minister of Justice; a decision is pending. As part of the municipal policing arrangement, the RCMP prepare an annual detachment performance plan which is shared only with headquarters in Ottawa, but not with the local municipal council. The respondent suggested that a police advisory committee was not necessary to receive this information, as it could be shared directly with the local elected officials. When queried whether a formal request for the detachment plan had been made to the local RCMP detachment, this respondent indicated that it had been, and it was their view that the local detachment was accountable only to headquarters and not to the local community. In another interview, a respondent provided more information on this point. The release of the RCMP performance plan appears to depend on regional policies on whether to provide the mayor with a copy of the report. In some regions they are shared and in others they are not provided to the community. The respondent added that all communities must receive a copy of this plan because they are paying for the RCMP services and should be aware on how well the police service is performing within those communities.

A respondent noted their community does not make use of performance metrics but does have a police board. This respondent indicated they were happy with the information they were currently receiving which was mainly standard workload police data, such as the number of calls for service.

A respondent reported being part of a volunteer police board for a community served by the RCMP. This respondent noted that they do not have any policies mandating police performance measures, but rely on the RCMP to provide their detachment performance plan. While the current detachment commander works closely with the volunteer board, this has not always been the case. The respondent indicated that if performance measures were to be introduced by the board, the police might view it as a challenge to their policing agreement. The respondent concluded that there is a good informal working relationship now, due to the commander's cooperative approach, but this could change with a new commander in the absence of a formal protocol.

Interviews with Government Officials and Stakeholders

Interviews were conducted with three respondents working in the area of policing policy for provincial governments. The respondents noted that although their policing legislation does not include the use of policing performance metrics, their respective policing boards and services do use such metrics as part of their management and governance structure responsible for planning and budgeting. The respondents further indicated research is being conducted on developing a framework for police performance standards to help standardize indicator design, terminology and definitions.

The respondents also expressed some concern with how much knowledge and understanding board members actually have regarding performance measures. A respondent noted the need for board members to receive some training or education that can be incorporated as part of their governance training. A respondent also noted that there was great diversity among the larger police services that have full- or part-time professional staff, compared to services from smaller communities where all the board members are volunteers and there is no professional staff support.

Conclusions

This report provides a "snapshot" of how Canadian police boards view and use the police performance metrics available to them. Data was gathered for this snapshot through content analysis of documents to which police boards gave access, through a review of the performance indicators provided by a sample of boards, and through semi-structured interviews with board members, provincial government officials, and other stakeholders. A major challenge in completing this research was that police board members are not easily identified and membership is not stable over time. While the Canadian Association of Police Boards assisted the research team in contacting individual police boards, it does not represent all the police boards, especially those communities where police boards have no professional support staff and the members are volunteers working full-time elsewhere.

The findings from the content analysis, performance indicator review, and interviews show that the police boards from the larger jurisdictions are using an evolving portfolio of performance metrics to help their respective police services report results and improve performance.  This research has found that, generally speaking, most police services are making a serious effort to improve their performance metrics system, but the degree to which their current performance indicators reliably measures performance varies widely amongst the police services. The best of these use a "balanced" approach to performance measurement organizing their performance metrics along at least the seven dimensions of 1) reducing criminal victimization, 2) calling adult and youth offenders to account in appropriate ways, 3) reducing fear of crime and enhance personal security, 4) increasing safety in public spaces, 5) using financial resources fairly, efficiently, and effectively, 6) using force and authority legitimately, fairly, and effectively, and 7) satisfying citizen demands for prompt, effective and fair service. Unfortunately, there is a significant design "capacity gap" preventing the adoption of the "best practices" found in the Balanced Model. Challenges to the development of performance metrics include a surfeit of indicators and the use of unclear measures. This capacity gap must be addressed before the policing community can attain meaningful national or provincial quality control or cost management standards. During these periods of variance between the actual measurement tools currently used and the more ideal measurement tools that will hopefully be developed sometime in the future, police services need to make changes to bring their processes in line with their outcome-based service delivery goals and service level standards.

The interviews also illustrated that there is great diversity in board understanding between those larger police boards that have professional staff versus those police board that are strictly volunteer-based. In this research no police service board members were found to have had any specific training on police performance measurement. Although the level of knowledge and understanding of performance measurement is low amongst most members of police service boards, once information is provided on performance metrics, and it is explained how they can be incorporated into planning processes, the board members often appreciated the utility of applying such metrics in to their oversight and management role. Improved information technology and training of both police staff and board members are required for most boards to move forward in adopting results-based management. Furthermore, the use of certain kinds of performance indicators, especially those that focus on complex community outcomes, and require the actions of multiple community agencies, are a challenge for the police. This challenge around measurement causality (who is actually achieving what) is not only limited to policing but is also common to other sectors such as health care (Department of Health and Wellness, 2004) and education (Higher Education Quality Council of Ontario, 2012). It was suggested that the police should turn over "social work" related performance measures to those agencies that have special mandates to deal with them, which could be applied to gauging policing success with regard to community-based interventions or on the issue of mental health.

The interviews also shed some light on the development and use of performance measures. Some respondents gave the impression that their board has already developed sound performance metrics that are part of their strategic planning and budgetary process, while others alluded to challenges with regard to the use of performance metrics.^{Footnote 20}  Board members representing larger policing services, which have more board members and often have full-or-part-time staff to provide support to the board, appear to be better aware of issues pertaining to the use of performance measures. Those individual board members who know more about police performance measures have usually been appointed to more than one term. However, this review also found widespread confusion around the definition of a performance measure, with many jurisdictions mistakenly thinking that promising to do something or simply taking an action without a clearly defined target is a performance indicator. In fact, a proper indicator must allow for gathering quantitative data on the achievement of an actual intended result, not just data on actions taken. This fundamental misunderstanding around the definition or characteristics of a true performance indicator seems to be linked to police business plans that point to numbers of activities carried but feature relatively few actual performance indicators.

The role of some police service boards, and their rapport with police chiefs, do not allow their review to directly lead to reallocations within police budgets or between municipal services. However, the application of more sophisticated police performance metrics could provide useful information to help direct and allocate costs once a budget has been completed and implemented.

Some board members did identify some areas for improvements for performance measures. For example, information technology could be used to better measure the deployment of patrols. To better track operational performance measures, including cost effectiveness and efficiency, it has been consistently suggested that police services need to further improve their computer systems and data analysis capacity, while continuing to focus on achieving a clearer sense of strategic direction. One board member raised the issue of police services examining the adoption of ISO standards. Some American police services, the Phoenix Police Department being the first, have now certified their quality management system to the standards of the International Organization for Standardization (ISO 9001:2000). Thus, the complex public service of policing can be measured at the very highest level of rigour achievable in management sciences.

The content analysis was useful in providing a general overview of the present state of police performance metrics. Surprisingly, a significant proportion of police services do not have any performance measurement framework in place. It is thus unclear how meaningful board expenditure review or budget approval actually occurs. Documents that were made available to the research team were mainly high order strategic plans or business plans. The analysis revealed that among the police services reviewed, there is a very broad range of performance measures. Some police organizations have provided a substantial overview of what they believe are performance indicators for their organization while the response of others was minimal. Some police services listed many goals and targets, but the measurements required to show the achievement of core outcomes are unclear or unknown.

Of the performance measures that were being applied, many could be classified as being strategic in some way. In fact, many police services appear to be attempting to focus more on measuring progress towards strategic goals than measurement of operational, tactical objectives. Tactical performance metrics, which measure operational activity, and strategic performance metrics form a continuum. In some instance, as police improve their performance measurement frameworks, there is an opportunity for tactical or operational performance metrics to be converted into more sophisticated, strategic performance metrics by adding another dimension to the measure, such as reporting the business activity as a rate or proportion of a another relevant measure.

A number of police services appear to track a significant amount of performance data and create performance ratios, but it was unclear as to the type of analysis used to determine whether the goals were being achieved. Where ratios or descriptive data are presented, further data analysis and explanation appears to be required. A relatively small portion of the board sample demonstrated a well-designed portfolio of performance indicators that shed light on core policing outcomes, service delivery productivity and overall value. Not surprisingly, where they were present, these indicators were used in a relevant fashion, extending well beyond simple public reporting, in their annual reports.

The examination of the information on police service indicators showed that performance measurement design and usage are evolving, albeit slowly, in a positive direction. The police services who have gained more experience in developing, implementing and reassessing their performance measures appeared to be on-track in evolving towards a 'balanced' performance measurement framework.

A positive observation was that many police services used comprehensive methodologies to determine community policing priorities, and help to select what areas to measure performance; from conducting environmental scans, to conducting SWOT analyses (Strengths, Weaknesses, Opportunities, and Threats) or applying SMART criteria (Specific, Measureable, Achievable, Relevant, Timely). However, these data collection activities were carried out by police services themselves and did not involve the police boards, although where these types of methods were used data was presented to the police boards.

Services that persevere in attempting to develop proper performance measures, more often than not end up using more appropriate data and the measures become more relevant. Those police services identified as 'beginners' need to evaluate their current processes for developing their performance metrics including the selection of relevant measures. Those police services that have not developed any such capacity at all require additional support to ensure they develop the proper performance metric framework.

The content analysis further showed that many police services appear to be working in a vacuum, with each selecting their own measures, or even worse emulating the wrong peers. Less-than-ideal models are shared, copied or duplicated by other policing services resulting in difficulties in discerning whether the indicators used are true performance measurements. The risk of emulating the wrong measurement models needs to be mitigated by clearly identifying the best practices associated with certain boards in the "balanced" model sample.

Police performance measurement is not an end unto itself. Even the most sophisticated and perfectly considered performance metrics will not improve organizational performance unless they are applied in decision-making. All the data and performance measures are of little use to police boards if these boards lack a clear idea of how to use them to ensure accountability, improve quality of service and reduce costs. There is no one magic performance measure, nor is there a need to be excessively broad in trying to measure all activities. Performance measures will continue to evolve, and board members will decide on the managerial purposes to which performance measurements may contribute to their organization. To achieve a positive evolution and accomplish cost management results while maintain policing quality, there appears to be a pressing need to build "best practice" measurement system design and implementation capacity within both the police boards and policing services.  

Bibliography

Please note that a number of references have been redacted from this bibliography to protect the anonymity of interview respondents. For the most part, the redacted citations are those for police service business plans and annual reports.

Alach, Z and Crous, C (2012). A Tough Nut to Crack: Performance Measurement in Specialist Policing. AIC Reports. Technical and Background Paper 53. Retrieved from: http://192.190.66.44/documents/F/F/6/%7BFF65539E-BE83-45F6-BA97-1BA6959CCD1B%7Dtbp053.pdf

Alpert, G.P., Flynn, D., and Piquero, A.R. (2001). Effective Community Policing Measures. Justice Research and Policy, 3,2: 79-94.

Alpert, G.P. and Moore, M. (1993). "Measuring Police Performance in the New Paradigm of Policing." In U.S. Department of Justice, Bureau of Justice Statistics, Performance Measures for the Criminal Justice System. Washington, D.C.: U.S. Government Printing Office.

Amari, D. (2004). Phoenix Police Department Counts on ISO 9001-2000 – 24/7! ISO Management Systems, Vol. 4, No. 3 (May-June): 25-30

Askin, J. (2009). The Demand Side of Performance Measurement: Explaining Councillor's Utilization of Performance Information in Policymaking. International Public Management Journal. Retrieved from: http://folk.uio.no/josteira/Askim%20IPMJ%20MS%20133_comb.pdf

Banker, R.D., Chang, H., Janakiraman, S.N. and Konstans, C. (2004). A Balanced Scorecard Analysis of Performance Metrics," European Journal of Operational Research, Vol 154: 423-436

Bayley, D.H. (1996). "Measuring Overall Effectiveness," in Lawrence T. Hoover (ed.), Quantifying Quality in Policing, Washington, D.C.: Police Executive Research Forum.

Beare, M. and Murray, T. (2007). Police and Government Relations: Who calls the Shots? Toronto: University of Toronto Press

Bourne, M., Neely, A, Mills, J and Platts, K. (2003). Implementing Performance Measurement Systems: A Literature Review. International Journal of Business Performance Management, Vol. 5, No. 1: 1-23

British Columbia, Ministry of Public Safety and Solicitor General (2005). Handbook: BC Police Board Resource Document on Roles and Responsibilities Under the Police Act. Retrieved from: http://www.pssg.gov.bc.ca/policeservices/shareddocs/policeboards-handbook.pdf

Canadian Association of Police Boards (2006). Board Composition and Duties. Retrieved November 22, 2006, retrieved from: www.capb.ca.

Canadian Centre for Justice Statistics, Policing Services Program (2002). Uniform Crime Reporting Incident-based Survey. Version 2.0. Reporting Manual, January 24, 2002

Caplice, C. and Sheffi, Y. (1995). A Review and Evaluation of Logistics Performance Measurement Systems. The International Journal of Logistics Management, Vol. 6. No. 1: 61-74

Coleman, T.G. (2008). Managing Strategic Knowledge in Policing: Do Police Leaders have Sufficient Knowledge About Organisational Performance to Make Informed Strategic Decisions? Police Practice and Research, Vol. 9. No 4: 307-322

Coleman, T.G. (2012). A Model for Improving the Strategic Measurement and Management of Policing: The Police Organisational Oerformance Index (POPI). Ph.D. Dissertation. University of Regina.

Collier P (2006). In Search of Purpose and Priorities: Police Performance Indicators in England and Wales. Public Money and Management 26(3): 165–172

Dadds, V. and Scheide, T (2000). Police Performance and Activity Measurement, Trends and Issues in Crime and Criminal Justice, 180. Australian Institute of Criminology.

Davis, R. (2012). Selected International Best Practices in Police Performance Measurement. Rand Corporation Technical Report Series.

Department of Health and Wellness, Province of New Brunswick (2004). A Report to New Brunswickers on Comparable Health and Health System Indicators (November), Communication Branch

Domaas A., and McIndoe, M. (2012). British Columbia Association of Police Boards: Report to the Executive Board. Retrieved from: http://deltapolice.ca/about/board/agenda/2012/agenda_120912.pdf

Faull, A. (2010). Missing the Target: When Measuring Performance Undermines Police Effectiveness. South African Crime Quarterly, 31:19–25.

Greatbatch, J and Associates (2012). A Framework for Effectiveness: Governance and Funding Issues.

Goldstein, H. (1990). Problem-oriented Policing. New York: McGraw Hill.

Hatry, H. P. (2002). Performance Measurement: Fashions and Fallacies. Public Performance & Management Review 25(4): 352-358.

Hatry, H. P. (2006). Performance Measurement: Getting Results. Urban Institute Press.

Higher Education Council of Ontario (2012). Performance Indicators for the Public Postsecondary System in Ontario. A consultation paper. Retrieved at: http://heqco.ca/SiteCollectionDocuments/HEQCOPerformanceIndicatorProjectConsultationPaper.pdf.

Home Office Police Standards Unit (2004). Managing Police Performance: A Practical Guide to Performance Management. London: Home Office

ISIS (2012). Final Research Report to CACP Board of Directors. Full Circle Community Safety: Changing the Conversation about Police Economics and Performance.  (July). Retrieved from: http://www.cacp.ca/ISIS/admin/Documents/upload/ISIS_2012_Final_Research_Report_(English)_for_CACP.pdf.

Johnsen, A. V., J. (2006). Is There a Nordic Perspective on Public Sector Performance Measurement? Financial Accountability & Management 22(3): 291-308.

Julnes, P., and Holzer, M. (2001). Promoting the Utilization of Performance Measures in Public Organizations: An Empirical Study of Factors Affecting Adoption and Implementation. Public Administration Review, Vol. 61, N. 6: 693-708.

Kiedrowski, J., Petrunik, M., and Melchers, R. (2009). Police Services Performance and Accountability: A Review of the Literature. Report Prepared for the Commission for Public Complaints against the RCMP.

Knoll, P. (2005). True Confessions of a Former Police Board Member. Paper presented at the CAPB Annual Symposium, Ottawa, Ontario.

LaLonde, M. W., and Kean, D. W. (2003). Municipal Police Board Governance in British Columbia. Retrieved November 2, 2006 from: www.pssg.gov.bc.ca/police_services.

Law Commission of Canada (2006). In Search of Security: The Future of Policing in Canada. Retrieved January 23, 2007, retrieved from http://www.policecouncil.ca/reports/LCC2006.pdf.

Maguire, E.R., and Uchida,C.D. (2000). "Measurement and Explanation in the Comparative Study of American Police Organizations," in Measurement and Analysis of Crime and Justice. Volume 4. Washington, US: Department of Justice-National Institute of Justice, Office of Justice Programs.

Maillarda, J., and Savageb, S.P. (2012). Comparing Performance: The Development of Police Performance Management in France and Britain. Policing & Society Vol. 22, No. 4, (December):363-383

Martin, D. L. (2004, June 29). Legal Sites of Executive-police Relations: Core Principles in a Canadian Context. Paper presented at the Ipperwash Inquiry/Osgoode Hall Law School: Symposium on Police/Government Relations, Toronto, Ontario.  Retrieved from: http://www.attorneygeneral.jus.gov.on.ca/inquiries/ipperwash/policy_part/meetings/pdf/Martin.pdf.

Milligan, S.O. and Fridell, L. (2006). Implementing an Agency-level Performance Measurement System: A Guide for Law Enforcement Executives. Police Executive Research Forum, Washington D.C.

Melchers, R. (2001). What Can (and Can't) Statistics Tell us About Crime and Criminal Justice?  Office of the Auditor General of Canada.

Moore, M.H. (2002). Recognizing Value in Policing: The Challenge of Measuring Police Performance. Washington, DC: Police Executive Research Forum.

Moore, M.H., and Braga, A. (2003). The "Bottom Line" of Policing. Police Executive Research Forum.

Moore, M.H., and Braga, A. (2004). Police Performance Measurement: A Normative Framework. Criminal Justice Ethics.

Moore, M.H. and Poethig, M. (1999). "The Police as an Agency of Municipal Government: Implications for Measuring Police Effectiveness." In R.H. Langworthy (ed.), Measuring What Matters: Proceedings from the Policing Research Institute Meetings (pp. 151-168), Washington, D.C., National Institute of Justice and the Office of Community Oriented Policing Services.

Moore, M., Thacher, D., Dodge, A., and Moore, T. ( 2003). Recognizing Value in Policing: The Challenge of Measuring Police Performance. Washington: PERF.

Mukherjee, A. (1990). Let's Tackle the Mismatches! Speaking Notes. Plenary Session: Crisis in Policing August 14.

Murphy, C. and McKenna, P. (2007). Rethinking Police Governance, Culture & Management. Prepared for the Task Force on Governance and Cultural Change in the RCMP. Public Safety Canada.

Neely A., and Adams C. (2001). Perspectives on Performance: The Performance Prism. Cranfield School of Management.

Neely, A.D., Gregory, M.J., and Platts, K.W. (1995). Performance Measurement System Design: A Literature Review and Research Agenda. International Journal of Operations and Production Management, Vol. 15, No. 4:80-116

OECD (1997). In Search of Results: Performance Management Practices. Paris: OECD.

OMBI (2011). Data Dictionary for Core Framework and MPMP Measures Police Services.

Ontario, Municipal Act (2001).  Retrieved from: http://www.e‑laws.gov.on.ca/html/statutes/english/elaws_statutes_01m25_e.htm.

Ontario, Ministry of Municipal Affairs and Housing (2007). Municipal Performance Measurement Program: Handbook. Retrieved from: http://www.mah.gov.on.ca/AssetFactory.aspx?did=4873.

Ontario, Police Services Act (1999). Ontario Regulation 3/99 (Adequacy and Effectiveness of Police Services). Retrieved from: http://www.e‑laws.gov.on.ca/html/regs/english/elaws_regs_970421_e.htm.

Pidd, M. (2007). "Perversity in Public Service Performance Measurement," in A. Neely (ed), Business Performance Management: Unifying Theory and Integrating Practice, 2nd Edition. Cambridge, UK: Cambridge University Press.

Porter, Louise E., and Prenzler, T. (2012). Police Oversight in the United Kingdom: The Balance of Independence and Collaboration. International Journal of Law, Crime and Justice, Vol. 30, Issue 3:152-171.

Statistics Canada (2011). Police Resources in Canada. Catalogue no. 85-225-X. Retrieved from: http://www.statcan.gc.ca/pub/85-225-x/85-225-x2011000-eng.pdf.

Stenning, P.C. (2004, June 29). The Idea of the Political "Independence" of the Police: International Interpretations and Experiences. Paper presented at the Ipperwash Inquiry/Osgoode Hall Law School: Symposium on Police/Government Relations, Toronto, Ontario.  Retrieved from http://www.attorneygeneral.jus.gov.on.ca/inquiries/ipperwash/policy_part/meetings/pdf/Stenning.pdf.

Streib, G.,and Poister, T. (1999). Performance Measurement in Municipal Government: Assessing the State of the Practice. Public Administration Review, Vol. 59, N. 4:325-355.

Talbot, C. (2010). Theories of Performance. London: Oxford Press.

Vollaard, B. (2003). Performance Contracts for Police Forces, CPB Netherlands Bureau for Economic Policy Analysis.

Wang, W. (2008). Convincing Legislators with Performance Measures. International Journal of Public Administration, 21: 654-667.

Walsh, D. and Conway, V. (2011). Police Governance and Accountability: Overview of Current Issues. Crime, Law and Social Change, 55,2-3: 61-86.

Appendix A: Additional Tables

Source: Ontario, Ministry of Municipal Affairs and Housing (2007).

Appendix B: Methodological Approach

Background

The growing interest in measuring police performance has been fuelled not only by a concern to improve efficiency and cost-effectiveness but also by a concern on the part of police oversight boards and commissions with how to improve the relationship between the police and the communities they serve. Questions are being asked about how police can best determine and apply the most appropriate performance measures to demonstrate organizational objectives such as crime reduction and prevention, improved public safety, and the cost-effective, efficient use of resources. The purpose of this project is to gather and analyze research data on how police boards and policing oversight bodies measure police performance and use the data they collect and analyze to better serve the public.

Project Objectives

In accordance with the RFP, the objectives are as follows:

Analytic Approach

To conduct a review on the use of police performance metrics by oversight boards, a multifaceted qualitative and quantitative analytic approach was taken that includes the following:

Key Questions

The following key questions were identified to help guide the direction for this project.

Sample – Purposive Sampling Technique

Currently, in Canada, there are over 230 police Boards monitoring similar kinds of policing services. As it is not economically and practically feasible to study every police board, it is necessary to employ a randomization process for sample selection to avoid selectivity bias and ensure there is no preferential treatment in the selection of any particular police board. Consequently, a purposive (non-probability) sampling technique was employed using the following criteria to attain some variation in the sampling of Police Boards in terms of the population size and other demographic characteristics (e.g., rural, large urban, small urban)^{Footnote 21}of the communities they serve.

• Size and complexity/scope of the police organization and its services
• Police organizations located in census metropolitan areas^{Footnote 22}
• Catchment area of the police services (regional vs. municipal)
• Jurisdictional leadership on performance measurement at municipal level (i.e., provincial legislative requirements for performance reporting or municipal innovative practices)
• Direct delivery policing services vs. contracted policing services

The sample will only focus on police members of the police oversight bodies (e.g., chair, deputy chair, and board members), and the equivalent provincial government bodies that are responsible for police performance or setting the standards/policies on police performance.

Sample Size

Table 1 gives an overview of the purposive sample selected for this project. The sample includes

29 of the largest municipal policing organizations, three provincial policing bodies, 11 policing bodies serving rural or small municipalities/towns, and five First Nations policing bodies for a total of 48 police boards. If, for any reason, any of these organizations are unable to participate in the study, additional police organizations may be randomly selected.

In addition, 12 provincial Organizations responsible for Police Oversight/Police Performance Mandate and four national associations representing police boards and police services were included in the sample.

Interviews

It was proposed to conduct semi-structured interviews with 48 or more police oversight bodies. These organizations were identified in Table 1. Representatives of three national associations representing the police boards and policing services will also be interviewed. The purpose of these interviews is to gain understanding of their knowledge of and views on the police performance metrics used in their respective jurisdictions.

Thus, semi-structured interviews are proposed for the following individuals/organizations.

• Provincial Police Organizations - 3
• Rural Police Forces' oversight board/commission – 11
• Police Services (large) Oversight board/commission – 29
• First Nations Police Oversight board/commission – 5
• Provincial Organizations responsible for Police Oversight/Police Performance Mandate – 12
• National Associations – 2
• Several follow up interviews

Total interviews proposed – 62

To complete these semi-structured interviews, two interview guides are being developed. The first guide will be for board members (e.g., chair, vice chair) sitting on the police boards. The semi-structured interview guide will provide an array of questions focusing on the use of performance metrics or other measures the police board may require from their policing services. The questions will be based on the research (e.g., literature review, review of documentation) gathered on the performance measurement metrics that the policing services are mandated to collect or are collecting as part of a performance measurement initiative.

The second interview guide will be used to interview representatives of provincial government departments or agencies that have mandated the use of performance metrics for assessing the performance of police services. The focus of these questions will be on whether performance metrics are legislatively mandated or required through policies and how the performance metrics being used were selected. The guide will also be used for the national associations where the focus of the questions asked will be on the overall objectives of performance metrics in policing.

The interview guides will be developed and shared with the project authority for their input. Each interview will be approximately 35 minutes. As part of the interviews, interviewees will be asked to provide supporting documentation.

The potential interviewees will receive a package of information. This will include an introductory letter (attached) from the project authority explaining the nature of the project. There will also be a voluntary consent form (attached) where the interviewee will be asked to review the information and sign it to confirm their participation. The interviewees will receive the interview guide in advance.

Collection of Data Sets

A content coding analysis sheet was developed that included:

1. Overview of the performance measures or metrics;
2. How the information was communicated (e.g., text, table, graph);
3. The type of performance reporting document;
4. How the performance measures or indicators were identified or categorized (e.g., the input-output –outcome model, 3E concepts: economy, effectiveness, efficiency);
5. Data Source(s) for the measures;
6. Calculation used or expressed (e.g., percentages, units/numbers);
7. Integers identified (numerator or denominator).

To identify printed data items, the police boards and police services' websites were searched using words "performance measures" or "metrics". Documents such as business or strategic plans for the current year (or the most current printed document available) were reviewed. While it was expected that respondent Boards would provide additional documentation for review (e.g., minutes of meetings or reports not located on the website), this was not the case.

Analysis of Data

Both quantitative and qualitative analysis will be used to analyze the information collected.

The quantitative analysis will focus on reviewing performance metrics data (e.g., ratios of front-line officers per 1,000 population, clearance rates, operational expenditures/unit costs, number of meetings with community groups, benchmarking activities) that the organizations identified consider useful for evaluating how well performance goals are being met. Where possible the information from the semi-structured interviews will be inserted into an MS Excel spreadsheet. This information can be contextualized through descriptive data analysis. It is anticipated that key information can be obtained electronically either from various police commissions or from Statistics Canada.

Our qualitative analysis will be based on interviews and supporting documents provided by interviewees.

Where open ended questions are used in the interviews textual analysis will applied to identify meaningful patterns in the information obtained. The textual analysis will identify response categories (e.g., police agency, local police commission), classify the data by placing each response into one of the categories identified, tabulate frequencies of data/text responses, identify composite responses (e.g., quotes), and report the information. The information will be analyzed at an aggregate level so that individuals cannot be identified (i.e., five representatives from oversight organizations claimed ...). Further information on textual analysis will be provided upon the awarding of the contract.

The research team will be conducting a three-part content analysis of the documents and other reports provided by the organizations. First, "open coding" will be used to determine general themes related to performance measurement. Second, a process of "axial-coding" will take place whereby the consultant will review the documents and tag specific passages as belonging under one of the categories or themes established via the open coding. Third, "selective coding" will be used to determine if there are any miscoded passages and discrepant information and to correct them as necessary. This process will involve two consultants. Where possible, information will be placed in tables and diagrams.

To descriptively analyze the information collected from the content analysis, a code sheet was prepared (with categories related to workload, resources, and performance metrics) to allow the research team to tally the frequencies with which indicators of particular issues or themes are present or absent. For the analysis it is assumed that such a tally will be a valid indication of the perceived value or importance of any particular item. The tally will include the reporting of raw counts of the number of terms or concepts identified in the document review. Attempts will also be made to compare police Boards (rural vs. urban), provinces/regions, the weight or importance given to specific performance metrics, and the existence of trends. Tables and graphs will be used to display the information. An attempt will also be made to re-examine the data collected from the annual reports and to regroup them in terms of categories such as input and output, process, and outcome or impact. Key narrative passages or quotes will also be identified and used to provide additional nuance to the analysis. Some of these passages will be highlighted in text boxes.

There may be a need, however, to seek some of the information from the Police Services. If this is the case, the oversight board will be asked to obtain this information for the research team.

Table 2 provides a summary of the proposed data to be collected.

Data Retention and Destruction

Data retention and destruction will follow the policies as established by the Panel on Research Ethics. In this particular case, the data and other information collected will be included in a report for delivery at a national conference on the economics of policing. The data will be retained until the completion of the research contract. Hard copies of information will be located in secure filing cabinet (Locked with a bar). Electronic information will be stored on a computer system that is password protected. Upon acceptance of the final deliverable by the Project Authority, the raw data will be destroyed and electronic information will be erased (including the deleting of information from the recycle bin and IPS caches/cookies). Hard copies will be destroyed using the consultant's shredder and any electronic information will be erased.

Respondent Anonymity and Confidentiality

Information gathered from interviews will be analyzed aggregately to ensure the respondent's anonymity. In terms of coding information, research identification numbers will be assigned for each respondent. The number assigned will be the only information kept with the interview notes. This is designed to ensure the information gathered is kept confidential Furthermore, there will be a limited number of individuals who will handle the confidential information and those who will have access to confidential information have experience in ensuring privacy and confidentiality. As part of the voluntary consent form as well as on the interview guide, participants will be informed that their information will be kept confidential. Information will only be gathered from individuals who have signed the consent form.

Practical and Sociopolitical Considerations

In the area of performance metrics there are several practical and sociopolitical considerations.

First, the research literature has pointed out that there are several limitations of performance measurement. Performance measurement programs by themselves are not appropriate for assessing outcomes, for determining future directions or for resource allocation. They are however an important component of a comprehensive evaluation strategy. The purpose of the study is to examine how performance metrics are developed and used and not necessarily to look at police results or police accountability.

Second, the nature of performance measurements (e.g., the set of metrics used) by itself can foster complications. For example, across the regions, there may be a lack of agreement on the definitions used thereby creating confusion and limiting the potential for the generalization and comparability of the information. Thus, there will be a need to point out these differences in the report and the limitations in any analysis. There is also a need to provide clarification of key terms.

Third, the objectives of performance measurements may vary among those being interviewed. For example, the police chief may be using performance measurements to determine the outcomes of a particular program, the oversight board/commission may be looking at it as a way of assessing the overall performance of the Chief and other senior officers, and the government agency may be using it as part of an overall government mandate to provide more accountability. As there may be some differences with regard to the purpose of performance measures, there is a need to be sensitive to these differences in terms of conducting and analyzing interviews and writing the report.

Fourth, while several police boards will be approached for interviews, a challenge may be that a particular member of the board or even the entire board may not consent to participate. Some individuals may refuse to participate because they feel they either lack sufficient knowledge or competency to talk about performance metrics or because they do not feel comfortable with the interviewing process. Other potential reasons for lack of participation may be that some individuals may feel they do not have enough time because of other commitments. For example, Fall is a very busy time for individuals involved in municipal budgetary processes. To gain access to these individuals, the researcher will explain the importance of this research project and explain that not knowing about the topic is by itself a very important finding for the project. This is also important as the lack of participation may result in a low response rate.

Fifth is the issue of non-response bias. This can occur when the opinions and perception of survey respondents do not accurately represent the overall survey sample. In this case, responses from some of the police boards may differ from the potential answers of those who did not participate in the interviews. Larger police boards that have civilian support staff or police officers assigned to work specifically on performance metrics may have more time to participate in an interview than do smaller police boards lacking such staff. In this instance, the non-response bias will weigh in favour of larger urban or regional police boards/policing services as opposed to smaller rural or municipal police boards. If non-response bias becomes apparent, there will be a need to examine the non-response rate and differences between respondents and non-respondents and conduct a statistical test to compare the differences. To overcome non-response bias, the researchers will work diligently to get the responses from all potential participants. The letters sent to all the board will state a response deadline to give participants time to consider whether they will participate in the interview. Post notification via e-mail or follow up contact via telephone will also be done to ensure a high response rate.

A sixth consideration involves the relationship between performance measurements and how services are delivered. Some performance indicators partly rely on the actions of other agencies. This reflects the inter-dependence existing between some policing services and other community service agencies that allow more to be achieved through some form of partnership than through going at it alone.

A seventh consideration is how the implementation of police performance measures might affect police labour relations. Generally, police unions may view performance measures with a certain amount of scepticism as the development of performance measures may be thought to lead to a reduction of human and financial resources. Most police union focus has been on the rating of individual performance as opposed to the use of organizational performance metrics. Under police union bargaining agreements, the way unions might react is an important part of any discussion on using performance measures that could lead to organizational changes resulting in cuts to police personnel. There may also be union resistance to performance measures because of the diffusion of authority inherent in their existing management union agreements. To address this issue, police boards will need to consult with police labour organizations and work creatively with them. For example, some of the documents reviewed on the development of police performance indicators note that some police services have established working groups that include the involvement of police unions.

Eighth, and finally, consideration should be given to the linking of performance measures to financial rewards or to the competition for scarce resources especially when there is a demand for departments to find ways to cut costs while maintaining or increasing existing performance levels. Competition between government departments in the bureaucracy is often linked to concerns with improving organizational performance and receiving recognition for such improvements from senior management and from one's political masters. These officials are likely to view more positively those departments who effectively present information based on performance metrics than those departments that cannot do so. A beneficial consequence of the competition among departments to achieve such recognition could be an increased commitment on the part of all departments to improve their use of performance measurements. In this case, elected officials and the public would both benefit from the department's improved performance assessments in decision making with regard to the allocation of resources.^{Footnote 24}

Image Description

Public Safety Canada

November 8, 2012

Dear Sir or Madam,

As you may be aware, in January 2012 Federal/Provincial/Territorial (FPT) Ministers Responsible for Justice and Public Safety endorsed holding a Summit on the Economics of Policing. The Summit, which will take place in January 2013, will be hosted by Public Safety Canada. All levels of government, as well as police stakeholders, are involved in the planning of this Summit.

As part of the work underway on the Economics of Policing, Public Safety Canada has contracted the services of the Compliance Strategy Group to undertake a study that will look at police performance metrics and how local police commissions and boards view these measurements. The objectives of this project are to:

1. describe the performance metrics that are used by policing bodies in Canada;
2. assess how these performance metrics are viewed by police services boards and commissions; and
3. identify on what types of performance metrics police oversight bodies want policing bodies to report. 

The role of data collection through various sources is central to this project. Accordingly, a sample of police boards and oversight bodies at the municipal, provincial and national levels will be randomly selected to participate in this study. The research team will also be gathering available published data from law enforcement agencies on the police performance metrics they use (i.e., ratio of front-line officers per 1,000 population, clearance rates, number of meetings with community groups, etc.).

Mr. John Kiedrowski, President of Compliance Strategy Group and his colleagues, will contact randomly selected police service boards and oversight bodies to participate in this study. Should your organization be randomly selected, I ask that you encourage your officials to participate in the telephone interviews and provide the necessary information to ensure the success of this project. It should be noted that Mr. Kiedrowski and his colleagues hold the security clearances required to handle information provided by your organization.  They are required to store protected information in accordance with the policies of the Canadian Industrial Security Directorate, Public Works and Government Services Canada. The final products created as part of this study will not include comments attributable to individuals and all interview notes will be destroyed once the research study is finalized.

Please feel free to contact the Project Coordinator, Mr. John Kiedrowski, Compliance Strategy Group, at any point during the research project at (613) 851-2056 or jk19@rogers.com. If you require further information that cannot be provided by a member of Mr. Kiedrowski's team, please contact Mr. Austin Lawrence, Manager of Research, Law Enforcement and Policing Branch, at (613) 949-9574 or austin.lawrence@ps-sp.gc.ca.

I thank you in advance for your cooperation and contribution to this project. Studies such as this one is very beneficial to the policing community in Canada and will assist us as we move forward, together, in strengthening Canada's policing advantage.

Sincerely,

Julie Mugford, Director
Research and National Coordination, Organized Crime Division
Law Enforcement and Policing Branch
Public Safety Canada

cc:
Anita Dagenais, Senior Director
RCMP Policy Division
Policing Policy Directorate

Interview Guide for Police Board Members

Introduction

Good morning/afternoon. My name is _________________ . I am calling from Compliance Strategy Group which was an awarded a project from Public Safety Canada to study how police Boards and commissions use performance measures*. As part of this study, I would like to interview you to determine your Police Service Board's use of police performance metrics. The information gathered will be developed into a paper for the Summit on Economics of Policing in January 2013.

The interview will last approximately 35-40 minutes. Before I begin, I need to go over a few points.

• You should have already received an advance copy of the overview of the project and interview guide so you will have some ideas on the type of questions that will be asked.
• I have received your signed copy of Letter to Consent to the interview.—Thank you.
• Your participation is anonymous; anything you say will not be attributed to you personally.
• Any notes I take during the interview will be destroyed at the end of the project. Only the researchers will have access to the raw interview materials.
• At the end of the interview, I will also ask for copies of related reports and documents. Most of these will be public documents, although we do have a level of security clearance to assure secure handling of any internal or restricted documents. All information we collect (both interview notes and documents) will be stored in a locked cabinet and only the researchers will have access to the raw interview materials.

Do you have any questions? If you don't have any (more) questions, may I begin the interview.

Background Information

The first three questions focus on some background information.

1. The following information focuses on some background information:

• Name of Individual Completing Interview:
• Position:
• Address:
• Work Telephone Number:
• E-mail address:

2. What is the number of personnel (sworn and Civilian) of the police services the Police Board governs?

3. How would you characterize the delivery of police services under your mandate? Policing services can be described as:

• ___ Urban policing services
• ___ Regional policing services
• ___ Rural policing services
• ___ Combination of urban and rural policing services with urban policing being a majority of services
• ___ Provides policing services under a mutual aid agreement
• ___ Policing services are provided under a provincial contract (e.g., RCMP, OPP, QPP)

Police Performance Metrics General Information

I now want to focus on some general information on police performance measurement models and metrics.

4. Does your Police Board mandate the use of police performance metrics/measurement?
If yes, how was this authorized?
If no, why not

5. To what extent are the provincial agencies that are responsible for policing or municipal services involved in the police performance metrics?

Overview of Performance Metrics

The following questions focus on the specific measures the police service presents to the Board to demonstrate results or outcomes.

6. What policing activities, services or functions are the focus of performance measures?

7. Why are these performance measures collected or for what purpose are they intended?

8. For each of these performance measures what specific information is collected, how is it collected and by whom?

9. For each of these performance measures you identified, in your view, how satisfied is the Board with the quality of the performance metrics selected?

10.  Is there any room for improvement? If yes, where?

11.  What policing activities or services are NOT covered by performance measures? (give as much detail as possible)

12.  Does your performance metrics model address how the Police Service works in partnership with other stakeholders such as Fire, EMS and the Courts?
If yes, which ones,
If no, why not

13.  Does your performance metrics model address how the Police Service works in partnership with community groups?
If yes, which ones,
If no, why not

14.  During the past 5 years, have the performance measures changed? If yes, how have these measures changed?

Police Board Involvement in Performance Measures

15.  How are these performance measures presented to the Board? For example are they incorporated in the police service's budgets, business plan, or strategic plans documents?

16.  In your view, how would you rate Board members' collective understanding of the information given to them on police performance?
___ A Great Deal___ Much___ Somewhat___ Little___ None___ Other please explain

17.  Are there currently resources available to provide support to the Board on verifying if police performance metrics are accurate or measure what they purport to measure?

18.  Once this information on performance is presented, what decisions or actions are taken by the Board?  For example, is the information used as part of the decision making process to adjust the delivery of policing services or is the information used to increase/reduce budgets?

19.  To what extent is the Board involved to identify or propose new performance metrics?
___ A Great Deal___ Much___ Somewhat___ Little___ None___ Other please explain

20.  Does the Police Board have a process in place to review, challenge, validate or improve existing police performance indicators or targets?
If yes, How does this process occur?
If no, please explain

21.  Are there any other comments you want to make regarding performance metrics in the policing context?

22.  Can I please have copies of any of the following report for the last three years where performance measures were presented or discussed:
___ Annual Reports ___ Strategic Plans, ___ Minutes to Meetings, ___ Police Services Reports on Performance Measurement, ___ Research Reports (internal and external), ___ Any other reports or information that will help to substantiate anything information presented in this interview.

Thank you for taking time to do this interview. you input is very much appreciated as this topic is an important and growing area of research.

Interview Guide for Government Officials and Stakeholders

Introduction

Good morning/afternoon. My name is _________________ . I am calling from Compliance Strategy Group who was an awarded a project from Public Safety Canada to study police performance metrics used by police boards. I want to conduct an interview with your Department/Agency on how police performance metrics are being applied by local police commissions and boards. The information used will be developed into a paper for the Summit on Economics of Policing in January 2013.

The interview will last approximately 35-40 minutes. Before I begin, I need to go over a few points.

• You should have already received an advance copy of the, overview of the project and interview guide so you will have some ideas on the type of questions that will be asked.
• I have received your signed copy of Letter to Consent to the interview.—Thank you.
• Your participation is anonymous; anything you say will not be attributed to you personally.
• Any notes I take during the interview will be destroyed at the end of the project. Only the researchers will have access to the raw interview materials.
• At the end of the interview, I will also ask for copies of related reports and documents. Most of these will be public documents, although we do have a level of security clearance to assure secure handling of any internal or restricted documents. It is important to point out that all information (interview notes and documents) will be stored in a locked cabinet and only the researchers will have access to the raw interview materials.

Do you have any questions? If you don't have any (more) questions, may I begin the interview.

Background Information

I am going to just ask some questions to give us some background information.

1. The following information focuses on some background information:

• Name of Individual Completing Interview:
• Position:
• Address:
• Work Telephone Number:
• E-mail address:

2. Which organizational structures or entities are responsible establishing police performance measures in your Department/Agency?  For example, is it the sole responsibility of the Department responsible for policing services or is it the responsibility of the municipal government or shared?  If shared, how is this accomplished?

3. What are the authorities for police performance measures?  Are they legislated by statutes or by policy (which ones)?  (Please provide supporting documentation/references)

4. How best would you describe the police performance measurement model that was implemented in the municipality/province?

The Implementation of Performance Metrics

The following questions focus on the specific measures the have been established.

5. What are the specific police performance tools and measurements used (i.e., collected, complied and communicated) in the province?

6. How are these police performance measures operationally defined (i.e., what concrete information is collected and compiled)?

7. How were these performance measures selected?  For example, are these performance measures based on existing measures, or a review of literature?

8. Under each of the police performance measures, what is the specific data or information that needs to be collected?

9. What have been some of the challenges associated with these performance measures?  Are there any concerns about the validity and reliability of the measures selected?

10.  What other police performance measures should be included?

11.  How is the performance metric system that is in place communicated to the police boards?  For example, are there training seminars.

12.  Is there a process in place to determine the impact of these performance metrics in terms of improving the police board's decision making in terms of resources, policies, or practices?

13.  Are there any other comments you want to make regarding performance metrics?

Thank you for taking time to do this interview. You input is very much appreciated as this topic is an important and growing area of research.

For further information contact

Date modified:

2022-08-10