Managing Departments for Results and Managing Horizontal Issues for Results

Main Points

20.1 In our examination of five large federal departments and 10 programs, we found that several departments have made tangible efforts to focus on results. There was evidence of commitment and support by senior management, extensive planning activities, identification of key performance indicators and some measuring against them.

20.2 However, we found that only 4 of the 10 programs have information on results that is used routinely. In most departments we examined, the persistent state of planning for results - rather than actually managing for results - has meant that progress in using information on results has been limited since we reported on this issue in 1997. It is time for departments to move beyond planning. There is a need for more widespread measurement of the results they accomplish and use of that information to improve programs.

20.3 The government needs to ensure that the evaluation function plays a more central role in its move toward managing for results. The evaluation capacity in many departments has atrophied at a time when more effort, not less, is needed to measure the results of programs.

20.4 More and more, the government needs to manage initiatives that span two or more federal departments. "Horizontal" issues like the Federal Disability Agenda are often key priorities of government. Our study of three horizontal issues found that, like many departmental programs we audited, they were not being managed for results.

20.5 Over the past several years, the government has been clear about the importance of making results central to good management and accountability, and this vision is increasingly being accepted across government. However, bringing that recognition into practice will require central agencies to play a stronger role. They need to provide encouragement and support for departments to move beyond merely planning for results. We found the Treasury Board Secretariat's leadership too limited and its support too dispersed to be of real help to departments trying to manage for results. Nor does the Secretariat have a strategy for dealing with horizontal issues. It needs a strategic approach to these issues, given their growing presence in government.

Background and other observations

20.6 Managing for results allows managers to make changes once they know what is working and what is not. It represents a significant difference in the way government programs are managed. Managers can then pay more attention to finding out whether programs are meeting their objectives and less to only carrying out activities or setting up structures and processes.

20.7 The concept of managing for results is now widely accepted across government. As a first step in putting it into practice, most departments had to realign their management structures and processes according to the results they were seeking, and not to their programs or activities. Each department had to establish department-wide objectives and think through how it wanted to structure itself in terms of the results it expected. Most of the departments we audited were engaged in significant efforts to plan how they would manage for results.

20.8 In managing a horizontal issue for results, it is critical to have a co-ordinating function that is supported by senior management (particularly in a lead department) and that has enough resources to do the job.

20.9 Departments receive limited practical guidance on managing for results, and there is little sharing of experience. They need more encouragement to move beyond planning for results; there is also a need to identify champions.

20.10 While managing for results presents challenges, there are known, practical ways to deal with them. They require that people embrace the idea of learning from experience - which includes learning from mistakes. It involves bringing more rigour to understanding past experiences and then adjusting practices in light of this learning.

The response of the Treasury Board Secretariat, on behalf of the government and the departments audited, is included at the end of this chapter. The Secretariat accepts our recommendations and indicates the actions that it is taking or intends to take to address them.

Managing Departments for Results: Not There Yet

Introduction

20.11 Canadians continue to expect government to manage tax dollars efficiently and to ensure that they are getting value for their taxes. To manage public resources effectively toward an intended result, government officials need to have credible information on the performance of the programs and services they manage. This is the information they will use to determine whether the results they expect are being achieved, and whether their programs are working well or need to be modified.

20.12 The government needs similar information to account for the funds and authorities it has received from Parliament. Chapter 19 of this Report discusses the progress the government is making toward better reporting of performance information to Parliament.

Government Has Committed to a Long-Term Focus on Results

20.13 As early as 1976, the Treasury Board Secretariat promoted performance measurement as a basis for planning, decision making and improving management practices. During the 1980s, program evaluation was given high prominence as a management tool to improve program performance. In 1995, the government launched the current move toward a results-based approach to management. The President of the Treasury Board announced in his first annual report to Parliament, Strengthening Government Review (1995), that the government was committed to delivering programs that work, by moving from a focus on rules and processes to a focus on results. He said that government wants to create a management culture that is fact-based, results-oriented, open and accountable. To deliver more efficient and more affordable programs and services, the government has embraced the principles of service quality, managing for results and continuous learning.

20.14 The President of the Treasury Board has reported annually to Parliament on the government's progress toward managing for results. The government has reiterated its commitment to a public service focussed on results in the Treasury Board Secretariat's publication, Results for Canadians: A Management Framework for the Government of Canada 2000.

20.15 In that publication, the Secretariat confirmed that it has a key role of leadership and facilitation to assist departments in achieving desired results. It aims to provide "whole-of-government" advice to ensure that Parliament and Canadians have the information they need to hold the government to account. These efforts should promote greater responsiveness and a focus on results in government operations.

20.16 The government has undertaken a number of reforms since the launch of the "managing for results" initiative in 1995. These include the reform of Estimates documents, the Modernization of Comptrollership Initiative, the Financial Information Strategy (FIS), an increased emphasis on service quality, and guidelines on preparing Treasury Board submissions that support managing for results. All of these emphasize and encourage a focus on results.

What Is "Managing for Results"?

20.17 Despite the government's commitment to managing for results, it continues to be a challenge and the concept still is not always understood. In our 1997 Report, Chapter 11, we discussed the government's efforts to move toward managing for results:

In the past, managers were primarily held accountable for the prudent use of the resources they were given, the authorities they used and the activities they carried out....This did not encourage a focus on results produced by the resources; rather, compliance dominated managers' attention (paragraph 11.9).

When managing for results, ministers, senior officials, and managers... make decisions based on what a program is achieving for Canadians - the results that citizens value - and at what cost....Holding managers accountable for results encourages them to focus more on results. (11.7-11.9)

20.18 A variety of terms are used to describe managing for results, including results-based management - frequently used in government publications - and performance-based management. These terms are essentially interchangeable, but for consistency we will use managing for results.

20.19 Managing public sector programs and organizations is a complex endeavour. There are several objectives to achieve, many information requirements to meet, external factors to take into account, public scrutiny to withstand, and many unknowns to cope with. Managing requires long-term strategic planning toward stated objectives while operating complex systems and processes day to day. Managers need to step back from time to time, and review activities and the outputs of activities to see if they are still the most appropriate to reach the long-term objectives. At the same time, as stated in Results for Canadians, managers must also be able to "ensure fairness, equity, and reasonableness of treatment to protect the broad interests of citizens." These enduring public sector values are part of the results governments seek to achieve. Exhibit 20.1 illustrates the main elements of managing an organization for results.

20.20 The exhibit illustrates the traditional cycle of planning, implementing, monitoring, reporting, and revising plans, but it adds a key component, learning: making decisions based on empirical information. This is at the heart of managing for results. By monitoring the results of its activities through a variety of means (evaluations, audits, performance measures, scans), an organization can learn what is working and what is not working. These activities can also provide information about other factors outside its control.

20.21 For example, monitoring will indicate whether AIDS information campaigns are helping to change behaviour; whether agri-food research and development activities are generating a positive return on investment (crops with higher yield and quality) while conserving the environment; and whether conservation programs are sufficiently reducing the risks to endangered species.

20.22 Based on information gathered through monitoring, the organization can then either modify its current operations or, more fundamentally, revise its business and strategic objectives. The organization can report to external bodies such as Parliament on what it has accomplished. Managers of government programs need information from the same sources if they are to manage for results and gauge their progress toward stated objectives. Gathering and assessing the relevant data and information - the monitoring element in Exhibit 20.1 - requires systematic effort and a variety of monitoring tools.

20.23 A critical step in moving toward managing for results is to align a department's various strategic objectives with the major groupings of its programs and services (often called business lines) and then with its individual programs. It is essential that departments clearly understand how the various program and service delivery activities support, and will lead to, the achievement of their higher-level or strategic objectives. Many departments find this a daunting challenge.

An overview of results and performance concepts

20.24 Discussions of performance and managing for results can be confusing, as they involve a number of specific terms and several concepts that are important to distinguish. Exhibit 20.2 presents some of the key terms and concepts.

What We Found in 1997

20.25 We reported in 1997 that managing for results was not widespread in government at that time. There were a number of government initiatives that supported a focus on results, but concerted effort was needed to bring about real change. We identified a few cases where managers were measuring results, communicating information on results and using that information to improve results. However, across government the practice of managing for results appeared quite limited and we wanted to encourage its progress. We identified lessons learned, among them key factors that had supported managing for results; they included the leadership and commitment of senior management and an organizational climate that encouraged managing for results. We concluded that ongoing attention was needed across the government if managing for results was to be adopted throughout government.

Focus of the audit

20.26 Our current audit looked at the progress a number of major departments have made in managing for results. The audit was in part a follow-up on the cases we reported in 1997, Chapter 11. This time our objectives were to:

• examine the extent to which selected departments are managing for results, including their use of information on performance to manage programs;
• identify lessons learned in managing for results; and
• assess the leadership the Treasury Board Secretariat has provided to support managing for results throughout government.

20.27 During this audit, we examined five large departments, covering a variety of program types. In each department we asked officials to help identify areas where the use of performance information could be expected. Exhibit 20.3 lists the programs and business lines that we selected (see Appendix A for background descriptions). We examined them to determine to what extent performance information is actually used and to what extent other elements of managing for results are present. We also sought to identify the lessons these departments have learned in managing for results, as well as the obstacles they have faced.

20.28 As follow-up to our 1996 Report, Chapter 3, on federal program evaluation, we also examined how the same five departments plan for evaluation and use evaluation results to improve programs.

20.29 The second part of this chapter describes a study that focussed on managing horizontal issues for results. We identified the challenges that horizontal issues present and some factors that have helped (and hindered) in managing them for, and reporting, results.

20.30 Further details on our objectives, criteria and approach can be found at the end of the chapter in About the Audit.

Observations and Recommendations

What Happened to Our 1997 Cases?

20.31 We followed up on the cases discussed in our 1997 Report, Chapter 11 namely, the North American Waterfowl Management Plan (Environment Canada), Investigation and Control (Human Resources Development Canada), Canadian Centre for Mineral and Energy Technology (Natural Resources Canada), and the Travellers Program (Canada Customs and Revenue Agency). Our key finding was that these programs have continued since 1997 to manage for results. Appendix B provides further details on our follow-up.

20.32 Our 1997 Report emphasized the importance of fostering an approach of managing for results in an organization, and the important part that senior leadership plays in this. Over time, programs change and so do leaders. Indeed, each of our case studies has changed since 1997, some considerably, and all of them have seen a change of leadership. Our follow-up observations suggest that if implanted well enough in an organization, managing for results can withstand change and continue, even as programs adapt to new circumstances.

A Framework for Department-Wide Managing for Results

20.33 Based on a review of the relevant literature and on the several specific cases we examined in the federal government, we presented a framework in 1997 for successfully managing a program for results. The framework had four elements:

• fostering an organizational climate that encourages managing for results;
• agreeing on expected results;
• measuring actual results to improve performance; and
• effectively reporting performance.

20.34 We think this framework is still valid to use in managing programs for results. However, based on what we found in our current examination of department-wide efforts, we have modified the framework slightly. We have highlighted the need to manage for results department-wide. We have also pointed to the need to use performance information for improving programs and identifying good practices (see Exhibit 20.4).

20.35 In addition, we have highlighted the need to develop and reach agreement in the department on a strategic results framework. Reaching agreement on the broad grouping of results that a department is trying to achieve is an important step in organizing to manage for results. A department needs its key players to agree on the strategic results it intends to achieve, on the general strategies it will use, and on the way it will structure itself to achieve those results. We observed that departments are using their business line structure - the planning, reporting and accountability structure - to link the department's internal management and accountability regime to its objectives, business lines, resource requirements and performance targets.

Managing for Results Still Not an Integral Part of Managing Departments

What we expected

20.36 In the three years since our 1997 study, the government has remained committed to a focus on results. In 1997, we suggested it could take the government four to five years to shift its focus from managing activities to using performance information routinely for better management of programs.

20.37 This time in our audit, we expected to find that the practice of managing for results had advanced, with its key elements evident in many areas of government. We looked for signs of senior support, performance expectations that were well articulated and understood, performance indicators, and systems for collecting and measuring data. We particularly expected to see performance being monitored in many government programs and the information being used to improve the management of operations and policy.

More attention to results

20.38 Overall, we saw tangible efforts in all the audited departments to foster a climate of results, including the strong commitment and support of senior management. Most of the departments were engaged in significant efforts to plan how they would manage for results. All the programs we examined were developing or fine-tuning performance measures.

20.39 We reported in 1997 that managing for results was not widespread across government and had made limited progress. Now there is more attention to results. All government departments now set out their key results commitments, which are used as the basis for their planning, reporting and accountability structure. In the departments we examined however, the actual measurement and use of performance information has made limited progress.

20.40 Alignment for results is needed. As we have indicated earlier, a department needs its senior managers to understand and agree on its key results commitments and how it will structure itself to achieve them. During our audit, we have found that departments, to focus on results, often need to realign their management structures and processes according to the key results they are seeking and not to their programs or activities. This can be challenging.

20.41 The department needs to begin to consider how to structure itself in terms of its key results commitments. This may require changing the organizational structure or establishing business lines that span parts or all of the organization. Departments frequently manage by business line, each headed by an assistant deputy minister who is responsible for the key results commitments that contribute to the agreed department-wide strategic results framework.

Use of performance information is limited

20.42 Our audit looked at how performance information is used at the department-wide or corporate level. We found little use of performance information to improve management, or to shift priorities and reallocate resources across business lines. Business lines in the five departments we examined are quite large and involve sizable groupings of departmental activities and programs. Perhaps as a result, these business lines are often managed separately by the deputy minister and the responsible assistant deputy minister.

20.43 In two of the five departments (Agriculture and Agri-Food Canada and Correctional Service Canada), we found that corporate managers receive information regularly on a number of performance indicators. This has allowed the departments to track important trends in performance and to keep informed of key changes in their environment. For example, Agriculture and Agri-Food Canada uses a "dashboard" of performance indicators to identify emerging problems and to signal the possible need for more information or future decisions.

20.44 We also looked at how performance information is used by programs. We asked the five departments to identify the programs most likely to be managing for results and, in particular, most likely to be using performance information. It took some time before most of the departments could identify such programs, often with our help. It was clear that managing for results was not widespread in most of these departments' programs.

20.45 In October this year, we reported our assessment of financial management capabilities in selected departments. A common gap we found again was the limited ability to integrate financial and non-financial information. Most departments indicated that they had only begun the task of integrating financial and operational systems. In this audit, we also found a poor link between results and cost information. In our examination of 10 programs, we found that departmental managers had only limited cost information available.

Planning for Results But Not Enough Implementation

Extensive planning to manage for results

20.46 Developing the tools to use in managing for results is an important step. We found considerable department-wide planning under way in the five departments we reviewed. Common planning activities included developing a performance and/or accountability framework, identifying and revising key indicators of performance to measure, and revising program business plans in light of the department's strategic results framework. The following are some planning activities we noted:

• At Agriculture and Agri-Food Canada, logic charts were being developed to demonstrate the linkages among a program's inputs or activities, its outputs, and its outcomes. Exhibit 20.5 illustrates a logic chart that the Prairie Farm Rehabilitation Administration developed for the Water Aeration of Surface Reservoirs project. The exhibit shows one of three results expected for this project and considered necessary to achieve the long-term benefit: better quality of water in the rural Prairies. The logic chart includes clear statements of intended outcomes; performance measures; and concrete, measurable targets with timeframes.
• In Environment Canada, numerous activities were under way to implement the Department's performance measurement strategy across the business lines. Planning activities in the Nature business line have included selecting performance indicators, defining data collection strategies, and building logic models for key intended results - including "conserving biological diversity", a result delivered primarily by the Department's Canadian Wildlife Service. The Department has stated that in the coming months the business lines will determine what data they need to support the logic models and indicators, and then will implement data collection strategies in a phased approach over the next several years.
• Natural Resources Canada has continued to develop its performance measurement framework, and has focussed on developing indicators of performance. The performance measurement working group has consulted extensively with program managers and stakeholders to review draft performance indicators and seek guidance on refining them. Natural Resources Canada has developed criteria to assess and refine each indicator and establish targets, where appropriate. Many of the current performance indicators have targets that are "directional" (for example, "to maintain or improve Natural Resources Canada's contribution"). The Department has stated that after it evaluates the directional targets, it will be in a position to consider establishing numerical targets.

A need to move beyond planning

20.47 As already noted, we found some progress in measuring performance, but use of the information is still limited. However, we found that planning to manage for results is extensive, especially at the corporate level, and in many cases it has gone on for years. Departmental officials often told us they expect to use performance information more regularly across the department, "soon". Practical use of performance information and costing of results always seem to be promised for "next year".

20.48 Commitment to move beyond the planning stage needs to be demonstrated across the organization. Some departments seem to be searching for the "perfect" departmental structure to manage for results, and they continually change what they have set out. In other departments, a change of deputy minister or minister triggers a reconsideration of the strategic results framework they have been developing.

20.49 In 1997, we reported on the good beginning of managing for results in Environment Canada. It had a strategic results framework in place and appeared ready to implement it across the Department to manage for results. Yet three years later, it is essentially at the same corporate planning stage, with a different results framework in place. It is still trying to agree on performance measures, and for the most part measurement has not yet begun. Industry Canada has been developing its strategic results framework for two years, with numerous iterations. The Department agreed on performance measures only recently. Natural Resources Canada and Agriculture and Agri-Food Canada have also continued to revise their planning tools, including their results frameworks and performance measures. The reasons may differ but the result is the same: not moving quickly enough beyond the phase of designing departmental frameworks that are crucial to the next stage of managing for results.

20.50 The challenge for departmental managers is to move on - to implement the various planning tools, measure against the established indicators and use this information to improve the management of their programs. Managing for results means that managers may change what they do once they know what is working and what is not. This applies equally to developing and using performance measures. Sound measures are those that managers will use over a period of time. If measures provide data that prove to be poor or not useful, the measurement strategy needs to change.

20.51 Senior management of departments should move their organizations beyond merely planning to manage for results. They should get on with measuring performance and using the information to achieve the intended results.

Some Are Managing for Results

20.52 We did note some elements of managing for results in all the selected departments.

Performance information used in some business lines and programs

20.53 In the 10 programs we examined, we did find some use of information on results (see Exhibit 20.6). We found four programs that use performance information routinely to improve their operation and design: Agriculture and Agri-Food Canada's Net Income Stabilization Account; Prairie Farm Rehabilitation Administration; Correctional Service Canada's Reintegration of Offenders; and Natural Resources Canada's Aeronautical and Technical Services. In these programs, the commitment of senior and middle management is evident; indicators of performance are clearly linked to planned results; and the information is used in planning and in program design. Managers consider performance information to be reliable, and useful for decision making. For example, information on results has helped in identifying the need for new programs and redesigning existing programs.

20.54 The six other programs ranged from some use of performance information to none. They used information more to improve program operations and design than to revise performance expectations and reallocate budgets. Overall, the programs we reviewed used performance information less than we had expected.

20.55 A previous audit addressed managing government services for results. Our April 2000 Report, Chapter 1, Service Quality cited several services that were collecting performance information. With the exception of telephone services, however, few of the service lines examined were using performance information to improve service.

20.56 Our follow-up of the 1997 audit of Human Resources Development Canada's transition to managing for results has reported that the Department has continued to make progress (Auditor General's Report, December 2000). In the Employment Insurance program, the protection of public funds against fraud and abuse continues to be managed for results; and key performance information is analyzed more thoroughly, making it more useful to managers.

Other elements of managing for results are present

20.57 Although we found that information on results was being put to only limited practical use, other elements of managing for results were evident in the programs and business lines we examined. Those elements included:

• fostering an organizational climate that encourages managing for results;
• measuring performance; and
• developing a strategic framework that outlines key results commitments and planned performance expectations.

20.58 Fostering an organizational climate for results is key. Senior commitment and leadership are key elements, particularly in moving forward with managing for results, past the planning stage. Those elements need to be supplemented by appropriate incentives that support performance measurement and the use of performance information. More efforts are needed to remove impediments and overcome resistance.

20.59 In some of the programs we examined, we found some form of performance contract among the senior managers that encourages them to focus on results. Performance contracts identified the results expected over the coming years, and were tied to relevant business plans. Performance contracts are a good incentive to manage for results, to the extent that they specify results and not just activities.

20.60 We observed that the pay-at-risk initiative was used in a few departments to focus attention on results and performance reporting. This government-wide initiative was introduced in 1998 to allow managers to earn up to 10 to 20 percent of pay on the basis of performance, measured against targets agreed to at the beginning of the year.

20.61 We found a number of other tangible initiatives to foster a climate that supports managing for results. Some departments recognized their limited capacity and expertise in this area. Some of them addressed it by creating a performance measurement group; others recruited consultants; and some made good use of the experience in their evaluation unit.

20.62 We also observed the following examples of a supportive environment:

• The Commissioner and senior officials of Correctional Service Canada were committed to obtaining information on results and using the information to analyze the results. Evidence of that commitment is the regular monitoring of results, which has pointed to specific programming needs of offenders. For example, the Aboriginal reintegration program is being developed to meet the programming needs of the Aboriginal inmate population. There is strong buy-in from middle managers, who ensure that data systems are maintained; they use performance information in managing their programs. Finally, the Service uses the pay-at-risk initiative as a lever to ensure that managers focus on results, and it closely monitors their performance contracts.
• In Agriculture and Agri-Food Canada, strong commitment and leadership was evident at the levels of deputy minister and assistant deputy ministers. The Department has dedicated resources, with additional Treasury Board funding, to develop a "dashboard" of performance indicators that were closely monitored by the Deputy Minister. In addition, a working group was established to launch the Department's performance measurement initiative. The Assistant Deputy Minister, Corporate Services Branch, provided guidance to the working group and has chaired an advisory committee on performance measurement. The committee members included the assistant deputy minister over each business line, senior managers, and several stakeholders.
• Industry Canada has designated assistant deputy ministers as champions of its "strategic objectives", which correspond to business lines. The Deputy Minister reinforced a strong message to managers that resource allocation among programs should be based on their ability to show results. Dedicated resources were given to the Corporate Planning and Performance Co-ordination Directorate to manage the process of preparing reports on plans and priorities and departmental performance reports. The Directorate represented all sectors of the Department. It also had a mandate to develop and implement a performance measurement strategy and support the corporate vision of results.

20.63 Some progress in measuring performance. Managing for results means measuring outputs and outcomes with a view to learning from experience. We found that at present, it is largely activities and outputs that are measured. Over time, we would expect to see an increase in the measuring of intermediate and end outcomes that are linked to the program activities. In time, we would expect departments to develop cost-effective measurement tools to cost activities, as a basis for allocating resources according to expected results. Finally, evaluation can help in this measurement task.

20.64 The five departments we audited set out a strategy for measuring results, either in the departmental report on plans and priorities or in a separate corporate planning document. We observed some level of performance measurement in the 10 programs we audited:

• Correctional Service Canada routinely measures its accomplishments and results, particularly in its Reintegration business line. The Service has a vast range of indicators and measures that senior managers use to manage this business line. Integrated data systems track performance at the individual level (for example, how each offender is progressing through his/her release plan), at the institution level (make-up of the offender population), and at the regional and national levels (rates of parole revocation and release rates).
• Managers of Industry Canada's strategic objective, "Connectedness", recognize that medium- and longer-term impacts should be measured. Recently developed performance information focusses on the socio-economic impacts of "connectedness" - people's ability to use information and communication technologies and to interact through them. Work continues with the Conference Board of Canada to further develop a "connectedness index", which is focussed on quantifying affordability, access, usage and socio-economic indicators. One of the next steps will be to refine a framework for examining the impacts and outcomes of connectedness.
• Natural Resources Canada's Aeronautical and Technical Services has made significant strides in measuring performance in this largely production-oriented environment. Cost recovery of aeronautical services and products (navigation charts and publications) has meant extensive tracking of cost information and a strong client focus. The program has implemented principles of a quality management framework to assure product quality, efficiencies, and responsiveness to clients. Service standards are set and tools to measure client satisfaction are used regularly.

20.65 Getting on with measuring the results they achieve is perhaps the most immediate challenge departments face. The next step beyond planning to manage for results is implementing a measurement strategy - actually measuring a wide range of outputs and outcomes, learning which measures are most useful by using the information, and adjusting measurement in the future.

Learning From Others' Experience

20.66 Initiatives in managing for results are common in other jurisdictions. Our 1997 study noted a number of results-oriented initiatives in several jurisdictions, both in Canada and elsewhere. We observed in 1997 that some of these jurisdictions had enacted accountability legislation; since then, more have done so.

20.67 In Alberta and British Columbia, legislation requires both government-wide and ministry preparation of multi-year business/performance plans to be presented to the legislature. It also requires annual reporting of progress made toward achieving the expected results set out in those plans. In Quebec, new legislation focussed on improved public service is highly supportive of managing for results. In order to enhance accountability, the use of multi-year strategic plans and performance agreements is now mandatory for all departments.

20.68 Chapter 19 of this Report, Reporting Performance to Parliament: Progress Too Slow, recommends that in light of the modest progress made in reporting performance to Parliament, it may be time for the government to consider introducing accountability legislation. As we observed in the audit discussed here, managing for results is still not an integral part of managing departments. There is a clear need to get beyond planning and secure further progress. Although legislation alone will not ensure good reporting or managing for results in departments, it could serve as a signal, as Chapter 19 suggests, that measuring and using performance information is not a passing fad.

20.69 Literature review identifies lessons learned. We conducted an extensive review of the literature to identify lessons learned and trends in other jurisdictions with experience in managing for results. Exhibit 20.7 presents a summary of these lessons, related to promoting favourable conditions for implementation; developing performance measurement systems; and using performance information. A full report can be found at www.oag-bvg.gc.ca.

20.70 We noted many consistencies between the findings of our literature review and findings from our examination work in departments on the lessons learned in implementing managing for results.

Need for Concerted Leadership by Treasury Board Secretariat

20.71 While the Treasury Board Secretariat has actively promoted improved reporting in a number of quite tangible ways (as discussed in Chapter 19), its efforts to encourage managing for results in the day-to-day operations of departments have been less evident.

20.72 A number of Treasury Board initiatives and publications relate to managing for results, including the Modernization of Comptrollership initiative, the President of the Treasury Board's annual report to Parliament on Managing for Results, and the Treasury Boards publication, Results for Canadians: A Management Framework for the Government of Canada (see Appendix C for a description of these and other initiatives). While these provide some recognition of the need to manage for results, discussion in them of specific practices is quite limited. These initiatives originated in different parts of the Secretariat that give different levels of attention to managing for results. It is often not their prime focus.

20.73 Limited evidence that the Secretariat uses information on departmental results. In 1997, we suggested that a significant incentive for departments to manage for results would be clear evidence that the Treasury Board was indeed using performance information in its deliberations. Our interviews at the Treasury Board Secretariat suggested that while it occasionally uses departmental performance information, it does not use it systematically or routinely in its decision making. Yet this is stated as a "key responsibility" of the Secretariat as a management board: to "compile information sufficient to assess program performance and program integrity across government."

20.74 Treasury Board Secretariat has not identified "results" expected in government's managing for results. The Secretariat has not set out the results planned for the government in managing for results - specifically, what is expected in departments and by when. In the absence of clear expectations, it may be difficult for the Treasury Board Secretariat to know the kind of support it should be providing and whether departments are making enough progress. The Secretariat has not had a clear plan to focus its activities in this area. Its publication Results for Canadians calls for a strong focus on results in managing the Government of Canada. Officials told us that the Secretariat will use this publication as a basis for action plans to meet the government's commitment to managing for results.

20.75 The Treasury Board Secretariat's efforts are present, but diffused. To be effective, it needs to concentrate more on defining concrete expectations and practical strategies for managing for results. Otherwise, it is not clear how it will fulfil the role it recently set for itself as a management board to champion managing for results.

20.76 The Treasury Board Secretariat should develop concrete ways to champion managing for results in government and to support departments in this effort. It should actively assemble and communicate practices that have proved successful in dealing with the challenges that departments face.

Evaluation Is a Vital Element

20.77 Evaluation has an important role to play in managing for results. It can provide important information on program performance that is not gathered by ongoing monitoring systems, and can help managers understand why programs are working or are not. Evaluation can assist in the development of the results-based management and accountability frameworks the Treasury Board now requires, which help managers identify the information they need to collect in order to demonstrate results. We would therefore expect evaluation to be an integral part of program management and part of a department's strategy for managing for results.

Following up on our 1996 audit

20.78 In 1996 we reported on the state of evaluation in the federal government (1996 Report, Chapter 3). We made several recommendations to the Treasury Board Secretariat and to deputy heads. In the current audit, we followed up on those recommendations and looked at how evaluation is being used in managing for results.

20.79 Revisiting our 1996 recommendations. The wording of our 1996 recommendations reflected the circumstances at that time. Since then, there have been a number of changes in the government's expenditure management system and in the Secretariat's reporting to Parliament. We made three recommendations to the Treasury Board Secretariat, based on the following observations.

20.80 We had found that the bulk of evaluation activity was focussed on operational and lower-level issues and concerns, and that significant aspects of government activity were not being evaluated and reported. In our view, evaluation needs to play a strategic role in government as a whole in addition to its role in departments. The government needs the capacity to effectively evaluate interdepartmental and other horizontal issues. We recommended that some direction should be provided to the evaluation functions in departments on overall evaluation priorities so that the government is able to evaluate the important areas, including horizontal issues, of government programming and interest (paragraph 3.74).

20.81 Parliament has frequently expressed an interest in the evaluation capability of government and the Public Accounts Committee has asked to be kept informed about the state of evaluation. We recommended that the Secretariat should report to Parliament on the state of the government's evaluation capacity (paragraph 3.86).

20.82 It was hard to identify what was then called evaluation. The "review" function was diffusing effectiveness measurement, which was central to evaluation. We thought that if review were to be encouraged, it should have appropriate standards that would differ from those used for evaluation. We recommended that quality standards be set out for evaluation and any other related effectiveness measurement activity of government (paragraph 3.106).

20.83 In our view, our 1996 recommendations are still relevant.

Our assessment of progress

20.84 We believe that progress on these recommendations over the past four years has been unsatisfactory. (Appendix D lists the recommendations and summarizes our assessment of progress). Recently, the government has been putting new efforts into evaluation.

20.85 But the Treasury Board Secretariat has not yet set out government-wide priorities for evaluation, as we recommended in 1996.

20.86 Although the first report of the Treasury Board President to Parliament (1995) did include evaluation and review, the focus of that annual report has changed. Parliament has not been informed since 1996 about the relatively weak state of evaluation in government, despite its clear interest in the subject.

20.87 The Secretariat has draft policies proposed for both internal audit and evaluation. The current version makes no reference to review as a separate activity, which may simplify measurement. The draft evaluation policy contains quality standards that, if promulgated, would satisfy our recommendation for evaluation standards.

Evaluation function has regressed

20.88 A need to rebuild the evaluation function. A recent study by the Secretariat shows that over the past four or five years, the evaluation function in the federal government has regressed. During most of this period, the Secretariat's attention to evaluation was limited. Key study findings include the following:

• Evaluations are still seen by some as a check on management rather than as a tool for management, but this attitude could be changing.
• There is room to develop managers' capacity to carry out evaluations of their programs for the purpose of organizational learning and improved implementation.
• Evaluation frameworks (or results-based management and accountability frameworks), although mandated by policy, have not always been created or, if created, not always implemented.
• Program Review led to a serious undermining of evaluation capacity across the government.

There is a clear need for concerted effort to rebuild the evaluation function.

20.89 There are signs of progress in some areas. In line with our 1996 recommendations, the Treasury Board Secretariat is now identifying ways it can support the evaluation function. In its publication Results for Canadians, it made the following commitment:

Working with departments and agencies, the [Secretariat] will develop and implement plans to better position and strengthen the program evaluation and audit functions within the broader effort to implement modern comptrollership and results-based management across government.

20.90 Further, Treasury Board Guidelines for the Preparation of Departmental Performance Reports discuss the important role evaluation can play in reporting. In addition, the recently revised Policy on Transfer Payments requires that departments and third parties prepare a results-based management and accountability framework and conduct evaluations of their programs and projects.

20.91 Evaluation in departmental business planning. Two recommendations in our 1996 audit were addressed to deputy ministers: that business plans include evaluation findings and identify priorities for evaluation, and that plans for evaluation be developed as part of the business planning process. While departments are no longer required to submit business plans to the Treasury Board Secretariat, they are still expected to undertake business planning. The five departments we examined have made satisfactory progress in this area.

20.92 We looked at evaluation planning practices in our five case departments. Their practices varied; each produces different types of planning documents containing different types of information. Key business planning documents are developed at the business line level. These documents often set out strategies for measuring performance, including planned evaluations.

20.93 A review of departmental evaluation plans in the five departments revealed that all of them group evaluation activities by business line or by departmental objective. Environment Canada's evaluation plan articulates clearly how evaluations support managers' efforts to achieve planned results and how they help senior managers meet the Department's strategic objectives. All departments said they are continuing to work on integrating departmental priorities in their evaluation plans, and increasingly they see evaluation as an important component of the department's strategy for meeting its objectives.

20.94 Some recent management interest. In two of the departments, we heard that evaluation has remained stable over the past few years. In the three other departments, evaluation is only now showing signs of revival after a few difficult years: more staff are being hired, and the departments are developing strategies to strengthen the professional capacities of their audit and evaluation personnel. Increasingly, program managers are contacting the evaluation unit for help in developing evaluation frameworks (results-based management and accountability frameworks), as required in Treasury Board approval documents. Heads of evaluation in the five departments noted that program managers are also showing more interest in tools that will help them demonstrate results and support them in decision making.

20.95 One of the Secretariat's conclusions in its study of the evaluation function is that there is a need to renew the mandate of evaluation in the new management regime. This is clearly a task for the Treasury Board Secretariat to undertake.

20.96 The Treasury Board Secretariat should strengthen the evaluation function to ensure that it plays a key role in the government's move to managing for results. The Secretariat should also ensure that the evaluation function has the capacity to address strategic issues, including horizontal issues, that are of concern to government as a whole and to Parliament.

Conclusion

20.97 We set out to identify the extent to which managing for results was being practised in the five selected departments and 10 programs we examined. We found a few programs that use performance information routinely. We found that other elements of managing for results have shown some progress. They include initiatives to foster a supportive climate, extensive planning activities, and some measurement of performance. But a lot remains to be done before a strategy for managing for results is fully established in government.

20.98 Most departments need to move beyond planning to manage for results. They need to get on with measuring their progress toward at least some of their planned results, and using that information to improve performance. The persistent state of planning in most departments has meant that progress has been limited since we reported on this issue in 1997.

20.99 We appreciate the many challenges that departments encounter in their efforts to manage for results, and we recognize that this process takes time. In our audits and in our review of other jurisdictions that are managing for results, we identified many of these challenges - in particular, the challenge of fostering a climate in the organization that makes it the norm to systematically track a program's performance in order to learn what is working and what is not.

20.100 With sustained effort, departments can make strides toward managing for results. We found in some cases that a focus on results has filtered down to the day-to-day operations, despite impediments such as organizational change. Managing for results in these cases has become part of doing business.

20.101 As we reported in 1997, we found enough experience to draw from in Canada and elsewhere to establish a framework for managing for results. There is a particular need for departments in the federal government to:

• demonstrate senior management commitment and leadership;
• put in place levers to encourage a focus on results;
• agree on a planning, reporting and accountability framework, key results commitments and planned results;
• develop internal tools and capacity for measuring results;
• use information on results for decision making at all management levels; and
• report performance against stated expectations in a balanced manner.

20.102 The concept of managing for results is now accepted across government. However, there is little practical guidance on managing for results, and the sharing of experience is limited. Departments need more encouragement to move beyond merely planning to manage for results. Champions need to be identified. We found that the leadership and support provided by the Treasury Board Secretariat has been too dispersed to be of enough help to departments in managing for results.

20.103 Evaluation is not playing a large enough role in managing for results. Often, the only way to understand how programs are contributing to outcomes is through a systematic evaluation study. The federal evaluation function has been allowed to atrophy and to focus too often on operational issues rather than measuring how well departments are meeting their key results commitments.

Managing Horizontal Issues for Results: Finding Solutions to the Challenges

Introduction

20.104 In the first part of this chapter, we reported on the progress five departments have made toward managing for results, and the progress in 10 of their programs. We found growing attention to results and to particular elements of managing for results, including planning and performance measurement. We concluded that a lot remains to be done before the government is fully managing for results.

20.105 This part of the chapter looks at what the government is doing to support managing for results when several departments are involved in "horizontal" issues that cross departmental boundaries.

Growing Attention to the Management of Horizontal Issues

20.106 The commitments the federal government makes to Canadians are often targeted at broad issue areas. In recent Speeches from the Throne, for example, the government undertook to improve the well-being of rural Canadians, better integrate persons with disabilities into society, co-ordinate with other jurisdictions to address children's issues, and address broad environmental issues like climate change and ecosystems management. These are horizontal issues - they cut across departmental lines, and they reflect the government's broad vision for Canada.

20.107 When a policy issue spans departmental mandates, no one department has all the levers, resources and expertise to manage it adequately. Departments must work together toward an overall objective and adopt a common vision for success. They must effectively manage the resources dedicated to the horizontal issue and consider collectively how to fill gaps in service and eliminate duplication. They must be able to demonstrate accomplishments and learn from their present performance in order to make progress toward the expected results. And the issue may require them to work together with other levels of government, and with partners outside government. When they do all of this, they are managing the horizontal issue for results.

20.108 Alberta reports some horizontal results. Starting with the 1998 business planning cycle, the Government of Alberta Business Plan includes a section dedicated to key cross-government initiatives planned or under way in the three-year period covered by the Plan. In this way, Alberta recognizes that many of the pressures and challenges it faces need to be addressed corporately - they extend beyond the boundaries of single ministries. For the past two years, the Alberta Cabinet has singled out four cross-government or horizontal initiatives: two involving societal issues, and two in public administration. It has identified ministries to champion the initiatives, and given them the responsibility to manage and report on it.

20.109 Deputy Ministers' Task Force. The 1996 federal Deputy Ministers' Task Force looked at the need for interdepartmental collaboration in the management of horizontal issues, and the challenges this represents. Its report, Managing Horizontal Policy Issues, made practical recommendations for improving policy co-ordination within the federal government. It also addressed some of the procedural, structural and cultural implications of managing horizontal issues.

20.110 In particular, the Task Force pointed to the need for:

• an accountability framework to clarify the respective mandates and contributions of lead and partner departments;
• consultation with stakeholders on cross-cutting issues;
• realistic expectations for timing and outcomes, given the complexities of managing cross-cutting issues;
• adequate resources to allow the lead department to plan, co-ordinate and implement a horizontal initiative; and
• appropriate incentives and rewards for collaboration and teamwork.

20.111 In addition, the Task Force report discussed the importance of evaluation as a means of learning from experience in managing horizontal issues, and the importance of reporting and other forms of communication to share results and success factors across government.

Horizontal Issues Addressed in Our Previous Audits

20.112 The Office of the Auditor General and the Commissioner of the Environment and Sustainable Development have carried out a significant amount of work on the management of horizontal issues. Recent chapters that looked at various partnership arrangements recommended stronger frameworks for accountability and management, and more effort to monitor, evaluate and report results. (See Auditor General's 1999 Report, chapters 5, 6 and 23; and 2000 Report of the Commissioner of the Environment and Sustainable Development, chapters 2, 5, 6, 7 and 8.)

20.113 Particularly germane to this discussion is the Commissioner's 2000 Report, Chapter 6, Working Together in the Federal Government. In six case studies, the audit identified key factors affecting the success of sustainable development initiatives that span departmental boundaries. The Commissioner's key findings and success factors are shown in Exhibit 20.8. They are consistent with the literature on managing horizontal issues, with the work of the Deputy Ministers' Task Force, and with the findings of this study.

Focus of the study

20.114 The objective of our study was to identify the advantages of and impediments to using performance information and reporting in the management of horizontal issues. We selected case studies to explore how and to what extent horizontal issues are managed to achieve common results, based on the potential they offered for lessons learned. Most are long-standing horizontal issues, are clear government priorities, and have clear objectives.

20.115 The following are the cases we selected for study:

• Family Violence Initiative;
• Government of Canada Disability Agenda; and
• Canadian Rural Partnership.

Appendix E provides background information on these cases.

20.116 We limited our study to collaboration among federal departments on clear government initiatives. We did not address the particular challenges of managing issues in collaboration with other jurisdictions. Other types of collaborative or delegated arrangements have already been the focus of audits in recent reports (in particular, the 1999 Report, chapters 5, 6 and 23). While several of the observations and findings in those chapters can be applied by federal departments in managing horizontal issues for results, here we will discuss in particular the challenge and complexity of co-ordination.

Observations and Recommendations

20.117 Based on our review of the relevant literature and our case study work, we developed a framework for managing horizontal issues (Appendix F). The framework focusses on co-ordination structures suited to achieving the government's objectives for an issue, and on elements critical to measuring and reporting results. The framework has five elements:

• identifying an effective co-ordination structure;
• agreeing on common objectives, results and strategies;
• measuring results to track performance;
• using information to improve performance; and
• effectively reporting performance.

20.118 As the reader will observe, this framework is based on the managing for results framework presented in Exhibit 20.4 in the first part of this chapter.

Effective Co-ordinating Structures Are Needed

Many approaches to co-ordination

20.119 Horizontal issues can be co-ordinated in a number of different ways; how, and how closely, will depend on what the issue requires. For example, co-ordination can be informal, limited to simple communication and networking between managers in different departments. Efforts at co-ordination can also be more formal, if departments recognize a need to link their common objectives and work co-operatively.

20.120 Some circumstances do not warrant strong co-ordination. Many policies and programs require only communication among departments to ensure that their efforts are complementary. Close and structured co-ordination in those cases can overburden the participating departments and be counterproductive to the initiative.

20.121 Our three cases are examples of structured co-ordination, each to a different degree. In structured co-ordination, departments work more in concert toward shared objectives, have shared roles and responsibilities, agree on a decision-making process, and co-operate under a defined government strategy or in view of a long-term public policy objective.

20.122 Closer and more structured co-ordination is an advantage when, for example:

• the issue is a high priority of the government;
• the issue clearly spans the mandates of several departments;
• there is a potential for real gains by integrating policy development;
• strong champions or senior officials endorse the establishment of co-ordinating structures;
• the financial commitment is significant; or
• there is strong commitment and clear agreement among participants.

20.123 Identifying a lead department. If a department is designated to lead the management of a horizontal issue, it has the critical role of ensuring that the issue is managed in a way that will meet the partners' objectives and obligations; it ensures that partners are kept informed, that performance is monitored, and that partners live up to their commitments. It is important that the lead department be recognized as such by all participants and assigned the powers it needs to discharge its responsibilities.

20.124 Each of the horizontal initiatives we selected has a lead department. The rural issue was considered significant enough to warrant the creation in 1996 of the Rural Secretariat in Agriculture and Agri-Food Canada. As the lead, the Department has a mandate to ensure that departments and agencies work closely and effectively with politicians, government officials and rural citizens to address rural problems.

20.125 Dedicated units in Health Canada and Human Resources Development Canada have also been identified to co-ordinate, respectively, the Family Violence Initiative and the Federal Disability Agenda. In addition, the Federal Disability Agenda is supported by structures including a House of Commons Subcommittee on the Status of Persons with Disabilities, and a variety of management and working groups created to direct or implement the initiative.

20.126 The Family Violence Initiative, whose annual allocation of $7 million is shared by seven departments, is co-ordinated by a special unit of Health Canada and directed by an interdepartmental working group. It receives input from an evaluation working group and has accountability and reporting frameworks that define relationships among the partners. While it is well structured, we were informed that it receives little senior-level attention in the lead department.

20.127 Exhibit 20.9 shows the key elements of co-ordination, structure and accountability in each case. Appendix E provides background information on each of the three case studies, along with key features of their co-ordination structure.

20.128 Co-ordination takes time and effort. One of the issues raised consistently by all lead departments in our case studies was the time and effort it takes to co-ordinate effectively. For example, they must build consensus, persuade partner departments to adopt certain positions, ensure that decisions are carried out, communicate with stakeholder groups, and ensure that timelines are respected. Yet lead departments often have very little authority to back up their responsibilities. While formal agreements endorsed by senior officials in partner departments are valuable, they are not enough.

20.129 Lead departments particularly singled out the time and effort needed for reporting. The lead department must, for example, ensure that data from many different sources are collected and collated; obtain partners' agreement on reporting requirements; and engage partners in ongoing discussions about the focus, content, format, and readership of the report.

20.130 Officials involved in the Canadian Rural Partnership and the Family Violence Initiative acknowledge that their first reporting experience was a challenge. In both cases, it took much longer than expected to negotiate the content of what was simply a report of activities, not results. Experience with these initial reports suggests that later, reporting on results will be an even greater challenge for management.

20.131 Our review of the literature and our discussions with departments participating in horizontal initiatives indicate that having dedicated resources is a big help in managing and co-ordinating. The Canadian Rural Partnership and the Family Violence Initiative benefited from Treasury Board funding earmarked for co-ordination. By comparison, the recent federal Budget did not include funds to co-ordinate the Federal Disability Agenda, requiring the Department to reallocate funds internally in 2000-2001 to allow the Office for Disability Issues to carry out this responsibility. Without resources, it is difficult for a lead department to meet the full slate of commitments identified at the start of an initiative. However, dedicating resources to co-ordination does not guarantee success; indeed, it needs to be complemented by senior management commitment and clear engagement of all partner departments.

Senior Support and Effective Tools for Issue Management

Senior leadership is key

20.132 Our case studies clearly demonstrate that support from senior levels of departments and clear political support are decisive factors in the successful management of horizontal issues. Senior officials who have a role in championing an issue work to ensure that all necessary support is available, including financial support. Horizontal issues that benefit from senior leadership are more focussed, better resourced, and more effective at meeting the government's commitments. The absence of such support from leaders leaves participants drained of energy and losing interest in the issue.

20.133 Leadership is needed in both the lead department and other participating departments. It is particularly important in cases that have no signed agreement among partners or a formal framework that clearly outlines the responsibilities of all parties. Even where such formal documents do exist, champions can do much to ensure that participants contribute to the initiative as expected and are held accountable for results.

20.134 There are many ways to show senior leadership. For example, senior departmental officials who champion the Canadian Rural Partnership ensured that resources were available for co-ordination, engaged senior officials of partner departments in contributing information for the first annual report, and maintained open lines of communication with the co-ordination unit so that they could be kept informed of progress in the initiative.

20.135 Leadership in the Federal Disability Agenda can be seen in the establishment of the House of Commons Subcommittee on the Status of Persons with Disabilities. A June 1999 report of the Subcommittee, Reflecting Interdependence: Disability, Parliament, Government and the Community, recommended that the government take real action to support persons with disabilities. It suggested, for example, that the government integrate a disability "lens" into all policy and program decisions, and that it report comprehensively on outcomes, results commitments and performance of federal activities in the area of disability. The Subcommittee's continued attention can keep the issue on the government's agenda for action.

20.136 Challenges in the absence of leadership. For some horizontal issues, the general absence of senior management leadership is difficult to overcome. Now in its third phase, the Family Violence Initiative has seen its funding from Treasury Board cut back significantly, with a corresponding decline of senior management attention to the initiative in many of the partner departments. The lead department has redirected funds for co-ordination obtained from the Treasury Board to other areas in the department, and has not provided the support needed for the initiative to keep up with its reporting schedule. The long history of ground-level co-operation and dedication of individual officers in the lead and the partner departments cannot make up for the absence of senior management leadership.

Formal frameworks are useful documents

20.137 Frameworks needed to set out common objectives. The more ambitious the objectives of the horizontal initiative, the greater is the need for clarity of partners' roles and responsibilities in managing, co-ordinating and reporting. Whether in a formal accountability framework document or a letter of agreement, all participants need to set out their understanding of the common objectives and state, in some detail, how and what they will contribute toward those objectives.

20.138 Such documents are particularly important when different partners give an issue different priority. In some cases, this occurs because some partner departments get Treasury Board funding for their part in horizontal programs and activities while others fund their part from their departmental budgets. The non-funded departments, for example, may resist having their internal activities subject to horizontal management. Moreover, non-funded departments do not have the formal obligations for reporting that are set out in Treasury Board funding agreements, and in a horizontal arrangement their reporting obligations are not clear.

20.139 Departments in our case studies noted the importance of having senior officials sign agreements so they are aware of the commitments their departments have made. While agreements among partners do not guarantee success, they are useful in clarifying accountabilities and expectations.

20.140 Among our case studies, only the Family Violence Initiative had developed formal accountability and reporting frameworks. During our study, partners in the Federal Disability Agenda and the Canadian Rural Partnership were developing detailed reporting frameworks but neither had a formal accountability framework. When an initiative requires more of departments than simple reporting of activities, or requires that partners alter programs to avoid duplication or to fill gaps, coming to an agreement on the roles and responsibilities of each will be important.

Finding effective levers and incentives

20.141 Levers and incentives are needed to support co-operation among partners and to overcome pressures on each to focus solely on its own vertical reporting and accounting requirements. A wider range of levers is available both to lead and to partner departments when there are formal structures and agreements. Departments can use communication, transparency, performance appraisals, rewarding teamwork and support for horizontal issues through, for example, promotion or recognition. By working in tandem with central agencies, departments can make more effective use of the visibility given to a horizontal issue, the resources allocated and various human resource management tools.

20.142 However, departmental officials we interviewed for our case studies noted the apparent lack of levers and incentives they could use to promote good management of horizontal issues. Central agencies have a role to play in developing and promoting such tools, if managing horizontal issues is a government priority.

Challenges in Reporting

20.143 Limited evidence of reporting or use of performance information. Canadians and parliamentarians want information on the overall accomplishments of these horizontal initiatives, collective information that reports on more than the individual activities of the departments involved in an initiative.

20.144 Our cases have not yet reported collective results. At the time of our study, two had produced reports, but both were reports on the activities of individual partner departments. Departments involved in horizontal initiatives note that developing and using common performance measures is a challenging task, and reporting on collective performance cannot be expected in the first years of an initiative. Reporting on results requires adequate planning, and departments involved in our case initiatives are working now to produce reports on results in the future.

20.145 Conflicting priorities. We found that to a large extent, the challenge in reporting on collective outcomes lies in the tensions between vertical accountability structures in departments and horizontal commitments and co-ordination. Current reporting tools and processes - like reports on plans and priorities, departmental performance reports, and internal reporting documents such as business or strategic plans - are designed to collect information on the performance of departmental programs and services. They are used to support the department's ability to demonstrate its accomplishments. If one partner department in an initiative produces information that is incompatible with what other partners produce, or if it would require significant resources to collect performance information as a special activity, then gathering the data and producing a report on collective results will likely be problematic.

20.146 Another difficulty is getting partners to agree on appropriate and measurable indicators of performance. In some cases, it requires that partner departments overcome their resistance to detailed management frameworks and outline specific indicators, measures, and responsibilities for data collection and the collation of performance data. Of our cases, only the Family Violence Initiative has developed a reporting framework that outlines roles and responsibilities in the collection of data for measures that all partners have agreed to. And yet it has not measured performance as planned and, at the time this study was conducted, it had submitted only one annual report to the Treasury Board Secretariat, for 1997-98.

20.147 A need to plan for reporting. In collective reporting, it is important to ensure that plans and expectations for reporting, and specifically the roles and responsibilities this entails, are commensurate with the powers and resources of the partners and the level of funding for the initiative. Reporting plans need to consider that some departments must collect information from parties outside the federal government: many horizontal initiatives involve societal issues that fall under other jurisdictions or require the participation of parties outside the public sector.

20.148 Costs not known. In Chapter 19 of this Report, we noted that information on the cost of achieving results was generally not available in departments. In horizontal issues, the challenge of obtaining cost information is exacerbated by the number of departments involved and the different funding arrangements in place. A particular difficulty is knowing what part of the budget of non-funded programs contributes to the results expected in the horizontal initiative. For example, a prenatal nutrition program meets the needs of Canadians generally and, at the same time, those of segments of the population (such as rural Canadians and children). The costs of programs delivered to rural Canadians, particularly the costs that do not specifically target them, are hard to determine. Knowing (and managing) the costs of achieving horizontal results is difficult, but not impossible. Only in the Family Violence Initiative was there any information on the program dollars involved in the horizontal issue.

Lessons Must Be Learned and Shared

20.149 Insufficient use of lessons. Our cases suggest that more effort could be made to use lessons from previous generations of an initiative. While partners may refer to results of past evaluations of the initiative, this information is not used routinely. In one initiative we examined, the turnover rate and the state of the collective memory were such that a new participant was unaware for some time that the initiative had existed for several years.

20.150 Lead departments in the cases we examined are taking some positive steps to learn from each other's experience. For example, the Office for Disability Issues recently invited the lead department for the Canadian Rural Partnership to share its lessons learned in reporting on a horizontal issue. The need to learn from past experience and the experience of others is also clear in other horizontal initiatives, including federal government commitments to green its operations, elements of which were recently audited by the Commissioner of the Environment and Sustainable Development. That case is described in Exhibit 20.10.

Some challenges and solutions identified

20.151 Exhibit 20.11 summarizes some of the main challenges and solutions we identified in our review of the literature and in our case studies, organized according to the broad headings in our framework for managing horizontal issues (see Appendix F). While this is by no means a listing of all the challenges encountered by lead and partner departments, it does suggest what managing horizontal issues for results can entail.

A Stronger Role for Treasury Board Secretariat

The role of the Treasury Board Secretariat needs to be better defined

20.152 Horizontal issues are an important area that requires concerted, strategic attention by the Treasury Board Secretariat. The Secretariat has been proactive in identifying programs that could be managed horizontally. It has made presentations and developed draft guidelines on reporting and accountability frameworks for managing horizontal issues, and on its Web site it has shared information on collective and societal results and indicators. The Secretariat has also supported individual initiatives. Lead departments in horizontal initiatives have indicated that they value its contribution.

20.153 The Treasury Board Secretariat requires that submissions for funding of horizontal initiatives contain a commitment to performance reporting and evaluation. However, not all horizontal initiatives receive funding through the submission process. And, as already noted, not all departments receive funding for their part of the horizontal initiative. For those that do not, there are no central requirements to report collectively or to participate in collective evaluations. It is important for the Secretariat to ensure that horizontal issues are managed and co-ordinated adequately and resourced in a way that facilitates evaluation and full reporting when funded and non-funded departments are partners in a horizontal initiative.

20.154 We have observed that the role played by the Treasury Board Secretariat in the management of horizontal (interdepartmental) issues is piecemeal and differs from one initiative to the next, apparently independent of the characteristics of the initiative. The Treasury Board Secretariat does not have a strategic approach for dealing with the fact that a growing number of issues are addressed horizontally.

20.155 In view of the many challenges to horizontal initiatives that we have discussed in this chapter, we support the recommendations in the Deputy Ministers' Task Force report, Managing Horizontal Policy Issues, and in several reports of the Commissioner for the Environment and Sustainable Development. These include recommendations that central agencies:

• clarify expected outcomes and accountabilities of the partners involved, and
• provide advice on the co-ordination and management of horizontal issues.

20.156 We also agree with the Commissioner's recommendation that the Treasury Board Secretariat prepare a "best practices" guide for interdepartmental co-ordination, with input from those who have worked on horizontal issues.

20.157 Stronger central leadership is needed if the government wants to continue to promote horizontal issues as a priority and to focus increasingly on horizontal policy development, implementation and accountability.

20.158 The Treasury Board Secretariat should play a stronger leadership role to ensure that where appropriate, horizontal issues are managed for results. This includes ensuring that resources are available for the co-ordination and management of horizontal issues that are government priorities, and ensuring that external reporting takes place.

20.159 The Treasury Board Secretariat should play a stronger role in communicating good practices and lessons learned in managing horizontal issues for results.

Conclusion

20.160 Our case studies indicate that there are many challenges in managing horizontal issues for results. Partners in horizontal initiatives can co-ordinate sufficiently to identify collective objectives, to share information, to develop frameworks for accountability and to report on activities. However, it is more difficult for them to manage tasks that require them to compromise or realign their own programs. Managing for results is difficult without sufficient resources; funds and human resources are needed to co-ordinate the efforts of partners and to implement reporting frameworks that can show Canadians and parliamentarians what has been accomplished by a horizontal initiative.

20.161 Government is still at the start of its move toward reporting and using performance information in managing horizontal issues for results. The Treasury Board Secretariat in particular has a role to play in supporting the effective use of levers and incentives by departments involved in horizontal initiatives, and in creating the conditions that favour more effective management of horizontal issues.

Government's response: While agreeing with the emphasis the government has placed on managing for results, the Auditor General concludes that the fundamental shift in perspective from process-oriented or results-oriented management has not yet occurred generally across the public service. The chapter highlights the need to strengthen measurement of results, in particular the evaluation function. In addition, the audit notes the particular challenges in managing horizontal issues and recommends a stronger role for the Treasury Board Secretariat in supporting departments in this regard.

With the tabling of Results for Canadians: A Management Framework for the Government of Canada in Parliament in March 2000, the government now has a framework to move forward in this area. As the government implements this framework, departments and agencies will need to improve their measurement of results and costs and use this information to deliver better results for Canadians. To ensure that these expected improvements take place, the Treasury Board Secretariat continues to monitor implementation and upgrade its plans to exercise appropriate leadership and support to departments and agencies in this endeavour. It also has made internal managerial changes to promote the effective implementation of this management framework.

The government recognizes that managing horizontal issues, including those involving other jurisdictions, is a challenge for departments. The Treasury Board Secretariat will continue to actively track major collaborative initiatives and ensure that lessons-learned are shared. It is also examining how to exercise the best possible leadership and support in responding to these new challenges.

About the Audit

Objectives

The objectives were to:

• examine the extent to which selected departments were managing for results, including their use of information on performance to manage programs;
• identify lessons learned in implementing managing for results;
• assess the leadership the Treasury Board Secretariat has provided to support managing for results throughout government; and
• in selected cases, to identify the advantages of and impediments to the use of performance information and reporting in the management of horizontal issues.

Scope

We examined five large departments - Agriculture and Agri-Food Canada, Correctional Service Canada, Environment Canada, Industry Canada, and Natural Resources Canada. In each department, we asked senior officials and/or corporate staff to help identify areas where the use of performance information could be expected. We selected and then examined a total of 10 programs/business lines to determine to what extent performance information is actually used and to what extent other elements of managing for results are in place. We also asked the departments to identify the lessons they learned in implementing their regimes for managing for results.

In addition, we followed up on the findings and recommendations of our 1996 Report, Chapter 3, Federal Program Evaluation. We examined the same departments selected for examination in our current audit. We looked at how these departments are planning for evaluations and using the information they provide on performance to manage for results.

For our study discussed in the second part of this chapter, we used three cases to explore good practices in managing horizontal issues for results and reporting results. Our selection of the case studies was based on the potential they offered for lessons learned, given that they are long-standing horizontal initiatives, are clear government priorities, and have clear objectives.

Criteria

To assess use made in selected departments of information on performance to manage programs and reporting on them:

• In selected departments, we expected to see at the corporate (or department-wide) level appropriate use of information on performance to improve program operations and design, to update planned results and budgets, and to report program performance at the corporate level.
• In selected programs/business lines, we expected to see appropriate use of information on performance to improve program operations and design, and to update planned results. At a minimum, we expected to see elements of managing for results in the programs and business lines we examined. These elements include fostering an organizational climate that encourages managing for results; measuring performance; and planning a strategic framework that demonstrates agreement on areas of key results, performance measures, and performance expectations.

In assessing the leadership provided by the Treasury Board Secretariat to support managing for results throughout government, we expected to find that the Secretariat:

• provides appropriate and consistent direction, guidance and feedback to departments on their efforts at managing for results;
• gathers and disseminates best practices and lessons learned;
• uses departmental results and performance information; and
• encourages departments and agencies to institute managing for results in a reasonable time frame.

In examining the management of horizontal issues for results, we reviewed the applicability of the framework we had developed for managing for results and looked for lessons identified in our cases. As this component of our work was a study, we did not set out specific criteria against which the cases were assessed.

Approach

We interviewed senior managers in the five selected departments and in the Treasury Board Secretariat (TBS), visited several field operations, and gathered relevant documents. We reviewed the literature in other jurisdictions on managing for results.

In our examination of the leadership role played by the Treasury Board Secretariat, we interviewed TBS officials, examined relevant documents and interviewed departmental officials for their perceptions of the Secretariat's leadership.

In our study of managing for results in selected horizontal initiatives, we interviewed departmental participants, including officials in lead departments. We had discussions with officials of the Treasury Board Secretariat, collected relevant documentation, and reviewed literature on horizontal issue management.

Audit Team

Assistant Auditor General: Maria Barrados
Principal: John Mayne
Director: Patricia MacDonald

Christina Brooks
Allison Fader
Adèle Lamoureux
Erin Molloy
Tom Wileman

For information, please contact John Mayne.