Reporting Performance to Parliament — Progress Too Slow

Main Points

19.1 Federal departments and agencies have made some progress in reporting on their performance to Parliament, but we are disappointed at its present pace. Continued at this rate, it would take too long for good reporting to become routine. We recognize that reporting on performance is not easy, and it takes time. Still, we believe that the effort deserves more attention: members of Parliament need information on what Canadians are getting for their taxes in order to hold the government to account. Good performance reporting has become an essential part of modern democratic systems.

19.2 To that end, it may be time for legislation. Several jurisdictions in Canada now have legislation requiring government to report on its performance to the legislature. Legislation highlights the importance of good reporting and puts it on a more permanent and stable footing.

19.3 Our examination of the Estimates documents - reports on plans and priorities and departmental performance reports - of 47 departments and agencies over the past four to five years found the following:

• While there has been progress, few departments set out concrete statements of what they expect to achieve and then report back specifically against those expectations.
• In reporting what they have accomplished, departments focus too much on listing activities and outputs and too little on linking them to and reporting on the intended outcomes.
• Reporting makes too little use of evaluation findings on the results that activities are accomplishing and on how programs are contributing to outcomes.
• There is little linking of financial and non-financial performance information; most accomplishments are not costed.
• Performance reports lack balance. Most departments report only the "good news" and make little mention of performance that did not meet expectations. As members of Parliament have said repeatedly, reports that present only good news are not credible.

19.4 We noted that the Treasury Board Secretariat has provided leadership for reporting. And we did find pockets of good reporting and a number of strengths in the present regime of reporting. Nevertheless, after five years of experimenting, departmental reporting overall has only improved modestly.

19.5 Three factors stand out as contributing to the current weak state of reporting:

• basic principles of good reporting are not understood or applied;
• performance reporting has political dimensions; and
• there are few incentives for good reporting or sanctions for poor reporting.

19.6 In addition to considering legislation, in our view the government needs to make the following changes in the present reporting regime:

• reach agreement on principles for reporting performance information;
• give more attention to the right incentives for incorporating evidence-based approaches into reporting;
• ensure stronger oversight by the Treasury Board Secretariat; and
• seek to enhance the involvement of parliamentarians in reviewing performance plans and reports.

Background and other observations

19.7 Federal departments submit performance reports to Parliament in the fall every year, outlining what they have accomplished. The reports are subject to consideration by their respective parliamentary standing committees. Review of the reports at that time would enable parliamentary committees to make recommendations for the upcoming spending decisions to be announced by the government later, in the Budget. In the spring, after the Budget, departments submit reports on their plans and priorities, describing what they are setting out to achieve in the upcoming year and beyond.

19.8 In 1997 we first reported on this new regime for reporting to Parliament. We found that a good start had been made and that the basic framework used for reporting was sound.

19.9 Credible reporting on performance has a number of benefits: it can enhance accountability and trust in government, provide information for improved programs and services and better parliamentary scrutiny of them, and serve as an incentive for departments and agencies to manage for the results they have set out to achieve.

19.10 On two occasions in recent years, the Standing Committee on Procedure and House Affairs has issued reports with recommendations on ways to provide for greater involvement of parliamentarians in the review of the Estimates.

19.11 The President of the Treasury Board annually tables, along with the departmental performance reports, a report in Parliament on government performance measurement, reporting and results-based management. In recent years it has provided useful performance information on collective results - areas where several departments or several governments are working collectively toward a result.

19.12 The Treasury Board Secretariat has taken important steps toward more electronic reporting. It needs to encourage departments to make greater use of this vehicle, and gather and share good practices.

The response of the Treasury Board Secretariat, on behalf of the government and the departments audited, is included at the end of this chapter. The Secretariat has responded positively to our chapter and recommendations therein. The Secretariat accepts our recommendations and indicates the actions that, where appropriate, it is taking or intends to take to address them.

Introduction

A good start in 1997

19.13 In 1997, we examined the state of the federal government's performance reporting to Parliament (Auditor General's 1997 Report, Chapter 5, Reporting Performance in the Expenditure Management System). We noted that the public and members of Parliament often lacked the kind of information that would demonstrate what Canadians were getting for the taxes they paid. Too often, departments described the activities they had undertaken and the services they had delivered but not the results they had achieved.

19.14 However, we did observe that several departments had made progress; we found aspects of good performance reporting in the information they had provided to Parliament. But we noted that in describing their expected performance, departments needed to be clearer and more concrete; in reporting their actual performance, they needed to focus more on the benefits they had gained for Canadians. The chapter made a number of recommendations for departments to improve their performance reports.

19.15 We also recommended that the Treasury Board Secretariat strengthen its leadership role. We noted that the role of parliamentary committees was vital to continued progress: if committees asked for information on performance and visibly used it in their deliberations, it would be a powerful incentive for departments to collect and report the information.

Many jurisdictions report on performance

19.16 A growing number of jurisdictions in Canada have performance reporting initiatives. As we reported in 1997, several jurisdictions in Canada (and elsewhere) are making tangible efforts to provide their legislatures and citizens with better information on how government programs have performed. Alberta has perhaps the most experience in Canada, reporting to the legislature on performance both government-wide and by ministry. Initiatives in Nova Scotia and New Brunswick that we noted in 1997 have continued, with some changes introduced by new governments. More recently, Quebec and British Columbia have passed legislation requiring the government to report to the legislature on performance. Saskatchewan also plans to enhance reporting.

19.17 We did not examine the success of initiatives in other jurisdictions or compare their performance information with that reported by Canada's federal government. We cite the experience and ongoing efforts of these jurisdictions to show that although it is not easy to provide better information on how government programs have performed, it can be done.

The Expenditure Management System

19.18 Past efforts by the federal government to improve reporting to Parliament included the creation of the Part III Main Estimates in 1981. The Part III documents were designed to provide information to members of Parliament on departmental spending plans and on the performance and results produced with the spending previously authorized. In 1983, the government agreed to include summaries of program evaluations in the Part IIIs.

19.19 In 1995, the government revised the Expenditure Management System it had used to manage its affairs and report on them to Parliament. As part of this initiative, it launched the Improved Reporting to Parliament Project.

19.20 From Parliament's perspective. The intent of the project was to provide Parliament with better information on the government's plans and performance. On a pilot basis, and with the agreement of Parliament, Part III of the Estimates was split into two documents:

• a Report on Plans and Priorities, tabled in the spring with Parts I and II of the Main Estimates, to establish performance targets and outline the general direction the minister wants the department to take during the upcoming year and the next two fiscal years; and
• a Performance Report, tabled in the fall, to indicate the extent to which results achieved to date reflect those that were planned.

19.21 Both of these new reports are Estimates documents, tabled in the House of Commons and subject to consideration by the appropriate standing committees. The timing of the spring plans and fall performance reports is intended to enhance the role of standing committees. Performance reports in the fall enable committees, in a timely way, to report their conclusions to the House and to make their views known to the government as it carries out its fall consultations on the upcoming budget. Standing committees may thereby influence spending plans and priorities for the next year or beyond. The plans reported in the spring can provide a focus for committee reporting to the House on future years' plans and priorities.

Clarifying results and performance concepts

19.22 Discussions of performance reporting can be confusing, as they involve a number of specific concepts and terms. Exhibit 19.1 presents some of them.

Focus of the audit

19.23 Our government-wide audit looked at the present state of performance reporting. This audit was in part a follow-up on the observations and recommendations we made in our 1997 Report, Chapter 5. Our objectives were to assess:

• the government's progress in reporting performance information to Parliament, including the leadership provided by the Treasury Board Secretariat and the use of performance information by standing committees; and
• the strengths and weaknesses of the federal regime of performance reporting.

19.24 We examined the Part III Estimates documents - the reports on plans and priorities and departmental performance reports of 47 departments and agencies (shown in Appendix A). These were the larger departments and agencies, including the 16 that had piloted performance reports in 1996, and a sample of smaller agencies. Over 80 departments and agencies provide these two Estimates documents to Parliament annually.

19.25 As this was a government-wide audit of the progress made to date in performance reporting, we did not audit the reliability or accuracy of performance information in reports on plans and priorities and departmental performance reports.

19.26 Further details on our audit objectives, criteria and approach can be found at the end of the chapter in About the Audit.

Observations and Recommendations

The Expenditure Management System Since 1997

19.27 The Improved Reporting to Parliament Project started in 1995 with five pilot departments, and expanded to include 16. Significant events have occurred since our 1997 audit:

• All departments and agencies (82 in 1999) submit reports on plans and priorities and performance reports.
• Based on the work of the Standing Committee on Procedure and House Affairs, a parliamentary subcommittee on the Business of Supply endorsed the Improved Reporting to Parliament Project in December 1998, and recommended a number of ways for standing committees to make good use of the Estimates reports. In its May 1999 response, the government agreed to engage parliamentarians further by initiating Phase 2 of Improved Reporting to Parliament.
• The Treasury Board Secretariat has established an Estimates Web site that includes departmental performance reports and reports on plans and priorities, with links to other performance information and information on cross-government initiatives.
• In March 2000, the government issued Results for Canadians, a results-based management framework for the Government of Canada.
• A second subcommittee established in May 2000 considered proposals for Phase 2; its report was tabled on 12 June 2000.

19.28 Business plans no longer submitted to Treasury Board. Perhaps the biggest change in the Expenditure Management System has been the evolution of departmental business plans. An integral element of the original Expenditure Management System was the requirement that departmental business plans be submitted to the Treasury Board for review, in place of multi-year operational plans. Business plans initially were to be concise, strategic three-year plans that set out goals, targets and performance measures. Several provinces, including Alberta, Ontario, and New Brunswick, have adopted business planning.

19.29 Business plans have evolved considerably since 1997, but as of 2000, their submission to Treasury Board is no longer required. The submission process began to be seen as cumbersome and not useful. For external communication purposes, a number of departments use their report on plans and priorities as their business plan.

19.30 Departments are encouraged to develop business plans for their own planning purposes. Treasury Board Secretariat analysts can get copies in the course of their ongoing dialogue with departments. On a selective and cyclical basis, any deputy minister may be asked to send a letter to the Secretary of the Treasury Board outlining current key issues and concerns in the department. The Treasury Board Secretariat intends to develop a yearly report to the Treasury Board on the key issues and risks faced by departments, together with options for dealing with them. The intent is that the President of the Treasury Board will present the results of this exercise to Cabinet at an early stage in the priority-setting cycle.

19.31 Planning, Reporting and Accountability Structure still approved by Treasury Board. An important element of the revised Expenditure Management System is the move by departments to a single, purpose-focussed accountability structure for reporting financial and non-financial information. This Planning, Reporting and Accountability Structure (PRAS) links the department's internal management and accountability regime to its objectives, business lines, resource requirements and performance targets. It is intended as the basis for reporting in business plans and Estimates documents. Departments are still required to get Treasury Board approval for their PRAS if they want to significantly change their reporting structure.

19.32 To provide a focus on results in the departmental performance reports and the reports on plans and priorities, departments are asked to prepare a chart of "Key Results Commitments" as a way of organizing their reports, and they are asked to identify the kind of information they will provide to demonstrate results. Work is ongoing in departments to align the PRAS with key results commitments and their business lines to ensure a link between expenditures and results. Changes in these charts are reviewed by Treasury Board Secretariat staff before the departmental performance reports are completed each year.

What We Expected in Reports

19.33 After discussion with Treasury Board Secretariat officials in 1997, we identified five characteristics (Exhibit 19.2) needed in performance reports to give a credible account of performance:

• clear presentation of context and strategies;
• clearly and concretely stated performance expectations;
• performance accomplishments reported against expectations;
• demonstrated capacity to learn and adapt; and
• fair and reliable information.

We would expect to see the first two characteristics in a good report on plans and priorities as well.

19.34 In examining performance reporting by departments, we considered each of these elements of good reporting. Our findings follow, starting with our observations on how departments stated the results they expected. Next, we describe how they reported their actual results, and then we present our observations on the remaining elements of good reporting. Finally, we look at the overall state of performance reporting after five years.

19.35 We recognize that it may take time to establish expectations and report accomplishments in a way that is meaningful and useful to the organization, but we would expect to see significant progress after four or five years, and many examples of good performance reports. Where reporting is still weak, we would expect to see departments informing readers of their plans to improve it.

Some Improvement in Stating Performance Expectations

A variety of terms used to state what is to be accomplished

19.36 On some of the key terms used in performance measurement and reporting, there is general agreement (these terms are set out in Exhibit 19.2). There is less agreement on the terms that describe other key elements. One important element of performance reporting is setting out the results a program is intended or expected to accomplish. A second key element is measuring the results it has actually accomplished.

19.37 Stating performance expectations. The term "objective" is often used as the statement of expected accomplishments but it tends to hide an important distinction between two different concepts. The first concept is a general statement of strategic direction, a more global statement of intent. The Treasury Board Secretariat has begun calling these the key results commitments - the benefits that departments have committed to provide to Canadians - and each department now sets them out in its Chart of Key Results Commitments.

19.38 The second concept is a statement identifying specifically what is to be accomplished over the next few years. This is what our 1997 chapter called performance expectations. Over the past several years, the Treasury Board Secretariat and departments have used a number of different terms - desired outcomes, specific goals, intended results, and desired results. More recently, the Secretariat introduced the term planned results for these more specific intended results, as the companion term to the more global results commitments. We will use "planned results" interchangeably with "performance expectations".

19.39 Measuring performance. In order to report performance, some way to measure it is needed. Performance measures or indicators are the terms usually used to identify a specific aspect of performance that can be measured, such as the number of full-time jobs created or number of cheques delivered on time to eligible recipients. Departments are asked to state in the chart of key results commitments the performance measures or indicators they will use to demonstrate progress toward their results commitments.

Setting concrete expectations remains a challenge

19.40 We examined the reports on plans and priorities of the 47 departments and agencies over the last six years. We looked at statements of expected performance - both the global results commitments and the more specific planned results. Exhibit 19.3 outlines the assessment we made and shows the average annual index score across departments. The scores indicate that progress in stating performance expectations has been slow overall, levelling off in the last two years.

19.41 However, statements of expected performance have become more outcome-focussed over the years. The reports for 2000-01 contained more clear and concrete statements of planned results than did those for 1996-97. Where progress was made, it was in more clearly stating expectations in terms of outputs or immediate outcomes and indicating a direction of change. There was little progress in specifying the timeframe for change and even less in stating the magnitude of change. Departments that scored highest usually stated expectations with both immediate and longer-term outcomes, and indicated the magnitude of change sought within a given timeframe. Exhibit 19.4 provides examples of clear and concrete performance expectations.

19.42 Departments have progressed to varying degrees in setting out expected performance. Some departments improved; others fell back - their later reports did not set out expectations in terms as clear and concrete as in their earlier reports. For example, in its 1997-98 Report on Plans and Priorities, Status of Women Canada set out, in reasonably clear and concrete terms, its performance expectations for contributing to equitable public policy:

Use of gender-based analysis at the planning and subsequent stages in government policy and program analysis and development, by the year 2002.

The corresponding expectation in the 2000-01 report, however, is less clear:

To have directed/led/driven the integration of gender-based analysis in the policy and program development of selected federal departments and key institutions.

19.43 Our Office continues to call for clear and concrete statements of the results that departments plan to achieve. Our 1997 chapter gave examples of planned results that were clearly stated and some that were not, some that were concrete and others that were not. (See Exhibit 19.4 for more recent examples.)

19.44 In our view, planned results need not necessarily be expressed as numerical targets to be concrete. As discussed in Appendix B, concrete expectations can include numerical ranges, a yes/no indication of a result, or a result indicated verbally on a comparative scale. And in setting out the specific result that is planned, it is useful to keep in mind the whole chain of outputs and outcomes to provide its context.

Inconsistent use of concepts

19.45 The performance reports we examined show widespread inconsistency in the content of key results commitments. They range from general statements of intent (in the vast majority) to quite clear and concrete statements of what is to be accomplished, that is, planned results. In the results commitment chart, the content under "to be demonstrated" is even more varied and hence much less useful to readers. Some departments give a list of activities to be carried out, and others a list of sub-objectives; some provide a list of performance measures, and still others, a list of performance expectations. And we have already noted the several different terms used to describe expected performance.

19.46 In our view, this miscellany of terms and concepts hinders reporting by confusing both the readers and the preparers of the reports. More uniformity in the types of performance expectations to be reported would help. The chart of key results commitments most recently requested by the Secretariat calls for two of the needed elements: results commitments and planned results. In the previous chart, the column "to be demonstrated by" was intended to capture the specific indicators by which a department would measure and report on accomplishments. This element of reporting is still needed. After several years of experimenting with terms and presentation approaches, departments should now be encouraged to use this three-part conceptual framework - key results commitments, planned results/performance expectations, and the specific measures that performance is "to be demonstrated by". Consistent use of this framework by departments would encourage the development of clearer and more concrete performance expectations and enable readers of performance reports to know what results they can expect.

19.47 The Treasury Board Secretariat and departments should agree on key elements of performance to be reported and the terms the reports will use. The Secretariat should provide stronger leadership for consistent common practice.

Slow Progress in Reporting Accomplishments

Improvements needed in reporting accomplishments

19.48 The essence of performance reporting is to report what has occurred and relate it to what was expected. The fall performance reports are to set out the results accomplished in the period up to 31 March of that year. Exhibit 19.5 discusses how we assessed the reporting of accomplishments, and shows how well departments have done overall. We found that in general, the reporting of accomplishments against expectations has made slow progress. About a third of the departments did show improvement year by year. And while half of the departments did better at reporting their accomplishments in 1999 than in their first year of reporting, a quarter of them did worse.

19.49 There is still a focus on outputs and activities. While most departments have been reporting more outputs over the years and listing fewer activities, in 1999 only six departments focussed more on outcomes than on activities and outputs. Some departments had drawn back from reporting outcomes: 14 did not report outcome accomplishments in 1997, and 16 in 1999. Movement overall has been toward a focus more on outputs than on outcomes, as departments have emphasized what they can control more than what they can influence. Reporting on outputs is necessary, but in the context of the outcomes they are meant to influence.

19.50 Exhibit 19.6 shows the move toward reporting more outputs. However, the number of outputs, activities and outcomes reported can be somewhat misleading if no account is taken of the quality of reporting. Although the exhibit indicates a declining trend since 1997 in reporting outcomes, an analysis of the way some of them are reported suggests that this is not necessarily negative. Many of the "outcomes" reported earlier were stated in very vague terms and without attribution; and departments reported their "results" at a high level - little more than a restatement of their objectives, which they claimed to have achieved. The trend toward reporting more outputs could be a much more realistic indication of what departments are actually measuring. From reporting claimed outcomes as accomplishments, some departments have progressed to reporting measured outputs as accomplishments. But good reporting of outcomes is still a challenge for many departments. A number of them continue to report their activities in the past year as accomplishments (see Exhibit 19.7).

19.51 Too much focus on the latest year. Most outcomes that governments seek take a number of years to achieve. A "performance story" focussed on outcomes requires a discussion of the chain of results over several years. In recognition of this fact, departmental performance reports clearly indicate on the front that they cover the period up to 31 March of that year rather than only the 12 months in the fiscal year.

19.52 Funds are provided annually, and some reporting of the outputs produced in the past fiscal year is quite appropriate - indeed, essential - to provide the context for the outcomes reported. However, we found that performance reports too often focus primarily or exclusively on the latest fiscal year. Reporting on one year encourages a focus on activities and outputs, at the expense of outcomes that may have resulted recently from activities and outputs of past years. Departments need to adopt a multi-year perspective on reporting to capture the outcomes they have accomplished and the related expenditures they have incurred.

19.53 Departmental performance reports are meant to report back on the key results commitments and the related specific performance expectations stated in earlier reports on plans and priorities. Evidence of accomplishments related to these performance plans must be provided and any changes in the plans explained. In many of the reports we examined, it was not easy to compare clearly what departments had accomplished with what they had planned. There are several reasons: the reporting structure used often changes, different wording may be used, and presentation can be confusing. Both the reports on plans and priorities and the performance reports have undergone experimentation and change over the past few years. There is now a need for greater consistency between the two.

19.54 The Treasury Board Secretariat and departments should ensure that departmental performance reports consistently and clearly report back on the expected performance set out in the reports on plans and priorities relevant for the period ending 31 March.

19.55 Only limited comparisons provided. We saw that as departments gained experience over the review period in reporting performance, there was a very slight increase in the number of them that made comparisons with previous years' performance. In 1999, about half provided at least one comparative reference, although very few departments provided comparisons regularly and systematically. Almost no departments compared their performance with that of similar entities (benchmarking). Readers of a performance report can judge whether performance was reasonable only by comparing it with something: with the previously stated expectations, with past years' performance, and/or with similar programs elsewhere.

More effort is needed to assess a program's contribution

19.56 Program managers do not control the outcomes they are seeking to accomplish. They undertake activities that produce the goods and services (outputs) that are expected to influence events toward those outcomes. As highlighted by Treasury Board Secretariat guidelines, it is therefore important that departments explore the extent to which their programs are contributing to outcomes.

19.57 Limited discussion of the influence that programs have on outcomes. Discussion in performance reports of the relationship among activities, outputs and outcomes had improved little over the period we reviewed. Few departments discussed how their activities and outputs had contributed to the accomplishments they were seeking. For example, only a few included a logic chart or results chain that linked inputs, outputs and outcomes. As a result, readers of a report would not have a clear picture of the department's contribution to longer-term outcomes, such as a healthy industrial sector or higher standard of living for citizens, which are often the kinds of end outcomes the government is trying to achieve.

19.58 A few departments, such as the Royal Canadian Mounted Police and Citizenship and Immigration Canada, did make some attempt to discuss the contribution they were making, primarily by listing the other jurisdictions and agencies that were working toward the same outcome. Citizenship and Immigration Canada describes the areas of co-operation with its partners, but neither it nor the RCMP attempts to discuss how each is contributing to the desired outcomes.

19.59 We think it is possible to take a more analytical approach to describing the contribution made by departments. A results chain (see Exhibit 19.1) that describes the outputs produced and the series of intended outcomes can help to "tell the performance story" and the roles of the different players in it. By identifying how outputs from various contributors have together led to a series of outcomes, the contribution of each can be shown. Similarly, the effects of the program's outputs can be presented with supporting evidence to demonstrate the case that the program played an influential role.

Evaluation findings can play an important part in reporting performance but are underused

19.60 A small number of departments included relevant results of evaluations in their performance reports. This was often useful, added information about the nature of the performance, its contribution to outcomes accomplished, and the factors associated with success or with underachievement. For example, in discussing a program, the RCMP's performance report summarizes in "Key Review" boxes the results of evaluations or reviews related to the program. In its Departmental Performance Report for 1999, Canada Economic Development for Quebec Regions made extensive reference to evaluations to identify what had succeeded and what had presented challenges. Exhibit 19.8 illustrates this.

19.61 As Treasury Board Secretariat guidelines on performance reporting make clear, evaluations can provide for better measurement of outcomes accomplished and better analysis of the real contribution a program is making. A more extensive and systematic use of program evaluations would enhance performance reporting, increase a department's ability to identify its contribution, and provide some insight into factors that have influenced performance. We noted that U.S. federal legislation requires agencies to include evaluation plans in their performance plans, and the findings in their performance reports. Including the plans for relevant evaluations in reports on plans and priorities as part of the "to be demonstrated by" information would identify more clearly the role evaluation will play in measuring a department's results.

19.62 Departments should include plans for relevant evaluations in their reports on plans and priorities, and any relevant findings from evaluations in their performance reports.

Need to Improve Balance and Indicate Reliability of Data

19.63 We assessed the presence of other characteristics of good performance reporting, namely, presenting a clear context and strategy, demonstrating a capacity to learn and adapt, providing fair and balanced information, and discussing the reliability of the data used.

More balanced reporting needed to enhance credibility

19.64 Departments have made little progress in identifying weak performance and showing how they have used the information to improve programs: the vast majority of performance reports contain no such discussion. There are exceptions, including Citizenship and Immigration Canada and the RCMP - who compare their accomplishments (or lack thereof) with their previously stated expectations, or discuss them transparently. However, most departments report only the "good news", and make little mention of performance that did not meet expectations.

19.65 Departments understandably are reluctant to report to Parliament and the public on performance that falls below expectations. However, selectively reporting only good news weakens the reliability of the entire performance report - including the good news. Members of Parliament have repeatedly said that they do not trust the credibility of reports that present only good news.

19.66 What is more important, departmental performance reports are reports by ministers to Parliament on what their departments have accomplished with the funds and authorities Parliament has provided to them. Ministers are accountable for all aspects of performance under their responsibility and for accounting fully for their stewardship of that responsibility.

19.67 Report what was learned. We have suggested that if departments present performance that did not meet expectations in a context of what they now know works and does not work, they can significantly reduce their potential embarrassment. They can also indicate how they plan to correct the problem. This approach not only makes for a more balanced and credible performance report but also provides value to the organization in proactively identifying problem areas and addressing them. We noted that Treasury Board Secretariat guidelines do stress both the importance of balanced reporting and the need to report the lessons learned about what does and does not work.

19.68 A good example of this can be found in the 1999 Departmental Performance Report of Citizenship and Immigration Canada. It notes that the Department fell short of its target for privately sponsored refugees by more than 20 percent, for two reasons: higher rates of refusal and longer processing times. The report outlines the actions the Department has taken to educate groups and sponsors in order to reduce refusal rates, and the operational changes it has made to reduce processing times.

More attention is needed to data reliability

19.69 Good reporting requires that information be reliable and presented fairly. Readers can be misled by information that is not accurate enough or does not present a balanced picture of what has been accomplished. Our audit did not examine the accuracy of the data in performance reports. Rather, we looked to see if the reports themselves discussed the reliability of the data.

19.70 We found little reference in the reports to the fairness and reliability of the performance information they contain. Very few reports mention the possibility that there may be shortcomings or problems in the data. And guidance from the Treasury Board Secretariat has paid little attention to the issue of data reliability.

19.71 We noted that several other jurisdictions give considerably more attention to this issue. Alberta, for example, requires that departments include in their performance reports a short description of the methodology used to arrive at any data presented. At the U.S. federal level, legislation requires agencies to discuss the verification and validation of information in their performance reports, and legislators have publicly stressed the importance of data quality. Exhibit 19.9 provides some good practices followed in the United Kingdom to improve the quality of performance information.

19.72 Our Office has pointed to a number of inaccuracies in the data or information provided in performance reports. For example, an audit of the Fisheries and Oceans fleet management program identified problems in the accuracy of information in the Departmental Performance Report (Chapter 31, Fisheries and Oceans - Fleet Management). In an audit of the RCMP earlier this year (Chapter 7), we found that information in the Departmental Performance Report was scant and inconsistent with information available in other sources. A previous audit of the Canadian International Development Agency (1998 Report, Chapter 21) observed that CIDA needed to review the information in its performance report more rigorously to ensure that it was accurate and balanced. We made a similar observation this year in Chapter 14 on CIDA's Canadian Partnership Branch.

19.73 The Treasury Board Secretariat should require that departmental performance reports indicate the extent to which performance information has been verified or validated.

19.74 Too early for cost information on results. The importance of proper cost information to good performance reporting and to managing for results continues to be a concern of this Office. In 1999, Chapter 21, we reported on the Financial Information Strategy (FIS) designed to help the government strengthen its management of business lines and its accountability to Parliament. Departments are to integrate financial information with non-financial (operational) information and use it both in day-to-day management and in accountability reporting to Parliament. The 1999 audit found that most departments were only starting to focus on the FIS, despite a target implementation date of 1 April 2001.

19.75 In October 2000, we reported our assessment of financial management capabilities in selected departments. A common gap we found again was the limited ability to integrate financial and non-financial information. Most departments indicated that they had only begun the task of integrating financial and operational systems. In our latest audit of the federal initiative at managing for results (Chapter 20 of this Report), we also found a poor link between results and cost information. Our examination of 10 program areas found that limited cost information was available to departmental managers.

19.76 This gap has clear implications for individual departments and for the government as a whole. Until they are able to integrate information on costs and on results, departments cannot give Parliament and Canadians a realistic picture of what it has actually cost to produce a given result. In addition, senior management cannot readily forecast the costs of major decisions, like a decision to increase or decrease the level of service a program provides.

19.77 Performance reports generally state expenditures by business line, indicating the cost of producing all the business line's results. However, a few departments have included some spending information by sub-business line or by key results commitment. The RCMP, for example, gives inputs (finances and staff) for some of its business line components, while Natural Resources Canada gives the costs of selected components. The Treasury Board Secretariat has a project under way for departments to share their practices for linking financial and cost information to results.

19.78 The Treasury Board Secretariat should enhance its efforts to identify and share with departments practical approaches to linking financial and non-financial information.

Performance Reporting Shows Modest Improvement

19.79 As we have mentioned, initiatives to improve reporting to Parliament have a long history. The 1980s saw the introduction of the Part III of the Estimates, which moved reporting toward a purpose-based focus. The current performance reports are a significant improvement over those early efforts. And even in the past few years, departments have considerably improved the presentation of their performance information. Most show a good effort to situate the department and many are quite reader-friendly.

19.80 It also is clear, as we observe in Chapter 20 of this Report, that the concept of "results", including outcomes, has become widely accepted as the basis for reporting performance. The structure of reporting is clearly based on a department's Chart of Key Results Commitments. These charts form the basis for setting out the expected performance and reporting the results actually accomplished. The culture seems to have become more focussed on results, even if full implementation has yet to come.

Some departments show progress in communicating performance

19.81 No department we reviewed was exceptional in reporting its performance; our review showed that the practice is still evolving. Some departments are still trying to find a workable format and approach, and no overall "best practice" or "preferred model" for performance reporting emerged among the departments we reviewed.

19.82 While we did not examine the accuracy of the data they used, we did note that a number of departments have made steady progress in reporting their performance over the last several years:

• Atlantic Canada Opportunities Agency
• Canada Economic Development for Quebec Regions
• Citizenship and Immigration Canada
• Royal Canadian Mounted Police
• Western Economic Diversification Canada

Their reports offer several examples of good reporting practice, particularly in communicating performance.

19.83 Good practices in some areas but weaknesses in others. In our view, the better performance reports stated, at a level below the key results commitments, what the departments had expected to achieve and then reported against these specific expectations what they actually achieved. The five departments we have indicated did this to different degrees, using different approaches. Atlantic Canada Opportunities Agency, for example, set out in a short table the general aim of each activity area, described the specific result expected, and stated the result achieved, followed by a short discussion.

19.84 The RCMP's Departmental Performance Report for 1999 provided for each program area a well-laid-out, one-page discussion of its planned results (such as contributing to a decline in violent crime), an outline of the problem, the strategies used, its partners, the actual results (outputs and outcomes) and its planned improvements. The performance story was quite clear. More concrete statements of expectations would improve the report, however.

19.85 Citizenship and Immigration Canada (CIC) has adopted a "performance at a glance" approach. Before the section on accomplishments it states the performance that had been expected in each area and indicates whether that performance was achieved or not. This would be a good method for reporting forthrightly on performance that is less satisfactory than expected. We note, however, that the expected performance could be stated in the CIC report in clearer and more concrete terms with a stronger focus on outcomes, which would allow the reader to judge whether the actual performance is what was expected.

19.86 Reports of other departments had good features. A number of other departments had innovative reporting practices. For example, Natural Resources Canada has experimented over the years with different approaches to reporting. One section of its 1999 Departmental Performance Report features tables of selected indicators that show the performance expected, the trend and the current status of performance for each indicator, a discussion of its significance and its relationship to Natural Resources Canada programs, and what the Department is doing to improve performance in that area. These tables are clear and easy to read, but they need to be integrated into the overall performance story the Department is telling about each of its goals and objectives.

19.87 Agriculture and Agri-Food Canada has also adopted a reader-friendly approach to performance reporting. Its 1999 Performance Report features a colour pull-out with a "report card" listing its key results commitments, and it has checked off those that were met. While the format is easy to read and the report is concise (especially given that it is a large department), the information it provides is highly aggregated and includes no clear and concrete performance expectations. Thus, it is difficult to determine whether the Department is performing well or not.

Good performance reporting overall is still some years away

19.88 Based on our examination of their performance information, the 47 departments together have made only slow progress since our 1997 audit in reporting their performance. Some departments have improved some aspects of reporting, but on the whole, the somewhat optimistic picture we presented in 1997 has not materialized.

19.89 The information the 47 departments presented in their reports on plans and priorities and their performance reports varied widely in quality. Some had made little progress over the three or four years; others had improved but had then fallen back below the level of their earlier reporting. Large and small departments differed little in the extent to which they improved their reporting.

19.90 Pilot departments did no better. We looked at the progress made by the 16 pilot departments (see Appendix A) that first reported in 1996, and compared it with the progress of other departments. Overall, their extra year of reporting experience and the additional flexibility they were provided did not put the pilot departments ahead of the others in the quality of their reporting. As a group, the pilots made uneven progress, and many other departments that had started a year later progressed further.

19.91 Some departments have not progressed. Several departments seem to have a lot of trouble "telling the performance story" in their performance reports. Many of their performance reports are little more than descriptions of activities carried out over the year. Performance expectations are vaguely stated and focus on activities, and they report as "accomplishments" the mere completion of those activities. Some of these reports have a section on how the department provides benefits to Canadians, but it is not linked to the department's activities and outputs and there is no discussion of intermediate outcomes so a reader can clearly see the connection.

19.92 Recommendations to departments in 1997 are still valid. The recommendations in our 1997 chapter on performance reporting are listed in Appendix C, which also indicates the extent to which they have been implemented. Overall, progress in addressing our recommendations has been unsatisfactory. The recommendations are still valid; the observations and recommendations in this chapter focus on further important areas that need to improve.

Treasury Board Secretariat Leadership for Reporting

19.93 Numerous efforts made over past years. The Treasury Board Secretariat has taken a relatively strong stance in promoting improved reporting to Parliament and, through an interactive and ongoing process, in developing departmental capacity for reporting. The Secretariat's efforts include the following:

• a Web site that contains a full array of past years' call letters, guidelines and reports (both reports on plans and priorities and departmental performance reports);
• regular information sessions (which were particularly well attended) when the Improved Reporting to Parliament Project was first introduced;
• the production of periodic question-and-answer newsletters;
• a symposium for practitioners in each of the past three years;
• peer review exercises; and
• internal technical assessments of reports.

19.94 We asked a few departments about the leadership shown by the Treasury Board Secretariat. They were generally satisfied with the Secretariat's support and leadership. The pilot departments particularly appreciated the flexibility they were given to try different reporting approaches. Yet there is a lack of practical guidance on reporting specific issues, like the performance of policy-making units and regulatory programs. Departmental officials said they would have appreciated more guidance on setting targets for results in such areas and defining appropriate indicators of performance.

19.95 Nevertheless, as already noted, progress by departments overall during these initial years of experimenting has been slow. Later, we discuss some possible reasons for this slow rate of progress, but it may be partly due to a lack of discipline in the reporting regime and the need for a stronger role by the Treasury Board Secretariat. As we have suggested, there are considerable differences in the way departments approach the basic elements of reporting like stating expectations clearly and concretely and reporting specifically against them.

Better electronic reporting needs to be encouraged

19.96 The Treasury Board Secretariat and some departments are working on innovative electronic or e-reporting. The Secretariat's Web site for the Improved Reporting to Parliament project provides a user-friendly guide to performance reporting. Users can access electronic versions of all departmental performance reports, reports on plans and priorities, and reports from the President of the Treasury Board for the current and previous year. They can also find an index of all references in performance reports to selected cross-department issues or government priorities, and an explanation of the Estimates process and the context in which these reports are produced.

19.97 Some departments have adopted their own innovative approaches to reporting through electronic media. Agriculture and Agri-Food Canada and Citizenship and Immigration Canada, for example, have each produced a more concise report by providing Web site references to more detailed information. This means that a reader can get more information on a particular area of interest by clicking on the appropriate Web address. A number of departments present their performance reports in a summary form that the reader can use to access details on particular issues through hot links. However, most departments simply post their performance reports on their Web sites.

19.98 More attention to e-reporting is needed. Electronic reporting solves the problem of providing both high-level summary information and more detailed information. Both serve various needs of parliamentarians and the public. Electronic reporting also opens the possibility of more timely analysis as well as horizontal and longitudinal analysis. This is clearly the direction of the future.

19.99 The Treasury Board Secretariat should encourage departments to share their experience in electronic reporting of departmental performance, and provide them with guidance on making their performance reports more concise and user-friendly.

Treasury Board Secretariat is focussing on collective reporting

19.100 In addition to its efforts to foster departmental reporting, the Treasury Board Secretariat has taken important steps to support collective reporting - reporting on performance in an area that involves more than one department or government. In recent years, it has taken the lead in government to develop a performance measurement framework for the Social Union Framework Agreement; has provided assistance to departments in developing their accountability and reporting frameworks; and has ensured that all Treasury Board submissions for funding of interdepartmental initiatives include a requirement for strong performance reporting. Lead departments of interdepartmental initiatives have indicated that they value the Secretariat's contribution.

19.101 The Treasury Board President's report provides valuable information. Since 1995, the President of the Treasury Board has tabled an annual overview document in Parliament. The first one, S trengthening Government Review, described how the government was managing, using and strengthening review. Subsequent annual reports have focussed more broadly on managing for results and reporting performance information, and have been tabled in Parliament along with all the departmental performance reports.

19.102 The President's reports are intended to provide an update on progress made by the government overall toward managing for results and reporting to Parliament on results. They have described government strategies for measuring and reporting on departmental, collective and societal results:

• In 1999, the report enhanced the information in the set of departmental performance reports tabled in Parliament by adding a compendium of all departments' charts of key results commitments.
• Successive reports have explored the presentation of performance information under each priority mentioned in the Speech from the Throne.
• Collective results are reported from different perspectives each year in response to requests from parliamentarians.
• Work to develop societal indicators is ongoing. The 1999 report made a significant contribution by presenting possible indicators to measure performance in three areas: Health, Environment and Public Safety; Economic Opportunity and Participation; and Social Participation and Inclusion.

19.103 However, it is true, as noted in the President's 1998 report, that "the government is not ready to give a comprehensive overview of government-wide performance." And the President's report itself is evolving, changing its focus and presentation every year. It does not report consistently what has been accomplished toward commitments made in previous years.

19.104 Nevertheless, important steps have been taken toward government-wide performance reporting. The challenge is to build on these initiatives, developing collective and societal indicators and presenting meaningful government-wide information in a convenient format for Parliament.

More Involvement of Parliamentarians Is Needed

Use by parliamentarians still limited

19.105 In 1997 we reported that the review and use of the new Estimates documents by standing committees was limited. This has not changed, essentially. Some committees have carried out reviews but most have not examined the Estimates documents. A major stumbling block identified frequently and noted in our 1997 Report is parliamentarians' frustration over the limits on the role they can play in the Estimates process.

Improvements proposed to the current regime for reporting to Parliament

19.106 Parliamentary committees have examined reporting to Parliament. Since our Report in 1997, the Standing Committee on Procedure and House Affairs has tabled two reports to the House of Commons that looked at reporting to Parliament. In December 1998, the Committee made a number of recommendations for new structures, tools, and incentives to improve the business of supply (see Appendix D for a synthesis of the recommendations and the government's response). It also firmly supported improved reporting to Parliament and stronger government accountability for Estimates matters. Two innovations that were suggested are noteworthy: the creation of a new standing committee on Estimates; and amendment of Standing Orders to give standing committees the ability to reallocate funds as an incentive to study the Estimates documents.

19.107 In its response, the government recognized the merits of the recommendations but did not support the substantial changes they implied. However, it agreed with the objective of improved reporting to Parliament and supported the continuation of its Improved Reporting to Parliament Project.

19.108 In its Thirty-Seventh Report tabled in June 2000, on Improved Financial Reporting to Parliament, the Standing Committee on Procedure and House Affairs made further recommendations. A subcommittee was struck to consider a number of proposals made by the government:

• Tailored Estimates information for Parliament. Performance plans would cover a three-year period. Performance reports would be based on key results and not processes, would disclose all relevant information, and would be based on government-wide costing standards. There would be increased reporting of results that involved more than one jurisdiction.
• Streamlined and consolidated reporting for Parliament. The government would enhance electronic access to information, consolidate reporting where possible, and make use of executive summaries.
• Strengthened parliamentary review. The role of standing committees would be strengthened by having them review the key results commitments of departments. The government would continue improving the Estimates reporting process, and encourage standing committees to review reports on plans and priorities and departmental performance reports.
• Improved reporting of expenditures by use. This proposal would consider integrating accrual accounting in the budgetary process and improve the spending information provided to Parliament in the Main and Supplementary Estimates.

19.109 For the most part, the subcommittee recommended that the government's proposals be accepted. In addition, it recommended that consideration be given again to establishing a standing committee on the Estimates, that more resources be provided to standing committees for review of Estimates documents, and that the reporting of activities or programs that span more than one department be consolidated. In testimony before the committee, this Office agreed in general with the Treasury Board Secretariat's proposals. Further detail on these proposals and the subcommittee's recommendations are in Appendix E to this chapter.

Members' suggestions for improvement

19.110 We interviewed a number of chairs and vice-chairs of standing committees as well as staff and researchers. We also looked at the reports of a number of roundtables on reporting to Parliament that had involved members, and at the results of several Treasury Board Secretariat surveys of members of Parliament. Members made a number of comments and suggestions for improving reports to Parliament and for their subsequent review:

• Changes are needed in the process for parliamentary review of the Estimates if it is to become more meaningful. Some members pointed to the recommendations made by the Subcommittee on the Business of Supply.
• The new Estimates documents are still not very user-friendly. Information on programs of interest to members can be hard to find, and tracking information from year to year can be difficult or impossible.
• Members need better dialogue with officials on reporting. One suggestion was that committees be more involved in the planning of performance reporting by, for example, reviewing performance expectations and results commitments.
• Departmental performance reports are not balanced, presenting only the "good news". As a result, their credibility is compromised.
• Another reporting weakness is the vagueness of performance expectations that "you can't hang your hat on."

Changes Are Required if Performance Reporting Is to Improve

The reporting regime has strengths

19.111 In reviewing the design and implementation of current reporting to Parliament, we noted a number of strengths:

• The clear intent is to focus strongly on reporting outcomes - the consequences of programs and services that matter to Canadians.
• All departments and agencies are reporting on their performance, and have done so now for three years.
• There is momentum: progress is being made, albeit slowly, and the government has made numerous efforts to engage parliamentarians on the issue of reporting.
• Government clearly sees the reporting initiative not as a singular effort but as part of meeting a larger need to manage for results, a key objective of its reform efforts.

19.112 Nevertheless, we found that progress has not been enough to ensure that departments and government will arrive at good performance reporting anytime soon. Clearly, along with some strengths there are a number of weaknesses in the current regime.

What is the problem?

19.113 Performance reporting has a long way to go. While we understand that good reporting takes time to develop, it is reasonable to expect that in the three to five years that departments have had, an organization dedicated to improving its performance reporting could have made significant progress. We were disappointed by the pace of the federal government's progress.

19.114 As we have noted, several departments have made steady progress each year, though they still have some way to go, and we would expect to see the same progress in the vast majority of departments and agencies. The uneven progress of some - the weakening of reporting, even - is a concern. At the current rate of progress, it will be many years before good performance reporting becomes routine. It is taking too long.

19.115 We had expected to see better progress and had not set out to try and explain the lack of progress. Nevertheless, three factors stand out as contributing to the current state of reporting:

• Too frequently, basic principles of good reporting are not understood or applied.
• Performance reporting takes place in a political environment.
• There have been no incentives available for good reporting practices or sanctions applied for bad.

19.116 Basic reporting principles not well accepted. As we have already observed, there is frequently confusion or disagreement over basic principles of reporting. For example we found the following:

• confusion over, and inconsistent interpretation of, what is needed to articulate results commitments, performance expectations and performance measures;
• persistent attempts to report performance for only the latest fiscal year rather than the more appropriate period, "up to 31 March of ..."; and
• infrequent recognition of the principle that performance reporting requires a meaningful comparison with previous levels of performance, with performance of similar programs elsewhere and/or with previously determined performance expectations stated clearly and concretely, so that readers can assess not only what was done - the results accomplished - but also how well it was done - how close it came to the intended outcome.

19.117 Poor performance reporting is safe reporting. In our view, perhaps the chief drawback at present is that an organization that does not work to improve its reporting suffers no adverse effects as a result. It may receive unfavourable comment from its parliamentary committee and some pressure to improve by mid-level officials in the Treasury Board Secretariat, but essentially departments can meet minimum requirements and not devote much attention to reporting. Reporting only a list of various activities carried out is a safe route to follow, ensuring for the most part that officials and the minister do not have to answer any tough questions about the department's performance as a result of its formal reports to Parliament.

Clear and common reporting principles are needed

19.118 In 1997, this Office outlined its expectations for good reporting (see Exhibit 19.2). We used them as criteria in this audit. The Treasury Board, too, has set out guidelines for reports on plans and priorities and departmental performance reports that are quite consistent with our expectations. There are also guidelines for departments on setting out the underlying framework for reporting results, namely the Planning, Reporting and Accountability Structure, which is approved by Treasury Board. All of these guidelines, however, have not been clear and consistent over the years and departments have used them inconsistently. As a result, it is likely that departments currently use a variety of criteria to determine what constitutes good performance reporting. Further, what has so far not been done on a formal basis is to engage the users of the performance reports - specifically, members of parliamentary committees - in stipulating what they want and expect performance reports to provide. Given the record to date, it may be time to move toward agreed reporting principles.

19.119 As we pointed out in 1997, an important feature of performance reporting and managing for results is that those involved use similar concepts and similar terms to describe the key elements of reporting, such as performance expectations.

19.120 We note that the CCAF has work under way with the Canadian Council of Legislative Auditors and senior officials of a number of Canadian governments, including the federal government, to explore principles of good reporting.

19.121 The government should work with Parliament and its auditor to agree on principles for reporting performance information to Parliament.

Performance is reported in a political environment

19.122 Reporting to Parliament on the performance of government programs and services takes place in a political environment that is frequently adversarial. Expectations for full and balanced reporting need to be tempered with the realization that what may be seen as a "report card" on government is a recent feature of public accountability. The intent of performance reporting is to enhance debate in Parliament and the public at large on the workings of government. But clearly, there is a risk that performance reporting could become a political tool of ministers or the opposition rather than a source of better information. At present, there is little incentive to report the "bad news". Performance reporting does not yet play a significant role in Parliament. It will take time for balanced performance reporting to become a useful factor in parliamentary debate and scrutiny.

19.123 The more that reporting is seen to provide a balanced and factual account of how well programs are working and what is learned when things do not work, the more effectively it can enhance parliamentary debate and build trust in government. The more routine that performance reporting becomes, the sooner it is likely to be accepted as a normal part of public accountability. The more that performance reports are seen to withhold information, the greater will be their vulnerability to political use.

More consideration of incentives needed

19.124 An award for good reporting is a good idea. The Treasury Board Secretariat is planning to present several awards each year for the best performance reports. An outside committee has been identified and the first awards were planned for fall 2000. In our view, this is a good step and provides some encouragement to departments to improve their reporting. It parallels the Auditor General's Award for Excellence in Annual Reporting by Crown Corporations, now in its seventh year.

19.125 Good reporting is based on good managing for results. The information that departments include in their performance reports should come from the information systems they use to manage. The more a department is managing for results, the more likely it will be to have good information for external reporting. Chapter 20 discusses a number of factors that can encourage a focus on results and hence improve reporting.

19.126 A role of assurance. While departments must be responsible for the integrity of their data, reporting regimes in other jurisdictions have often included the provision of assurance in the role of their external auditors. In Alberta, the Auditor General audits ministries' reporting of methodology and, at the same time, reviews their information systems and whether they are compatible with the ministries' internal control frameworks. In New Zealand, the Audit Office provides assurance on the fairness of reported performance achievements, while the audit office of Western Australia provides assurance that performance indicators are relevant and appropriate. The Texas audit office audits performance data for accuracy, while in Florida, performance measures proposed by state agencies are reviewed by a legislative office for validity and reliability.

19.127 At the federal level in Canada, the Office of the Auditor General assesses the performance information in the annual reports of three newly created agencies and provides assurance of its fairness and reliability.

19.128 A role for internal audit. Modern comptrollership contends that internal audit should audit and verify the accuracy and completeness of performance information. To date, however, this role for internal audit has not been substantively addressed. The procedures for verifying performance information are still not well defined, and internal audit units in departments generally lack the needed skills and experience. A draft Treasury Board policy on internal audit does call for it to play this assurance role. We agree with that policy. Internal audit ought to be the first place a department turns to for assurance of its data quality.

19.129 The Treasury Board Secretariat and departments should develop the capacity of internal audit to provide assurance to deputy heads on the quality of information reported to Parliament and information used for decision making.

A stronger challenge of expectations is required

19.130 In some jurisdictions, departmental performance expectations are challenged externally. We note that in many other jurisdictions, departments' statements of expected performance are subjected to a degree of challenge from outside the organization. Government caucus members of the legislature in Alberta and Ontario have been involved in reviewing performance expectations and setting targets, as well as reviewing reported results. In other cases, such as the U.S. and the states of Florida and Texas, legislative and executive oversight bodies review and challenge agencies' performance expectations and specific targets for performance as part of the budgetary approval process.

19.131 Treasury Board Secretariat review is limited. Treasury Board Secretariat officials provide some level of challenge to departments but it is informal and limited. Questions typically come from the Secretariat's liaison officer for performance reporting, not the Treasury Board analyst who is responsible for reviewing departments' requests for funding and for program changes and assessing program integrity (identifying strategic program issues and the critical risks to the continued achievement of results). The Secretariat may not always be in a position to question a quantity specified as a planned result - the magnitude of the change expected. But as a management board it could usefully challenge the way departments have stated their performance expectations and how they arrived at them. Results reported against clearly stated planned results would thus be a key element in the Secretariat's assessment of departmental programs.

19.132 Parliamentary review of expected performance would be beneficial. There would be an advantage to Parliament's involvement in reviewing departments' statements of expected performance. This review could be after the reports on plans and priorities are tabled in the spring, or part of a review of performance reports in the fall; it would provide a healthy challenge to departments. Involvement of parliamentarians in articulating expected performance would likely motivate departments to report back to Parliament fairly and reliably. It would also provide an incentive for parliamentarians to review and use the Estimates information more fully.

19.133 Equally important, however, would be an effective challenge within departments of both the performance expectations and the actual results reported back against them. This would normally involve the senior management of the department. Setting and challenging performance expectations is an integral part of effective planning and priority setting. If departments are to manage for results, they need to plan for realistic and challenging results and to state them as clear, concrete and time-specific performance expectations that all senior managers accept. In turn, the results reported against the expectations first need to undergo scrutiny by senior management, not only to assure the validity of the information but also to reaffirm its relevance. If the results achieved are not meeting the performance expectations, changes in the program may be needed, or the planned results may need to be reconsidered.

19.134 The Treasury Board Secretariat should provide a stronger challenge to departments' statements of expected performance.

19.135 The government should encourage departments to discuss their performance expectations with the related standing committees.

A stronger role for parliamentary committees

19.136 We suggested in 1997 that standing committees might wish to strengthen their review of Estimates documents. In February 1998, we provided all members with Parliamentary Committee Review of the Revised Estimates Documents, a report that described the Estimates system and offered suggestions for carrying out such a review. We still believe that a strengthened review would be useful for committee work and would influence departments to improve the performance information they provide.

19.137 Mechanisms for reporting to the House are important. In 1997, we noted that new arrangements were needed to facilitate reporting by standing committees on the fall departmental performance reports. At present, the rules of the House provide for committee scrutiny of the Main Estimates as part of the business of Supply. The committees also need the reports on plans and priorities in order to understand the votes in the Main Estimates. Since 1994, the rules have allowed committees to consider and report on those plans and priorities. In addition, a standing committee can study and report on any matter relating to the mandate, management and operation of the departments and agencies it oversees. While there is nothing to preclude committees' considering departmental performance reports and reporting their findings to the House, there is no rule explicitly empowering them to do so. The adoption of an explicit Standing Order would indicate the House's willingness to receive and consider committee reports based on the examination of departmental performance reports. It would encourage committees to hold more hearings and report more often to the House. For example, the House adopted a specific Standing Order in 1994 to empower the Standing Committee on Finance to hold pre-Budget consultations.

19.138 Parliament may wish to consider an enhanced role for its committees in reviewing Estimates documents, including review of the statements by departments of their expected performance.

Is it time for legislation?

19.139 As we have noted, in 1995 the government revised the Expenditure Management System and initiated the Improved Reporting to Parliament project to improve the Estimates documents. To date, these reforms have not been established in legislation.

19.140 Under the Financial Administration Act, the Treasury Board is responsible for all matters related to financial management, including the Estimates. The Treasury Board Secretariat has issued guidance to departments, requiring that financial and non-financial performance information be provided to Parliament. However, the form and content of the Estimates documents are not specified in legislation, and may be changed by the government, although in the past the government has also sought Parliament's agreement.

19.141 Parliament agreed in 1997 to split the Part III Estimates into the reports on plans and priorities and the departmental performance reports, but it did so only on a pilot basis. For the past three years, all departments have provided performance information to Parliament but there is no certainty that parliamentarians will continue to receive this type of information from departments after the pilot project ends.

19.142 In June 2000 the Standing Committee on Procedure and House Affairs recommended that parliamentary review be strengthened and, in particular, that the government reconsider the recommendations in the Committee's 1998 report on the subject. In the earlier report, the Committee had endorsed the pilot project to improve reporting to Parliament and had recommended that standing committees make full use of the information on departmental plans and performance in their study of the Estimates. The Committee considered that good performance reports should increase Parliament's ability to know what the government is accomplishing with the dollars and authorities granted to it and to effectively challenge government plans for priorities and spending in the future.

19.143 It is clear that good reporting on performance can increase Parliament's ability to scrutinize public expenditures. In light of the modest progress to date, we think a stronger and more politically visible initiative is needed, one that will focus the attention of parliamentarians and officials on reporting. In our view, it is time for the government to consider introducing accountability legislation.

19.144 Legislation alone will not ensure good reporting. But it could serve as a signal that Estimates reform is not a passing fad. The enactment of legislation would involve parliamentarians in designing a regime suitable to their needs. Moreover, legislation can secure further progress in a number of ways. It can:

• address concerns about the staying power of the federal reporting initiative. After five years of experimenting, legislation would ensure continuity over successive governments by formalizing the arrangement's permanence.
• provide an institutional anchor for results-based reporting initiatives, giving them standing to help ensure that they will continue to be taken seriously. The present initiative has clear objectives and some momentum, but it needs discipline and drive (see paragraph 19.111).
• provide Parliament with a focus for the scrutiny of public spending. Faced with the requirements of full and routine disclosure, departmental reporting may become more balanced and include performance that falls below expectations, making reports more credible to parliamentarians (paragraph 19.65).
• establish reporting principles and stipulate required form and content, based on experience to date. This would lead to a common basis for assessing the performance of departments, using consistent terms and interpretations (paragraph 19.118).
• allow central agencies to more effectively champion good performance reporting (paragraph19.95).
• confirm the goal of reporting performance not only by department but also government-wide or by sector (paragraph 19.104).

19.145 However, some pitfalls associated with enacting legislation must be recognized. Care needs to be taken to not impose too much rigidity in a legislated reporting regime. Conformance by requirement rather than by conviction would not be very productive. Accountability legislation will have the desired effect only if all concerned - ministers, parliamentarians and officials - show their continuing commitment to good reporting.

19.146 A number of other countries have recognized the need for legislation, as have several Canadian provinces. We noted in 1997 that some of these jurisdictions had enacted accountability legislation; since then, more have done so (see Exhibit 19.10).

19.147 The government should give consideration to legislative proposals that would require regular reporting of financial and non-financial performance information to Parliament by departments and, as appropriate, by the government as a whole, in order to demonstrate its commitment to a focus on results and to strengthen public accountability across government.

Conclusion

19.148 Good performance information is essential to informing Parliament and the public about what the government has accomplished with taxpayers' dollars. It is also essential to good management. Moving to a results-based culture is not easy and we recognize that it takes time to establish good performance reporting.

19.149 Nevertheless, we had expected to find noticeable progress by now, and we were disappointed. Only marginal progress has been made; the majority of departments and agencies we examined had progressed very little toward good performance reporting. This was despite the leadership shown by the Treasury Board Secretariat. At the present pace, it will take too many years for good reporting to become routine.

19.150 Aspects of current reporting to Parliament are sound: the focus on outcomes, the inclusion of all departments and agencies, the efforts made to engage parliamentarians, and the perception of reporting as part of a larger government priority - managing for results.

19.151 However, we think significant changes are needed in the current reporting regime to move things along:

• the development, communication and consistent use of basic reporting principles;
• a requirement to report on the reliability of performance data;
• enhanced efforts to link financial and non-financial performance information;
• a stronger internal and external challenge to the performance expectations set by departments;
• strengthened review by parliamentary committees of reports on plans and priorities and departmental performance reports; and
• consideration given to enshrining in legislation the requirement for reporting performance to Parliament.

Government's response: We note that the Auditor General agrees with the approach the government is taking to improve reporting to Parliament, but feels that the pace of improvement has been too slow. In paragraph 19.151 , he summarizes his recommendations to increase the pace of improvement.

Other than the recommendation that the government actively consider legislation as the means for confirming the current reporting approach, the government has actions under way or planned that respond to the Auditor General's recommendations. Results for Canadians: A Management Framework for the Government of Canada, tabled in Parliament in March 2000, provides the foundations for the changes in management culture that are also essential to improve reporting to the public and Parliament. Improved Reporting to Parliament is one of the six key initiatives identified with respect to the implementation of the Results for Canadians management framework.

Among other things, good reporting must reflect how departments actually manage and deliver results. The federal approach puts the emphasis on:

• results in serving Canadians;
• measuring performance, making improvements and learning from experience; and
• being open and transparent with Parliament and Canadians.

Better linking of results and cost information and other improvements in measuring performance are key elements of this plan, as is continuing to work with Parliament with the aim of improving electronic access and developing standards for reporting.

In addition to its recommendations, the chapter notes some of the recent achievements in such areas as collaborative initiatives and electronic reporting. The chapter is balanced, recognizing significant accomplishments to date, and identifying areas that need further improvement. For these reasons, we believe the chapter is helpful and can serve as a benchmark for measuring further progress.

About the Audit

Objectives

One objective of the audit was to assess progress in reporting performance information to Parliament, by examining the following:

• the current state (quality) of departmental performance plans and reports;
• the leadership provided by Treasury Board Secretariat in furthering reporting; and
• the use that standing committees make of performance plans and reports, and any impediments to that use.

We also set out to assess the strengths and weaknesses of the federal regime for reporting on performance.

Scope

We reviewed the Estimates documents of 47 departments and agencies, specifically their reports on plans and priorities for 1995-96 to 2000-01, and departmental performance reports up to and including 1999. These were the larger departments and agencies, including the 16 that had piloted performance reports in 1996, and a sample of smaller agencies. We followed up on the observations and recommendations of our 1997 Report, Chapter 5, Reporting Performance in the Expenditure Management System.

As this was a government-wide audit of the current status of performance reporting and its progress, we did not audit the reliability or accuracy of performance information in reports on plans and priorities and departmental performance reports.

We reviewed testimony from a number of hearings on Estimates documents held by standing committees.

Criteria

We assessed performance reporting against the criteria set out in our 1997 chapter on performance reporting, namely:

• clear presentation of context and strategies;
• clearly and concretely stated performance expectations;
• reporting of outcome-focussed performance accomplishments against expectations;
• demonstrated capacity to learn and adapt; and
• reporting of fair and reliable information.

More detailed expectations for good performance reporting are provided in Exhibit 19.2.

We expected that the Treasury Board Secretariat would:

• have procedures in place to ensure a reasonable rate of progress in improving performance reporting;
• provide adequate guidance and direction, leadership, and demonstrated commitment, including the use of incentives and the building of expertise in departments;
• gather and disseminate best practices and lessons learned;
• work with parliamentarians to develop means of enhancing parliamentary use of departmental performance information, and
• report to Parliament on the government-wide progress of performance reporting and managing for results.

We expected that parliamentarians would identify the use they make of departmental performance reports and the impediments to that use.

In examining the reporting regime, we looked for the extent to which it allows readers of performance reports to assess how well an entity is performing, supports accountability to Parliament, and is consistent with other government initiatives.

Approach

To assess the state of performance reporting, we developed a rating system based on our criteria for good reporting. We developed indices for the key criteria of setting out clear and concrete accomplishments and of reporting outcome-focussed accomplishments against expectations. We used a similar approach to examine the three other criteria of good reporting. We also undertook a more qualitative analysis, reviewing the Estimates documents for general readability, presentation, and consistency from year to year.

We conducted interviews with departmental and Treasury Board Secretariat officials and discussed improved reporting with a number of standing committee chairs and vice chairs. We also interviewed Library of Parliament staff about the use of performance information by parliamentary standing committees. We observed several standing committee hearings and reviewed related testimony and several reports pertaining to parliamentary involvement in the review of Estimates documents.

We reviewed Treasury Board Secretariat documents and attended several events sponsored by the Secretariat in support of performance reporting. We reviewed literature on accountability legislation and reporting practices in other jurisdictions.

Audit Team

Assistant Auditor General: Maria Barrados
Principal: John Mayne
Directors: Carol Motuz and Patricia MacDonald

Christina Brooks
Adèle Lamoureux
Erin Molloy
Tom Wileman

For information, please contact John Mayne.