Rating Selected Departmental Performance Reports

Main Points

5.1 We assessed the quality of the departmental performance reports of three departments—Environment Canada, Fisheries and Oceans Canada, and Natural Resources Canada—to determine whether progress had been made in improving the quality of their reports against our criteria. In assessing the departments' performance reports against our rating model, we found that the quality of the reports had improved marginally, at most, over the two-year period between 2002–03 and 2003–04. Over the nine-year period, between 1995–96 and 2003–04, we found that two departments had achieved some modest improvements while the other department showed mixed results. Despite these modest improvements in performance reporting over the nine years, the latest performance reports still fall short of our criteria for good- quality reporting.

5.2 We found that most of the performance reports provide a good overview of each department's organizational context and planned strategic outcomes. However, performance expectations are not always clear and concrete, and the information does not necessarily focus on program results. Furthermore, the reported results are not always balanced and supported by data sources and data limitations. As well, the reports provide little evidence that performance information is used to make decisions about improving program results in future years.

5.3 While we cannot generalize from the results of rating three reports in each of three years to trends across all government departments, our findings continue to leave us as concerned about the overall quality of reporting as we were in our previous audits and studies. Based on these findings, it is reasonable to suggest that, without greater effort by departments and scrutiny by parliamentary committees, any significant improvements in the quality of performance reporting may take decades rather than just a few years. In our view, this rate of improvement is not good enough for parliamentarians and Canadians to be able to hold departments and agencies to account for their performance.

Background and other observations

5.4 Parliament holds government accountable for spending in the previous year, influences future plans and priorities, and approves expenditures for the upcoming fiscal year. Parliamentarians should rely on performance reports to keep informed about the performance of the government on key issues. They should also use these reports to hold ministers to account for departmental spending. As well, Canadians want to know if their taxes were spent where parliamentarians intended. They also want to know if they received good value from their government in return for their taxes. Consequently, effective accountability to parliamentarians and Canadians depends in large part on good-quality performance reporting.

5.5 In response to this need, Parliament and the government have sought to improve the quality of departmental performance reporting for some time. For example, Part III of the Estimates was created in 1981 and then separated into planning and reporting documents on a pilot basis in 1995 and permanently in 1997.

5.6 Our Office has also been concerned about the quality of performance reporting. We have reflected this concern through our audits since at least 1988 when we found clear shortcomings while also noting some improvements. In 2000 we examined the quality of departmental performance reports and concluded that federal departments' and agencies' progress in improving the quality of their performance reporting to Parliament was disappointing.

5.7 To assist us in our subsequent work, we developed a formal set of criteria for good reporting and presented it in 2002 as a model for rating departmental performance reports. In the following year, we reported on the results of applying this rating model to the departmental performance reports of nine departments. While we identified some promising practices, together with many challenges, overall we found that the reports did not measure up against our model as well as we had expected.

5.8 This is the second time that we have used our model to rate a selection of performance reports. However, it is the first time we have compared a department's reports over one or more periods to assess progress in improving the quality of reporting. Overall, we found that progress in improving performance reporting to Parliament continues to be a challenge for the three departments, even after almost a decade of effort.

The government has responded. The government has responded to our observations and informs us that it will continue its efforts to improve performance reporting across government departments.

Introduction

Performance information continues to be important

5.9 Good-quality performance reporting is a key means for Canadians to hold their government to account. This is because Canadians want to know the value they are getting for their tax dollars and the difference that government departments are making in their lives.

5.10 Good performance reporting is also fundamental to effective governance and accountability to Parliament. In order to monitor government programs and services effectively, parliamentarians need to be provided with timely, accurate information about the cost and performance of key government programs. They also need reports that are relatively short and easy to read.

5.11 Departments should also use the same kind of performance information to help them produce better results. As we said in our 2000 Report, Chapter 20, Managing Departments for Results and Managing Horizontal Issues for Results, we associate good performance reporting with sound performance management. Good performance reports intended for external accountability should reflect how departments are managed internally. In our view, they should be a routine spin-off from sound performance management in a department.

5.12 For some time, the government has sought to improve the quality of information provided to parliamentarians by government departments. In 1981, it created the Estimates Part III in an effort to improve and expand the information on planned departmental spending and on departmental performance. In 1995, Parliament initiated the Improved Reporting to Parliament Project, which separated the planning and reporting documents in an attempt to provide parliamentarians with an improved structure for delivering complex information.

5.13 About 90 federal departments and agencies now submit a performance report to Parliament every fall. Each report should outline the accomplishments of the department to the end of the fiscal year just completed against the commitments made in its report on plans and priorities published the previous year.

5.14 Through our work we have supported parliamentarians in their efforts to improve the quality of departmental reporting (Exhibit 5.1). In 1997 we examined the state of the government's reporting regime. We found that the reporting framework was basic but sound and that it provided a promising start for reporting. However, when we looked again at performance reporting in 2000, we were disappointed that only modest improvement had been made in improving the quality of departmental performance reports.

5.15 The Eighth Report of the Standing Committee on Public Accounts for 2001 emphasized the importance of good-quality performance reporting. The Committee recommended that we "conduct random audits of the information contained in the performance reports of departments and agencies in order to verify, among other things, that the information contained in these reports is a fair representation of accomplishments against goals and objectives."

5.16 In 2002 we responded to this request by producing a formal set of criteria for good-quality performance reporting and presented the criteria as a model for rating departmental performance reports. These criteria elaborated on those developed in our earlier work.

5.17 Among other initiatives to advance performance reporting, each year the Treasury Board Secretariat produces guidelines for departments to use when they prepare their performance reports (Exhibit 5.2 ). The guidelines include performance reporting principles that are generally consistent with our criteria.

Criteria for rating departmental performance reports

5.18 Our model for good-quality departmental performance reports encompasses the following five criteria:

• Organizational context and strategic outcomes are clear.
• Performance expectations are clear and concrete.
• Key results are reported against expectations.
• Performance information is credible and balanced.
• Use of performance information is demonstrated.

5.19 A report that fully meets all five criteria would demonstrate the attributes of excellent reporting. However, the rating model is not designed to provide assurance that information in a performance report is reliable because it does not include an audit of performance information.

5.20 Overall, the five criteria, each of which includes several subcriteria, represent reasonable expectations for a credible performance story about the difference that a department's programs and services are making to Canadians. The first criterion identifies the business the department is in, the second brings forward the commitments made the previous year, and the third presents the accomplishments against those commitments. The other two criteria provide the basis for assessing whether the results reported are believable and how the performance information can be used to produce better results in subsequent years. The model is summarized in Exhibit 1.1 of our May 2003 Chapter, Rating Departmental Performance Reports.

5.21 Our model of what makes a good-quality performance report provides for five levels of achievement—basic, fair, good, very good, and excellent. We anticipate that, as departments learn about what contributes to excellence in reporting and what level they have achieved, they will strive to reach a higher level for each criterion. In this sense, our rating model is also a learning or developmental model.

5.22 However, because some criteria build on other criteria, it may be difficult to achieve a higher level on a later criterion if the level of reporting on an earlier criterion is weak. For example, if performance expectations are not clearly and concretely stated, it will be difficult to report any results against them.

5.23 When a departmental performance report approaches the upper levels on all five criteria, then an auditor might be able to provide written assurance to parliamentarians and Canadians that the report fairly presents the department's accomplishments against its expected results. However, because auditors normally provide assurance on both the fairness and the reliability of information in reports, then additional audit work would have to be done on the reliability of the performance information prior to providing this assurance.

Focus of the audit

5.24 The objective of our audit was to assess departmental performance reporting by assessing the overall quality of departmental performance reports of a small group of departments to determine the extent of progress made in improving the quality of the reports. Specifically, we assessed the progress made over a two-year period, between 2002–03 and 2003–04, compared with the progress made over nearly a decade, between 1995–96 and 2003–04.

5.25 We selected the performance reports of 3 departments—Environment Canada, Fisheries and Oceans Canada, and Natural Resources Canada. These departments were among the 16 pilot departments participating in the Improved Reporting to Parliament Project that started in 1995.

5.26 We chose these departments because they share some common elements. First, many of their programs are oriented toward protecting the environment while also promoting resource utilization. Second, much of their work involves scientific and other activities that do not provide services directly to Canadians but instead contribute to the work of other departments, organizations, or levels of government that have a more direct impact on Canadians.

5.27 In using the rating model developed in 2002, we wanted to have consistent standards for all of our ratings; thus, we applied current expectations for good performance reporting to the reports from all three years (1995–96, 2002–03, and 2003–04).

5.28 We did not audit the role of the Treasury Board Secretariat because the President of the Treasury Board tables the performance reports in Parliament on behalf of departments and agencies. The primary accountability for the performance reports is with departments and agencies.

5.29 More details on the audit's objective, scope, and approach are provided in About the Audit at the end of the chapter.

Observations

Progress continues to be disappointing

5.30 We expected to find gradual improvements in the quality of performance reporting from one year to the next, with a clear increase in the quality of the reports over the longer term. However, we found the results to be disappointing.

5.31 As expected, in the three departments' first efforts as participants in the Improved Reporting to Parliament Project, their reports for 1995–96 achieved ratings that ranged from almost the basic level through to levels between fair and good. Our ratings on individual criteria for both the 2002–03 and 2003–04 reports ranged from almost the basic level to between good and very good. However, only one report among the nine reports rated achieved a very good rating on one of the criteria.

5.32 Each of the three departments introduced some promising practices over the past nine reporting years. For example, all three reports of Fisheries and Oceans Canada provide a table that shows a crosswalk of the Department's business lines by the strategic outcomes; this enables readers to see which parts of the organization contribute to each outcome and who is responsible. In both its 2002–03 and 2003–04 reports, Environment Canada offers an opportunity for readers to provide feedback through a questionnaire.

5.33 All three departments told us they made continuous efforts to improve their reporting practices from one year to the next. They also said they worked to satisfy the guidelines, released by the Treasury Board Secretariat each year, for preparing departmental performance reports. For example, in response to Treasury Board Secretariat guidelines, the 2003–04 Performance Report of Natural Resources Canada provides a summary of its performance that includes key commitments by each strategic outcome together with a self-assessment of how well the Department performed and a reference to the section in the report where supporting information can be found.

5.34 In response to the recommendations outlined in the Sixth Report of the Standing Committee on Government Operations and Estimates in 2003, departments were asked to include a short summary of parliamentary committee reports relevant to their work along with a link or reference to further information. All three departments did this in their 2003–04 reports. This may contribute to a better-informed dialogue between departments and House committees.

5.35 We found that, while parliamentarians would likely be able to readily grasp a department's main business when they read the reports, they would be less likely to clearly see the results of the department's business. For example, through the use of Web links, Fisheries and Oceans Canada makes information available about the state of the fisheries but does not present this information in its performance report. While Natural Resources Canada reports periodically to Parliament on the reserve levels of Canada's most important metals, its performance report only states that the reserve levels of these metals is a concern.

5.36 To some extent, our ratings of the reports of the three departments reflect the complex nature of their business. As noted earlier, the work of these departments may involve the sometimes competing demands of resource utilization and resource protection. Further, much of their business is science-based, and their services to Canadians may be delivered through other organizations. However, while their business may not conveniently lend itself to performance reporting, the three departments told us of their ongoing efforts to improve the quality of their reporting over the past nine years.

5.37 In our assessments of the three departments' performance reports against our rating model, we found that the quality of the reports improved marginally, at most, over the two-year period. Over the nine-year period, we found that two departments achieved some modest improvements while the other department showed mixed results. Although we cannot generalize from the results of rating three reports in each of three years to trends across all government departments, our findings from this audit continue to leave us as concerned about the overall quality of reporting as we were in our previous audits and studies.

5.38 Despite examples of promising practices that demonstrate it is possible for departments to improve on some elements of reporting to Parliament, these promising practices are overshadowed by the disappointing results found when we applied our rating model to the reports. These departments continue to face many challenges in their performance reporting.

5.39 Based on our findings, it is reasonable to suggest that, without greater effort by departments and scrutiny by parliamentary committees, any significant improvements in the quality of performance reporting may take decades rather than just a few years. In our view, this rate of improvement is not good enough for parliamentarians and Canadians to be able to hold departments and agencies to account for their performance.

Organizational context

5.40 Our first criterion for good performance reports requires that a department clearly communicate what business it is in and what it does for Canadians. It can do this by ensuring that its organizational context and strategic outcomes are clear and that they reflect the departmental mandate and mission. The criterion expects that reports will clearly communicate how a department's activities are organized, such as through business lines. It also requires a logical sequence from the department's business lines to the key results, which are clearly expressed as planned strategic outcomes. These should take into consideration the department's operating environment. As well, this criterion requires key external partners to be identified and a credible description of the risks the department faces to be provided.

5.41 We expected that each report would provide a meaningful performance story of the department's work by placing it in this broader context. We also expected that the quality of these performance stories and the descriptions of their operating environment would have improved over the nine-year period.

A good overview of the organizational context is generally provided

5.42 The reports generally achieved higher ratings on this criterion than the other four criteria in our rating model. Overall, for all the reports we rated, reporting for this criterion ranged from almost basic to between good and very good, with one report achieving a very good rating (Exhibit 5.3). Over the two-year period, the quality of reporting remained consistent for two departments but improved for the third. Over the nine-year period, two of the departments improved on this criterion, while the third department declined. Notably, Environment Canada's reports improved over both periods.

Some departments have adopted promising practices

5.43 In both 2002–03 and 2003–04, the reports of Fisheries and Oceans Canada and Environment Canada provided good descriptions of their operating environments. They also provided accountability structures and identified key partners. Both departments provided Web links in their reports so that readers can get more information about their partners. This approach helps to keep the reports focussed on performance. Fisheries and Oceans Canada also reported on the risks that influence the Department in achieving its strategic outcomes.

Some challenges remain

5.44 A performance report should state the department's main business by providing its mission and mandate. This information tells readers what the organization's purpose is and provides reference to the legislation that makes it possible for the organization to do its work. In both 2002–03 and 2003–04, the performance reports of Natural Resources Canada did not provide the Department's mission and mandate, while the Department's report on plans and priorities for the corresponding years provided the mandate. The rating for these performance reports could easily be improved against our model by including both the Department's mission and mandate.

5.45 The reports of all three departments could be improved by providing a discussion on how their strategic outcomes contribute to relevant government priorities. As well, they should discuss any challenges faced in balancing the departments' own work with their contributions to joint work with other departments.

Performance expectations

5.46 Our second criterion expects performance expectations to be clear and concrete. More precisely, departments should clearly state their commitments to Canadians. These commitments or performance expectations are initially provided in a department's report on plans and priorities and should be clearly identified and aligned with federal government priorities. The expectations are then carried over into the ensuing performance report. As well, performance expectations should be expressed as outputs or outcomes. They should also specify the direction of planned changes—for example, whether certain outputs or outcomes will be increased, maintained, or decreased as a result of the department's program activities. Finally, the time frame in which those changes will be made should be specified.

5.47 We expected departments to state their expectations in a clear and precise way, accompanied by sufficient information to clearly show how the departments' activities and strategies would meet these expectations. We also expected the quality of these statements to have improved over the nine-year period by increasingly being expressed as outputs and outcomes, rather than just as activities.

Performance expectations are not always clear and concrete

5.48 Overall, for all the reports we rated, reporting for this criterion ranged from basic to fair on our model (Exhibit 5.4). Over the two-year period, two of the departments achieved a higher rating in 2003–04 than in 2002–03. Over the nine-year period, one department improved, and two showed no change.

Use of results chains improves reporting

5.49 In 2002–03 and 2003–04, Fisheries and Oceans Canada's reports presented good results chains or logic models that summarize at a glance the Department's performance expectations, partnerships, business lines, and activities. Environment Canada's 2003–04 Performance Report provided a logic model for each business line that also outlines the Department's performance expectations in terms of immediate and intermediate outcomes.

Improvements are needed

5.50 Despite being rated from basic to fair on our rating model, some key performance expectations presented in the reports of all three departments were inconsistent with the expectations presented in their planning reports. These changes to the performance expectations were not explained in the performance report.

5.51 The reports could be significantly improved if performance expectations were expressed as outputs and outcomes, with each commitment having a clearly-stated direction, amount, and time frame. We would also like to see a description of the strategies to be used for producing better results in subsequent years.

Key results reported against expectations

5.52 The third criterion of our model requires key results to be reported against performance expectations. This criterion expects results to be presented as outputs or outcomes. It also expects reports to state whether the department achieved its commitments and to explain any gaps in performance. This includes aligning outputs and outcomes with performance expectations, addressing the challenges of achieving expected results, identifying the partners that contributed to the performance outcomes, determining the level of resources required, and interpreting the results. We believe that this criterion is at the very core of good performance reporting.

Outputs and outcomes are not widely reported

5.53 Overall, for all the reports we rated, reporting for this criterion ranged from basic to fair, with only one report approaching a good level of reporting (Exhibit 5.5). The results reported tended to focus largely on activities rather than on the outputs produced and on the outcomes to which they contribute. According to our rating model, over the two-year period two departments marginally improved their reporting, and the same two departments improved by up to one level over the nine-year period. Given the government's emphasis on results-based management over the past decade, we expected to see greater improvement on this criterion.

Some promising practices are provided

5.54 Despite finding some limited improvements on this criterion, some promising practices are provided. In 2003–04, while all three departments' reports made some reference to evaluations and internal audits, only Natural Resources Canada integrated the findings of an evaluation with its reporting of departmental results. All three departments also provided an estimate of the level of resources used to achieve each strategic outcome.

5.55 The 2002–03 report from Environment Canada provided an Internet link to a table that displayed the Department's expectations and how well it has performed against them. However, this table would be more useful if it were shorter and included in the performance report itself.

Challenges remain

5.56 The performance reports could be improved by systematically integrating the findings from evaluations and audits in discussions of the results. Estimates of the resources used to achieve concrete targets should also be provided. The gap between the planned results and the actual results should be explained. As well, information should be provided so that results achieved can be interpreted, such as trends over the past five years or comparisons with other departments.

Credible and balanced results

5.57 Our fourth criterion requires performance information to be credible and balanced. First, it focusses on the quality of performance information and the reliability of information sources as the basis for judging the credibility of the data. Second, it focusses on whether the reporting of good results is balanced with the reporting of shortcomings and whether the level of detail for key results is appropriate. This criterion is fundamental to a compelling and credible performance report.

5.58 We expected to find evidence to show that the information was credible. We also expected that results falling short of expectations would be reported along with the success stories. Further, we expected that the amount of information provided for the key results would be proportional to their importance.

Departments do not generally report credible and balanced results

5.59 Overall, for all the reports we rated, reporting for this criterion ranged from almost basic to fair (Exhibit 5.6). Over the two-year period, one department improved on this criterion, one showed no change, and one declined. There was no improvement on this criterion over the nine-year period for any of the departments, and we could not find any promising practices.

Many challenges remain

5.60 The reports could be improved by reporting data sources and limitations. They should also provide a discussion on how to interpret their performance information, including the relevance or importance of the performance achieved. It is important to report performance shortcomings as well as successes; this would reduce the risk that users, such as parliamentarians, might discount the positive results as biased, incomplete, and lacking in credibility.

Use of performance information

5.61 Managing well to improve results is a challenge. The previous four criteria are based on a more traditional approach to assessing performance reporting. For these criteria, the organizational context must be provided; the performance should be defined, measured, and reported against expectations; and the information must be credible and balanced. However, even if all these criteria have been met, if feedback on past performance is not used to improve future results, then the use of the report may be restricted to addressing the external accountability requirement of reporting to Parliament.

5.62 Performance information is not collected just to create accountability reports; it should also be used to help departments make sound internal decisions to manage for better results. For example, when evaluations of programs provide evidence about which programs work well and which ones do not, then funding could be redirected away from programs that have been shown not to work well to those that do work well. In addition, the performance report should provide a credible discussion about the capacity of a department to produce sustainable results by continuing to perform well in the future. A good-quality report also highlights lessons learned and identifies how weak performance will be corrected in future years.

5.63 We expected that performance reports would demonstrate ways in which departmental management is using performance information, including how it will use the information to establish future performance expectations and improve future performance.

Performance information is seldom used

5.64 Overall, for all of the reports we rated, reporting for this criterion ranged from almost basic to between basic and fair, with most reports achieving only a basic rating (Exhibit 5.7). According to our rating model, over the two-year period one department improved, one showed no change, and one declined. Over the nine-year period, two departments improved on this criterion, while the report of the third department remained the same.

Use of performance information in decision making improves reports

5.65 In 2003–04, Natural Resources Canada reported how it used the results provided by an evaluation about road surfaces and the fuel efficiency of vehicles. Based on this information, resources previously devoted to research on concrete roads were redirected to research on other road materials. We consider this a promising practice because it illustrates how the Department used performance information to potentially improve future performance.

Future performance needs to be discussed

5.66 The reports could be improved by including both explanations of corrective action taken to address performance issues and plans to improve performance. Reports should also discuss the risks associated with continuing to either meet or make progress on performance targets in the future. The ratings on this criterion suggest that this could be one of the most challenging aspects of performance reporting. In facing this challenge, departments will need to incorporate good performance information into their decision-making processes, evaluate whether they are heading in the right direction, and use the information to correct problems.

Conclusion

5.67 This is the second time that we have used our model to rate a selection of performance reports. However, it is the first time we have compared a department's reports over one or more periods to assess progress in improving the quality of reporting. Overall, we found that progress in improving performance reporting to Parliament continues to be a challenge for the three departments, even after almost a decade of effort.

5.68 We found that, between 1995–96 and 2003–04, two departments at least doubled the number of pages in their performance reports. We are not convinced that the greater length has translated into a proportional increase in the quality of the reports. Some of the greater length can be attributed to additional requirements from the Treasury Board Secretariat and the trend toward greater complexity of the information. However, some of the increased length is a reflection of more information being reported than is necessary and information being too detailed for the intended readers. While the 1995–96 reports were handicapped by focussing largely on activities rather than on outputs and outcomes, their performance stories were more succinct and, therefore, more understandable.

5.69 The findings from this audit are consistent with our observations from our previous work. In 1988, we concluded that reports to Parliament did not provide a fully satisfactory basis for accountability. In 1992, we again indicated that these reports did not provide the necessary breadth of information. In 1997, when the government reviewed its Expenditure Management System, we stated that progress in performance reporting to Parliament was being made. In our 2000 Report, Chapter 19, Reporting Performance to Parliament: Progress Too Slow, we concluded, "At the current rate of progress, it will be many years before good performance reporting becomes routine." In 2003, when we rated nine reports, we found that progress in performance reporting remains disappointing.

5.70 In our previous work, we listed five factors that contribute to the quality of performance reporting. We believe these are still valid.

5.71 First, the basic principles of good reporting are frequently not well understood or applied by departments. This remains a factor, despite the Treasury Board Secretariat's annual guidelines that reinforce the principles of good reporting.

5.72 Second, performance reporting takes place in a political environment. This continues to be a factor because the value of reporting shortcomings has not yet been widely accepted. Balanced reporting, that is, admitting to shortcomings as well as successes, is apparently not yet part of the management culture of government.

5.73 Third, there are no incentives for good reporting practices or sanctions applied for poor reporting. Because House committees do not control departmental budgets, as their counterparts do in some other jurisdictions, it is more difficult for parliamentarians to encourage good reporting or penalize poor reporting through the use of financial measures. As well, some jurisdictions scrutinize performance reports and name the poorly-performing departments as a way of motivating them to improve their reports.

5.74 Fourth, many departments do not consider performance reports to be a high priority. Often, the reports do not get the involvement or attention of senior departmental management that they should. This may be because information in performance reports is neither used by departments to manage for results nor used by the government to manage government-wide initiatives.

5.75 Fifth, despite initiatives of Parliament and the government aimed at improving the quality of performance reports over the past few decades, in our view parliamentary committees have not taken advantage of the reports in their discussions with the departments audited. Greater interest in and scrutiny of the reports by parliamentary committees would encourage departments to improve the quality of their performance reports.

5.76 These factors cannot be overcome easily. They can only be addressed by fundamentally changing the management culture of government. Consequently, we are not making any formal recommendations to departments because our message continues to be that they should make a greater effort to better understand the basic principles of good reporting and they should continue to work on improving their performance reports. While the onus is on departments to improve these reports, the Treasury Board Secretariat should also continue to help departments by reviewing their performance reports and providing leadership and opportunities to increase their understanding and acceptance of the principles and practices of good performance reporting.

5.77 Parliamentary review of performance reports through the various standing committees is essential for improving the quality of performance reports. Committees could demand clarification and explanations of departmental results. They could then challenge departments and agencies on how well they have performed against their commitments to Canadians. Only when parliamentarians are engaged with departments in an ongoing dialogue about departmental performance will the quality of these reports improve.

Government's response. We welcome the efforts of the Auditor General in assessing departmental performance reports and in providing advice on areas that need improvement. The Government of Canada remains strongly committed to ongoing improvements in reporting to Parliament, recognizing that sound, transparent, and effective reporting is key to supporting Parliament in its role of holding the government to account for the management and use of public funds.

Reports on plans and priorities and departmental performance reports are primary instruments of accountability to Parliament. They are a mechanism through which ministers and their organizations demonstrate responsibility and accountability. These reports explain to Parliament their departmental plans, priorities, and expected results, and account for the performance achieved. In collaboration with departments, the Treasury Board Secretariat (TBS) has recently revised its departmental guidelines and reinforced the results-based focus of these reporting instruments. The TBS will continue to work with departments, parliamentarians, and the Auditor General to help improve practices highlighted in the Report.

In December 2004, the Treasury Board Secretariat introduced the Management, Resources and Results Structure (MRRS) Policy. It will provide a standard, government-wide approach to planning and reporting on resource expenditures and results. To ensure better quality of performance reporting, the TBS will provide assistance and guidance to departments as they develop and adjust their management, resources, and results structures. This is a major shift in planning and performance reporting practices and will require a phased-in approach to full implementation. We expect that the MRRS policy will serve as a consistent and enduring foundation for financial and non-financial reporting to Parliament.

In the coming months, the government, in collaboration with parliamentarians, will develop a blueprint for improved reporting to Parliament that will better respond to needs of parliamentarians and other stakeholders for transparent, timely, and easy-to-understand performance information. The government will also invest in technological systems that support performance data and reporting. Combined, these efforts should bring about a gradual shift in the management culture and in performance reporting practices across government departments.

We welcome the opportunities for continued collaboration with the Auditor General, parliamentarians, departments, and others to enable the required shift toward improved performance reporting and management practices.

About the Audit

Objective

The objective of this audit was to assess departmental performance reporting by assessing the overall quality of departmental performance reports of a small group of departments to determine the extent of progress made in improving the quality of the reports. Specifically, we assessed the progress made over a two-year period, between 2002–03 and 2003–04, compared with the progress made over nearly a decade, between 1995–96 and 2003–04.

Scope

We selected the performance reports of three departments—Environment Canada, Fisheries and Oceans Canada, and Natural Resources Canada. These departments were among the second round of pilot departments participating on the Improved Reporting to Parliament Project, which provided for the splitting of the Estimates Part III into planning and reporting documents.

We did not audit the Treasury Board Secretariat.

Approach

We assessed three performance reports from each of three departments against our rating model. These reports covered the period ending 31 March 1996, 31 March 2003, and 31 March 2004. We followed the approach that we described in our April 2002 Report, Chapter 6, A Model for Rating Departmental Performance Reports, and applied to nine departmental performance reports in our May 2003 Report, Chapter 1, Rating Departmental Performance Reports.

In the rating model, each of the five criteria is represented by a continuum that has five levels or stages of development. These are basic, fair, good, very good, and excellent. An exceptional performance report would achieve an excellent rating in each criterion.

Three raters (two from separate teams within our Office and another from outside the Office) applied the rating model to the performance reports to produce their rating. They then discussed their ratings to arrive at a consensus rating.

Although we used the rating model developed in 2002, we applied current expectations for good performance reporting to the reports from all three years (1995–96, 2002–03, and 2003–04).

We did not audit the departments' systems and procedures for producing the information included in their reports or reach a conclusion on the reliability of the performance information contained in the reports.

Audit team

Assistant Auditor General: Hugh McRoberts
Principal: Barry Leighton

Doreen Deveen
John McGrath
Anupheap Ngoun
Ruth Sullivan

For information, please contact Communications at (613) 995-3708 or
1-888-761-5953 (toll-free).