A Model for Rating Departmental Performance Reports

Main Points

6.1 Departmental performance reports are an important means for Parliament to hold ministers to account for the money their departments spend and the results they achieve. A good performance report tells a department's or an agency's performance story. It tells Parliament what difference a department or an agency has made for Canadians, by presenting a coherent picture of performance. Good performance reports should tell Canadians what value they are getting for the taxes they pay.

6.2 As we reported in 2000, federal departments and agencies have made some progress over the past seven years in improving the quality of their performance reporting to Parliament, but their progress has been too slow. In 2001, a report by the House of Commons Standing Committee on Public Accounts stressed the value of reporting on performance. The Committee asked that we continue to assess departmental performance reports.

6.3 This chapter is a response to the Committee's request. It provides a method to assess the quality of performance reports. We offer a model for rating departmental performance reports by five criteria for good reporting; the model identifies different levels of reporting.

6.4 We expect that the rating model will be one of the tools available to departments that will help them improve their performance reports more rapidly than they have in the past. When a performance report is rated on this model over a number of years, the model can also be used to assess the department's progress in reporting.

6.5 Rating a department's performance report enables parliamentarians to do the following:

• compare the report with those of other departments that have also been rated;
• ask the department to take specific steps that will improve its report; and
• assess the department's progress in improving its report if it has been rated previously.

6.6 We demonstrate the usefulness of our rating model by applying it to three recent performance reports: those of the Royal Canadian Mounted Police, Environment Canada, and Fisheries and Oceans Canada.

Background and other observations

6.7 Most federal departments and agencies submit a performance report to Parliament every fall. The report outlines what the department has accomplished over the past year toward the commitments it made in its earlier report on plans and priorities.

6.8 We first commented on this reporting regime in 1997. We found that a good start had been made and that the basic reporting framework was sound. In 2000 we followed up on that government-wide audit with another assessment of the government's progress in reporting performance. We described the strengths and weaknesses of the reporting regime and said that progress was too slow.

6.9 Since 1995, when the Improved Reporting to Parliament project began, the Treasury Board Secretariat has played a leadership role in improving the government's performance reporting regime. In its guidelines for preparing the 2000-01 departmental performance reports, the Treasury Board Secretariat included principles of good reporting and a lexicon of reporting terms.

6.10 Based on our previous work, on pilot testing of the rating model, and on consultation with a variety of experts, we have elaborated on the five criteria for good performance reporting that we introduced in 1997. They are consistent with the principles set out by the Treasury Board Secretariat in 2001. Improvements to the model will be made as we continue to use it, and further changes will be suggested by departments as they gain experience from applying it to their own reports.

The government has responded. The government is generally supportive of this chapter and our model. Its comments are included at the end of the chapter.

Introduction

Good performance reporting is fundamental

6.11 Good performance reporting is fundamental to effective accountability to Parliament for the decisions and actions of government. Confidence and trust in government are supported when Parliament is provided with fair and reliable performance information through a credible regime of performance reporting.

6.12 Good performance reporting includes information that tells Canadians what value they are getting for their taxes and, overall, the difference a department is making for Canadians. As part of the Estimates, over 80 performance reports are delivered to Parliament every fall. Good performance information can be useful to parliamentary committees in their work of scrutinizing key programs and services.

6.13 In telling Parliament and the public what has worked best and what has not, effective performance reporting can also be an incentive for departments to manage their activities for the results they have said they will achieve. Performance information used for external reporting is based on the same kind of information needed to manage internally for better results.

6.14 Every fall, in addition to the performance reports by individual departments, Parliament receives a companion report on government-wide performance, issued by the President of the Treasury Board. In recent years, that report focussed on the results of initiatives that involve departments as partners in contributing to a shared outcome. The 2001 report summarized Canada's overall performance against 19 societal indicators. This annual report has become an important vehicle for reporting on government-wide and horizontal results.

6.15 Interest in performance reporting goes back a long way. Since 1981, the Office of the Auditor General and the government have shown strong interest in improving departments' performance reporting to Parliament (Exhibit 6.1).

Basic framework for reporting is sound

6.16 In 1997 we examined the state of the federal government's performance reporting to Parliament (Auditor General's 1997 Report, Chapter 5, Reporting Performance in the Expenditure Management System). We found that the government had made a good start and that the basic reporting framework was sound. While several departments had improved some aspects of their performance reporting, we noted that Parliament received little information it could use to identify the value Canadians get for their taxes. Instead of reporting the results they had achieved, departments were largely describing their activities, products, and services.

6.17 We said that to improve their performance reporting, departments needed to do the following:

• focus more on the benefits they had gained for Canadians;
• report more on outcomes than on activities and outputs;
• compare their contribution to outcomes with the contribution made by other parties;
• provide more balance by including information on performance that did not meet their expectations;
• provide more information on the strengths and limitations of reported information; and
• include more information on interdepartmental and horizontal issues.

Disappointing pace of improvement

6.18 In 2000 we reported on our government-wide audit of progress made since 1997 in performance reporting (Auditor General's 2000 Report, Chapter 19, Reporting Performance to Parliament: Progress Too Slow). We noted that to improve their performance reports, departments would need to take a number of steps, including the following :

• set concrete expectations;
• use consistent terms;
• improve the reporting of accomplishments (and not report just activities and outputs);
• place performance in the context of past years (not just the latest year);
• achieve a better balance of reporting between good results and shortcomings; and
• give attention to the reliability of the data.

We said we were still disappointed by the pace at which departments were making the needed improvements to their performance reports.

Parliament's interest in improved reporting

6.19 Some parliamentarians are showing a greater interest in departmental performance reports. In 2001, the Eighth Report of the Standing Committee on Public Accounts stressed the importance of good performance reporting. The Committee also recommended that we "conduct random audits of the information contained in the performance reports of departments and agencies in order to verify, among other things, that the information contained in these reports is a fair representation of accomplishments against goals and objectives."

6.20 In response to this request, we have developed an approach for reviewing performance reports that will demonstrate how well departments are reporting. It will also provide an opportunity for departmental officials who prepare the reports to better understand the attributes of good performance reporting and how reporting can be improved. Our model for rating departmental performance reports uses five criteria or attributes of good reporting and reflects progressive stages or levels of reporting against them. The outcome of this approach should contribute to continued improvement in departments' reports.

Obstacles to good performance reporting

6.21 In our 2000 Report, we highlighted three factors that contribute to the current weak state of reporting:

• basic principles of good reporting are not understood or applied;
• performance reporting takes place in a political environment; and
• there are few incentives for good reporting and few sanctions for poor reporting.

6.22 One condition that fosters effective performance reporting is senior management that supports comparing the department's results with those in previous years and with those of similar organizations. Another condition is a political culture that supports transparency by readier acceptance of any reported shortcomings in results and of action to correct them.

6.23 Along with better conditions for reporting, there are strategies that can encourage improvement. Among them is the practice of rating performance reports and making the results public. Areas that need to improve can be identified, and exemplary reports can be recognized and applauded.

6.24 Departments' performance reports are at different stages of development, and their pace of improvement varies. While some improvements may mean a large investment of a department's resources, others can be made with current resources.

Focus of the study

6.25 This study continued our work of recent years on the federal regime of performance reporting. Our objectives were the following:

• develop a model to assess departmental performance reports;
• determine the robustness of the model; and
• determine the feasibility of using the model to provide ongoing assessments.

6.26 After developing the rating model, we tested its robustness by having different experts use it to rate four departments' performance reports. Our experts arrived at similar ratings. We also had officials from the four departments review the ratings for reasonableness. We then asked our teams who audit these departments to review the ratings. Finally, when we were satisfied that the model was robust, we used it to rate three reports, one of which was drawn from the earlier test group. These are the three sample reports discussed in this chapter that show how the model can be used. We did not audit the departments' systems and procedures for producing the information they included in the reports. Further details on our study objectives, scope, and approach are provided in About the Study at the end of the chapter.

Observations

A model for rating performance reports

Criteria for reporting

6.27 Building on our work of 1997 and 2000, we elaborated on our five criteria of good performance reporting. They should help a department produce a compelling performance story. A performance report that achieves the maximum rating by these criteria would demonstrate the attributes of exceptional public reporting:

• Organizational context and strategic outcomes are clear.
• Performance expectations are clear and concrete.
• Key results are reported against expectations.
• Reliability of performance information is supported.
• Use of performance information is demonstrated.

Each of these five criteria also includes a number of detailed subcriteria.

6.28 Taken together, the criteria represent expectations for a credible performance story. The first three reflect what has been accomplished; the other two indicate the quality and use of the performance information. The fourth criterion calls for the department to show how the reader can judge the accuracy and credibility of its performance information. However, our model is not designed to provide assurance that the information in a performance report is accurate.

Treasury Board's principles

6.29 In 2001, the Treasury Board Secretariat released guidelines for departments to use in preparing their next round of performance reports. The guidelines included six broad principles:

• provide a coherent and balanced picture of performance that is brief and to the point;
• focus on outcomes, not outputs;
• associate performance with earlier commitments and explain any changes;
• set performance in context;
• link resources to outcomes; and
• explain why the public can have confidence in the methodology and data used to substantiate performance.

6.30 The Treasury Board's principles were informed by the criteria we released in 1997 and 2000 and are consistent with our five criteria. They and our subcriteria provide a more detailed specification of what constitutes good reporting and were used in developing the model to rate performance reports. The Appendix to this chapter provides more details on the attributes of good reporting.

Levels of good performance reporting

6.31 Our model provides five levels of achievement that a good report would demonstrate (Exhibit 6.2). The fifth level represents the attributes of excellence in a performance report. By meeting each criterion of the model at progressively higher levels, performance reports will demonstrate increasing mastery of these attributes. Some of the attributes represent concrete steps a department should take in sequence to reach a higher level; others call for more of the same kind of action by the department.

6.32 The lowest level requires of the department only some basic information. This includes linking its performance expectations, expressed mostly as activities, to its mandate and mission and clearly identifying its strategic outcomes. At a higher level, we expect to see results-oriented performance expectations presented in a logical sequence, so readers can judge how much the department's activities and outputs have contributed to the outcomes. A rating model with a number of levels allows a department to identify the stage it has reached with its performance report.

6.33 The model characterizes the five levels for each of the five criteria as basic, fair, good, very good, and excellent. The model is anchored at the lower end by what our work has shown is the bare minimum for performance reporting. The upper level represents excellent reporting. We believe this level of reporting is within reach of most departments if they sustain steady progress.

6.34 We derived these levels from extensive audit work and reviews of performance reports. We also obtained expert advice on what steps departments should take at which levels. Nonetheless, we are still developing our model and expect to continue adjusting it as we use it to rate performance reports.

6.35 We have incorporated some flexibility in the model, in light of the many changes asked of departments since the Improved Reporting to Parliament project began in 1995. For example, in its 2001 guidelines for departmental performance reports, the Treasury Board Secretariat encouraged departments to place greater emphasis on reporting their results by strategic outcome rather than by business line. Because many departments are still making that transition, we designed the rating model to accommodate either way of reporting results. If a department were to report on its operations both by strategic outcome and by business line, we expect that it would show in its performance report how the two were linked.

Need for logical and consistent terms

6.36 In the many performance reports we have reviewed, we have found that the terms used by departments and by the Treasury Board Secretariat have changed over time. Some departments continue using the same terms, instead of alternatives the Secretariat has encouraged with its 2001 Results-Based Management Lexicon. For effective communication and better understanding, a common set of terms still needs to be used more consistently.

6.37 Rather than attempting to precisely define each term, we applied the model to the logical relationships underlying the different concepts. Exhibit 6.3 sets out a framework that identifies the main terms used for planning and reporting performance. These terms are consistent with the Treasury Board Secretariat's 2002 guidelines for preparing reports on plans and priorities.

6.38 The planning and reporting framework is built on the same logic that underpins a results chain. In addition to activities and their outputs, results chains traditionally use immediate outcomes, intermediate outcomes, and end outcomes. These three outcome levels should be able to capture all the key results a department reports. However, the level of detail a department chooses for reporting results depends on how specific it wants to be in stating its performance expectations. We suggest that the level of detail chosen for both the planning stage and the reporting stage be similar.

The results chain

6.39 The results chain can be a useful tool for effective and credible performance reporting. Also known as a logic model, a results chain links a department's performance expectations to its key results. Results chains are a convenient way to summarize the benefits a department provides for Canadians.

6.40 Applying the logic of the results chain to the planning stage helps a department identify how its interventions can contribute to a planned outcome. Applying this same logic to the reporting stage helps it present the evidence of its contribution to that outcome.

6.41 Using the logic of a results chain may make it easier to handle the question of attribution. Departments may be involved in activities that produce outcomes shared by partners—for example, other federal departments, other levels of government, the private sector, the non-government sector, and the local community. Shared outcomes can be associated with horizontal initiatives; some horizontal programs involve several federal departments, while others also involve other levels of government. A results chain can help a department show more precisely its own contribution to a shared outcome that includes contributions by its partners, stakeholders, and others.

6.42 Results chains can identify factors over which a department has limited control. More elaborate results chains can show the positive and negative unintended consequences of a department's activities.

Improving performance reports

6.43 Our model is designed to rate performance reports by a set of criteria. It can be used to rate previous reports and compare them with current and subsequent reports to assess a department's progress. The rating of a department's performance report gives Parliament a basis for

• comparing the report with those of other departments that have also been rated,
• asking that a department take concrete steps to improve its performance report, and
• when a department's report has been rated previously, assessing the department's progress in improving it.

6.44 When a department uses the rating model to assess its own performance report, it can locate the level of reporting it has reached. That level should suggest some next steps the department can take to help advance its performance report to a higher level.

6.45 Major steps to improve reporting that may take a significant investment of resources could include acquiring information technologies that provide information on results. Other steps can be taken with existing human and financial resources, such as using a results chain.

6.46 Offering our rating criteria now may encourage departments to carry out self-assessments using the model and improve their next reports. We designed the model to be sensitive enough to identify improvement on a number of dimensions. A review of a department's report by the Office of the Auditor General, combined with departmental self-assessment and improvement, should help departments make faster progress in improving their reports.

Applying the model

Three reports used to illustrate the ratings

6.47 We invited three departments to have their 2000-01 performance reports rated against our five criteria so we could demonstrate in this chapter how the rating model would work. The reports were provided voluntarily and do not represent any trends in performance reporting across government.

6.48 The three reports include some innovative elements and serve as benchmarks for testing the rating model. The Royal Canadian Mounted Police won the first CCAF-Treasury Board of Canada Award 2001 for the Pursuit of Excellence in Public Performance Reporting (for large departments with budgets over $0.5 billion) with its 1999-2000 Performance Report. Environment Canada has consistently produced performance reports with strong components. Fisheries and Oceans Canada has moved from reporting results by business line to reporting them by planned strategic outcome, while providing an excellent crosswalk between the two ways of reporting.

Methods of rating the reports

6.49 We do not give the reports an overall rating, nor a detailed rating by subcriterion, because our purpose here is to illustrate the kind of analysis that will support the model. Exhibit 6.4 summarizes our rating of the reports by each of the five criteria, using an average score.

6.50 There are two main ways of arriving at a rating. Each criterion has a number of subcriteria (summarized in the Appendix). Initially, reports are scored on each subcriterion against five levels of achievement. One way of rating is simply to average the subcriterion scores for each criterion. This has the advantage of using all the information on a report across each criterion to give a picture of the report's overall quality. At the same time, however, if a report rated very well on most of the subcriteria but very weak on another, the averaged rating would hide that weakness.

6.51 An alternative method would use the lowest score on any subcriterion as the rating for the overall criterion. This approach would not give credit for any progress a department had achieved on the other subcriteria. It would thus set a more demanding standard than the first method. However, if a report rated very well on most of the subcriteria but very weak on another, the second method would hide the strengths. This would lead to a lower rating for the criterion, if not for the entire report.

Assessing the reports

6.52 Overall, we found that the three reports did fairly well on the first criterion but not as well on many of the others. They were generally rated at the lower levels on the fifth criterion.

6.53 Turning to the first of our five criteria, Exhibit 6.4 shows that the three departments were good to very good at providing clear and sound information on their operating environments. They still have areas they could improve. For example, while one report provided solid information on the department's operations and challenges, it did not discuss the risks to its achieving the planned strategic outcomes. Nor did any of the reports identify all the department's key partners or their roles. As a result, readers may get the impression that these departments are solely responsible for their outcomes whether they contribute to the state of the environment, health of fish stocks, public safety, or some other broad responsibility of government.

6.54 Moving from our first criterion and its focus on the broad context of planned outcomes to the second criterion, we look at how clearly and concretely the report states the department's performance expectations. We found that they contained mostly general commitments. Two of the reports had few performance expectations that stated a direction (to increase, maintain, or decrease something), an amount (by a number or percent), and a timeframe (by when a change would be made). All three departments seemed to find it difficult to express their performance expectations clearly and concretely.

6.55 Our third criterion looks at whether departments have made any progress in reporting on their accomplishments. Two of the departments were stronger at making commitments than at demonstrating that they were delivering on their commitments. The three reports were all fairly strong in aligning results with previously stated performance expectations, and two provided the overall cost of the department's activities by planned strategic outcome. We recognize this as a promising practice by these departments (Exhibit 6.5). All three departments could do better at showing what they have accomplished with taxpayers' money.

6.56 The departments provided little help to readers to judge the accuracy of the information in the reports. As our fourth criterion indicates, a report should include support for the department's confidence in the reliability of the reported information. This would add to the credibility of its performance story.

6.57 Our fifth criterion rates how well a performance report demonstrates the department's use of its performance information. The ratings were low. We had hoped to find departments demonstrating that they used performance information to make choices in managing their programs, services, or strategies so they could deliver better results to Canadians.

6.58 Overall, our findings suggest that by using the model, we can provide parliamentarians with credible ratings of performance reports that, over a number of years, will allow them to judge the rate of improvement. Using our model to rate these three reports also suggested areas where reporting can improve.

Conclusion

6.59 As an extension of previous work, we developed a model for rating departmental performance reports that incorporated existing criteria of good reporting. We tested the model and concluded that it was sufficiently robust. We used it to rate the performance reports of three departments and demonstrated how the model can be applied.

6.60 Are assessments of performance reports feasible? We are confident that by using this rating model over a number of years, we can report to Parliament on how performance reporting is progressing. We also believe that departments can use the model to assess their own performance reports and identify ways to improve them. Improvements to the model will be made as we continue to use it, and further changes will be suggested by departments as they gain experience from applying it to their own reports.

Government's response. The government is generally supportive of this chapter, which formally introduces the initial version of the Auditor General's model for rating departmental performance reports. We believe that as the model is used and refined over time, it can become a tool to support improved reporting to Parliament and to Canadians.

The model is generally consistent with the principle-based guidance on departmental performance reporting that was released by the Treasury Board Secretariat in the summer of 2001. The Secretariat believes the Model for Rating Departmental Performance Reports could be an important learning and feedback tool that will allow departments to take stock of the information in their performance reports and take the necessary steps to improve their public reporting. We would encourage the Office of the Auditor General to continue to approach the review of departmental performance reports in a way that supports their continuous improvement.

About the Study

Objectives

The objectives of this study were to do the following:

• develop a model to assess departmental performance reports;
• determine the robustness of the model; and
• determine the feasibility of using the model to provide ongoing assessments of performance reports.

Scope

We invited three departments to use their 2000-01 departmental performance reports to demonstrate the applicability of the developmental model. The reports were provided voluntarily and do not represent a sample of the 80 or so performance reports tabled in the House of Commons. We could not therefore draw any conclusions about any trends in performance reporting across government.

Approach

Building on our work of 1997 and 2000, we set out five criteria of good performance reporting that, when taken together, we expect would produce a compelling departmental performance story:

• Organizational context and strategic outcomes are clear.
• Performance expectations are clear and concrete.
• Key results are reported against expectations.
• The reliability of performance information is supported.
• Use of performance information is demonstrated.

Each of these five criteria was shown in our model as a progression over five levels or stages of development. A picture of the ideal performance report was then developed by using the fifth level of each criterion. This ideal performance report represents excellent public reporting.

After developing the rating model, we tested its soundness by having different experts use the tool to rate four departments' performance reports. Our experts rated the reports independently and reached similar scores. We also had officials from the four departments review their respective ratings for reasonableness. We then asked our teams who audit these four departments to review the ratings. We concluded that our model was robust.

We then rated the performance reports volunteered by three departments, one of which had also been included in the test group. These are the sample reports we used to discuss how the model is used, including how it can distinguish between various levels of performance on each of the five criteria for good reporting. We did not audit the departments' systems and procedures for producing the information they included in their reports.

Our ratings, and those from the departmental self-assessments, were then discussed during a clearance session during which agreement was reached on the ratings as well as on why high levels were not achieved. We also had our teams who audit these departments review the ratings. While the final ratings are our own, they also represent a consensus between our Office and the three volunteer departments.

Study team

Assistant Auditor General: Maria Barrados
Principal: Barry Leighton
Director: Yves Genest

Denis Jobin
Catherine Livingstone
Albert Melanson

For information, please contact Communications at (613) 995-3708 or
1-888-761-5953 (toll-free).