Rating Departmental Performance Reports

Main Points

1.1 Performance reporting is more important than ever for parliamentarians and Canadians. As events unfold and debates take place on health, the environment, and security, Canadians want to know the value they are getting for their taxes and that their elected representatives can hold ministers to account for the money their departments spend. Departmental performance reports are a key tool for ensuring effective accountability.

1.2 In 2000 an audit of departmental performance reports concluded that the progress federal departments and agencies have made to improve the quality of their performance reporting to Parliament was too slow. In 2002 we presented a model to rate departmental performance reports.

1.3 In this audit, we rated nine departmental performance reports tabled in the fall of 2002 against this model. Through this exercise we identified some promising practices, as well as many challenges that need to be addressed. Overall, we found that progress in performance reporting remains too slow for these nine departments.

1.4 We found that performance reports provide a good overview of each department's organizational context and planned strategic outcomes. However, performance expectations are not always clear and concrete, information does not necessarily focus on program results, and reported results are not always credible and balanced. As well, the reports provide little evidence that performance information is used to make decisions. We also looked at one horizontal issue and found that the reports give only limited recognition to the importance of reporting on such issues.

1.5 We will continue to update and use our rating model to encourage improvement in the quality of departmental performance reports, as well as to identify promising practices and key challenges.

The government has responded. The government welcomes the efforts of the Auditor General to assess departmental performance reports. Its comments are included at the end of this chapter.

Introduction

Performance information continues to be important

1.6 The Office of the Auditor General believes that good performance reporting is fundamental to effective governance and accountability to Parliament.

1.7 Performance reporting is important for Canadians who want to know the value they are getting for their tax dollars. It enables them to understand the difference a department is making for them. Parliamentarians also need timely access to good quality performance information about government activities, so that they can scrutinize programs and services more effectively.

1.8 Every fall, over 80 federal departments and agencies submit performance reports to Parliament that outline their accomplishments to the end of the last fiscal year as established in previous reports on plans and priorities.

1.9 In 1997 we examined the state of the federal government's reporting system following the implementation of the Improved Reporting to Parliament Project that was initiated two years earlier. We found that the government had made a good start and that the basic reporting framework was sound. However, when we conducted a follow-up audit in 2000, we found that the pace at which departments were making the needed improvements to their performance reports was too slow.

1.10 In 2001 the Eighth Report of the Standing Committee on Public Accounts stressed the importance of good performance reporting. The Committee also recommended that we "conduct random audits of the information contained in the performance reports of departments and agencies in order to verify, among other things, that the information contained in these reports is a fair representation of accomplishments against goals and objectives."

1.11 In response to this request, we developed a model to rate performance reports, which was released in 2002. This model outlines the criteria necessary to achieve good reporting. We are still updating our model and will adjust it as we use it to rate performance reports so that it will continue to reflect current best practices in reporting.

1.12 The Treasury Board Secretariat stressed the importance of good performance reporting in Results for Canadians: A Management Framework for the Government of Canada (2000) which stated that "Strengthening accountability to Parliament and to citizens is an integral part of the management board's change agenda." In 2002 the President of the Treasury Board released a report entitled Canada's Performance 2002 that summarizes Canada's overall performance on 19 societal indicators. A similar report was released in 2001.

1.13 The Treasury Board Secretariat and the Office of the Auditor General plan to release the Managing for Results Self Assessment Tool in 2003. This tool will describe the transition path toward effectively managing for results. Many of the tool's concepts are similar to those of performance reporting as both approaches require credible and balanced results information.

1.14 The Treasury Board Secretariat's guidelines for preparing departmental performance reports outline performance reporting principles and provide advice about how to interpret and apply them. Even though there is no one-to-one relationship between the model and the Treasury Board Secretariat's guidelines for performance reporting, the criteria are broadly consistent with the principles in the guidelines. The model has received the general support of the Secretariat. The principles in the CCAF's (formerly the Canadian Comprehensive Auditing Foundation) 2002  Reporting Principles—Taking Public Performance Reporting to a New Level are also consistent with our criteria.

A model for rating departmental performance reports

1.15 Given the importance of performance reporting, we continue to work to improve our model for rating departmental performance reports. This model rates reports according to five criteria. A performance report that fully meets all five criteria would demonstrate the attributes of excellent reporting. The five criteria, each of which include several sub-criteria, are as follows:

• organizational context and strategic outcomes are clear,
• performance expectations are clear and concrete,
• key results are reported against expectations,
• performance information is credible and balanced, and
• use of performance information is demonstrated.

1.16 Together, the criteria represent expectations for a credible performance story about the difference a department can make for Canadians (Exhibit 1.1). The first three criteria reflect what has been accomplished; the other two indicate the quality and use of the performance information. In particular, the fourth criterion expects the department to show what it does to ensure the quality of its performance information. However, our model is not designed to provide assurance that the information in a performance report is accurate because it does not include an audit of performance information. As well, if a report omits a key piece of performance information, it would not likely be taken into account in the rating.

1.17 Our model provides five achievement levels. The fifth level represents the attributes of excellence in performance reporting. By meeting each criterion of the model at progressively higher levels, performance reports demonstrate that departments have increasingly mastered these attributes.

Focus of the audit

1.18 The objective of this audit is to rate several departmental performance reports by

• determining the overall quality of these departmental performance reports, which provide Parliament and Canadians with information about departmental performance; and
• determining if horizontal issues (paragraph 1.57) are properly identified in these performance reports, using a shared outcome as an example.

1.19 We selected the performance reports of the majority of departments and agencies that have mandates that relate to the security of Canadians. These are

• Citizenship and Immigration Canada,
• Correctional Service Canada,
• the Department of Justice Canada,
• the Immigration and Refugee Board,
• National Defence,
• the National Parole Board,
• the Royal Canadian Mounted Police (RCMP),
• the Solicitor General Canada, and
• Transport Canada.

These organizations represent $19.3 billion in expenditures, or 12 percent of the government's total expenditures for fiscal year 2001-02.

1.20 More details about this audit's objectives, scope and approach are provided in About the Audit at the end of this chapter.

Observations and Recommendations

Much remains to be done

1.21 Our audit found that the reports are not always as clear and useful as they need to be (Exhibit 1.2). Overall, the reports achieved a fairly good rating on the first criterion. They almost reached, or reached, a fair rating on the second, third and fourth criteria. However, they barely exceeded the basic level on the fifth criterion. These results show that the performance reports we rated have a long way to go before they reach the highest levels of the model. Several previous audits also expressed significant concern about the overall quality of reporting. Yet, based on our rating of these nine reports, this audit shows that performance reporting continues to face many challenges. Nonetheless, we have identified some promising practices in the nine reports that we rated, which illustrate how performance reports can be improved. These practices show how it is possible to improve reporting to Parliament.

Organizational context and strategic outcomes

1.22 The first criterion requires that a department's organizational context and strategic outcomes are clear. It assesses whether the report clearly states a department's business lines and whether these are reflected in its mandate and mission. It also determines if there is a logical sequence and link between its business lines and its key results, which include its planned strategic outcomes. These factors should be aligned with those that influence an organization's operating environment. This criterion also asks that key external partners be identified and that a credible description of the risks the department faces be provided.

1.23 The first criterion assumes that public sector organizations do not exist in a vacuum. That is, they are subject to many internal and external opportunities and pressures, and they have to solve increasingly complex issues in order to fulfil their mandate. Therefore, a department's performance story is only meaningful when it is placed in this broader context.

The reports generally provided a good overview of the organizational context and strategic outcomes

1.24 All but one of the reports achieved a rating of fair to very good on the first criterion (Exhibit 1.3). This is consistent with what we found in our 2002 Report when we demonstrated how our model could be used.

Promising practices in reporting organizational context and strategic outcomes

1.25 The reports provide good overviews of their planned strategic outcomes. Transport Canada, National Defence, and Citizenship and Immigration Canada provide Web links in their reports so that readers can get more background information. By providing more comprehensive information about the department and its organizational context through these Web links, this approach helps to keep the reports comparatively short and focussed on performance.

1.26 The reports also provide good information about how the department's business lines contribute to planned strategic outcomes, and how they are aligned with their missions and mandates, as well as federal government priorities. Transport Canada and the Solicitor General Canada did well in this area. Transport Canada's report includes a graphic that clearly shows how it links its strategic objectives with its business lines, resources, and broad government themes. It also shows how the department's activities and outputs contribute to its strategic objectives.

Challenges to reporting on organizational context and strategic outcomes

1.27 Although the reports did relatively well with respect to the first criterion, there is still room for improvement. An important aspect of organizational context is the key partnerships through which departments deliver some of their programs. In most cases, only a list of partners is provided. There is no information about which department has the lead, or how the activities and outputs contributed to a shared outcome. We address this issue again when we discuss the reporting of horizontal issues.

Performance expectations

1.28 According to the second criterion, performance expectations should be clear and concrete. That is, departments should clearly state what they will do for Canadians. This criterion focusses on performance expectations and on how they relate to the organization's strategic and operational plans. Performance expectations should be clearly identified and aligned with federal government priorities. They should also be expressed as outputs, or outcomes at higher levels in the model. They should include the direction of planned changes and the timeframe in which those changes will be made. The report should state, for example, how much outcomes will be increased, maintained, or decreased as a result of the department's programs.

1.29 The second criterion also captures how the organization plans to achieve its performance goals. While most of this information is included in the corresponding report on plans and priorities, the performance report should contain enough information to enable readers to understand how the department's strategies and activities will achieve the expected results.

Performance expectations are not always clear and concrete

1.30 With the exception of two departments, the reports did not rate well on the second criterion. Overall, we found that performance expectations are not clear and concrete, and that they are not aligned with the organization's strategic and operational plans (Exhibit 1.4).

Promising practices in reporting performance expectations

1.31 The RCMP report provides an executive summary chart that assesses its performance against the problem being addressed. However, because the chart does not include the organization's specific performance expectations, it is not as informative as it could be.

1.32 Throughout the Citizenship and Immigration Canada report, clear and concrete statements are made about how the department's performance expectations could affect Canadians. For example, it describes how achieving certain immigration levels will offer economic and social benefits for Canada. It explains how immigrants play an important role economically by building human capital and filling labour market gaps. According to the report, Canadian society also benefits from the cultures, traditions, and knowledge that immigrants offer, and from the strong communities and social stability that reunited families provide. The report states that attaining this performance expectation is to be demonstrated by "achieving planned emigration levels of 200,000 to 225,000 newcomers to Canada for 2001." This performance expectation states clearly the direction, amount, and timeframe of the planned change.

1.33 The Immigration and Refugee Board performance report discusses strategies for achieving planned levels of activities and outputs and how to deal with high volumes of outstanding immigration claims. For example, the report states that resources will be used to develop streamlined processes to better manage the record number of outstanding refugee claims.

Challenges to reporting performance expectations

1.34 Reports on plans and priorities are intended to establish performance expectations and outline the general direction a minister wants a department to take during the Estimates year and the two fiscal years that follow. Because performance reports are designed to provide information on results that were achieved, they are compared with the relevant reports on plans and priorities. Reports on plans and priorities set performance targets while performance reports state the extent to which they were met, and provide explanations for the level of performance achieved.

1.35 We found that, in many cases, performance reports were not consistent with departments' reports on plans and priorities. In fact, in seven of the nine reports that we rated, the performance expectations in the planning reports were, to varying degrees, different from those in the performance reports. In some cases, performance expectations were added, consolidated, modified, or deleted, and the differences were not explained. As a result, accountability for previous commitments is not clear. For example, the Solicitor General Canada's report is inconsistent with the Department's report on plans and priorities.

1.36 We also found cases where targets are vague. For example, the RCMP report states that a reduction in organized crime would be demonstrated by long-term investigations that indicate results that are superior to those in the previous year. Another practice is to provide targets that are activity-based rather than ones that focus on outcomes. For example, the Department of Justice report states that a fairer, more effective youth justice system will be measured by the youth justice legislation.

Performance outcomes

1.37 The third criterion requires that key results be reported against performance expectations and that departmental contributions be addressed. It focusses on results that are expressed as outputs, or outcomes, at the higher levels in the model, and whether or not the department achieved what it said it would achieve. The elements of this criterion include aligning outputs and outcomes with performance expectations, addressing the challenges of achieving expected results, identifying who contributed to the performance outcomes, determining the level of resources required, and explaining how results are interpreted.

1.38 This complex criterion is at the heart of good performance reporting. Unlike the first two criteria, which set the stage for performance reporting, this criterion addresses the key attributes of the performance information that is reported.

Outcomes are not widely reported

1.39 The ratings for the third criterion ranged from basic to good. In particular, the amount of results information reported by departments varies a great deal. Overall, we consider the results to be very mixed (Exhibit 1.5). Five reports provide information on the level of resources needed to achieve the strategic outcomes but none had information on resources for key outputs. As well, they pay little attention to who is responsible for achieving these outcomes.

Promising practices in reporting outcomes

1.40 The Immigration and Refugee Board report provides a lot of detailed information about its operational outputs. For example, the Board's report provides data about several referred and finalized refugee claims, specifically, about the average processing time and the cost of processing each claim. Despite the fact that it does not report performance expectations, Correctional Service Canada does report some quantified information for a number of key results such as re-offending rates during supervision.

1.41 Many of the reports provide Web links to their evaluations and internal audits. For example, the RCMP report includes a section that conveniently summarizes its evaluations and internal audits. While the Citizenship and Immigration Canada report indicates that only one evaluation report was completed during the 2001-02 fiscal year, it presents a balanced account of its key conclusions. The Solicitor General Canada report often refers to evaluation reports that are related to several strategic outcomes, but it does not provide information on their content.

1.42 Identifying the level of resources allocated to achieve specific results is difficult. However, most of the reports managed to do so at the broad strategic outcome level. The Correctional Service Canada report provides a promising example of how to communicate this type of information. However, like other departments, it does not identify the resources allocated to achieve its key outputs and immediate outcomes. All the reports provide resources that are allocated according to business lines. Only one report identifies the financial contribution of each business line to each strategic outcome. It does this by equating its business lines to its strategic outcomes. For the most part, the reports do not provide financial information at the next level down for strategic outcomes or business lines namely, for major programs.

Challenges to reporting outcomes

1.43 We observed that some information about what activities were undertaken to produce the outputs or contribute to the outcomes is presented as the actual measures of outputs or outcomes. For example, the Citizenship and Immigration Canada report presents descriptions of recent legislative changes as measures of its performance. Other reports also present activities such as meetings, organized events and completed studies as results. While useful and relevant, any information on activities should also be accompanied by explanations about how they contribute to the results that are achieved for Canadians.

1.44 As we indicated earlier, providing highlights of key findings from relevant evaluations and audits helps to explain the department's performance and the challenges it faces. We noted that Transport Canada's report does not provide this type of information, although such evaluations and audits were in fact conducted. A number of these evaluations and audits address issues that are of significant interest to Canadians. It would have been helpful if the Department had informed Parliament about its efforts to address these issues.

1.45 As previously noted, the Immigration and Refugee Board report provides a lot of information about its outputs. However, data on the numbers and proportions of claims accepted, rejected, withdrawn or abandoned through the refugee protection process are not provided, although they would be of interest to parliamentarians. Information on the numbers and proportions of appeals allowed and dismissed through the immigration appeal process would also be helpful. We also observed that the Board provides little information on its outcomes.

Credible and balanced results

1.46 The fourth criterion states that performance information should be credible and balanced. First, it focusses on the quality of performance information and the reliability of information sources as the basis for judging the credibility of the data. Second, it focusses on whether there is balanced reporting of good results and shortcomings, and whether the level of detail for key results is appropriate. This criterion is fundamental to good performance reporting. If all the elements of good reporting are in place, but the information provided is not credible or balanced, then the report will not be useful for parliamentarians or Canadians.

Departments do not generally report balanced results

1.47 The ratings for this criterion were between basic and fair. Overall, the reports did not achieve a good rating (Exhibit 1.6).

Promising practices in reporting credible and balanced results

1.48 The National Parole Board report explains the limitations of the data on post-warrant expiry recidivism. It mentions that the number of re-offences measured by this indicator is probably much larger in reality because of certain limitations of the data. The Immigration and Refugee Board reports the number of complaints filed against members of the Board. Also, sections of the RCMP report provide endnotes that contain information about the external data sources.

Challenges to reporting credible and balanced results

1.49 We noted that, overall, the reports tend to present only positive performance information rather than both successes and shortcomings. This tendency to focus only on "good news" produces reports that convey a one-sided view of departments' performance. Consequently, a lack of balanced reporting means that potential users, such as parliamentarians, may be tempted to discount them as biased, incomplete or lacking credibility.

1.50 We also identified several examples of data that were of limited quality. The reports provide little information about how departments manage data quality, what the limitations of the data are, and what they intend to do to make improvements.

Use of performance information

1.51 Managing well to obtain improved results is a challenge. As noted in our December 2000 Report, Chapter 20, Managing Departments for Results and Managing Horizontal Issues for Results, there is a need for more widespread measurement of the results departments accomplish and the use of that information to improve programs. The previous four criteria are based on a more traditional approach to reporting performance, where the organizational context must be provided, the performance defined, measured, and reported against expectations, and the information must be of the highest quality. However, even if all of these conditions have been met, if feedback on past performance is not used to improve future results, then it may only be a reporting exercise for external accountability. A performance report is also an invaluable tool for departments to demonstrate that they have learned from their shortcomings and challenges, and that they are using this information to improve their future performance. This should also encourage balanced reporting.

1.52 The fifth criterion expects that performance reports will show the ways in which performance information is used, including how it will be used to establish future performance expectations. It assumes that performance information is not simply collected to create reports. Rather, it will be used to help departments make strategic decisions, highlight lessons learned, and identify how weak performance will be corrected. The performance information should also provide a credible discussion about the capacity of a department to produce sustainable results by continuing to perform well in the future.

The use of performance information is not demonstrated

1.53 Most of the reports we rated only achieved a basic level on the fifth criterion (Exhibit 1.7). When departments produce key results that are of a reasonable quality, they should be able to demonstrate how this information is used to help make decisions about how to improve future performance.

Promising practices in reporting the use of performance information

1.54 The National Parole Board report states that it has had difficulty with managing pardon workloads and that this has an impact on processing times. In response to the growing backlog of applications for pardons, the Board lists the activities it has undertaken to improve its performance in this area.

1.55 The Department of Justice report provides a short overview of lessons learned and the challenges it is facing for each of its business lines. Although it might be a promising practice that other departments could adopt, the relevance of the lessons learned is not always clear.

Challenges to reporting the use of performance information

1.56 None of the reports we rated discuss the capability of departments and agencies to continue to meet performance targets in the future. The ratings suggest that this could be one of the most challenging aspects of performance reporting. Facing this challenge will require that good information be incorporated into the decision-making process and that the information be used to correct problems and to evaluate whether or not the department is heading in the right direction.

Horizontal issues

The challenge of reporting horizontal issues

1.57 Horizontal issues are programs and activities that span the mandates of several departments and agencies and that receive a contribution from each. Reporting on these issues to Parliament presents particular challenges for the government as a whole, as well as for each of the respective departments. Departments have to determine at which point their contribution to a shared outcome requires that it be reported to Parliament. The government has recently used Canada's Performance 2002 to report on the results of some horizontal areas.

1.58 We expected that performance reports would provide sufficient and credible information to Parliament about the department's performance for significant horizontal issues. We also expected that departments participating in a horizontal activity would

• describe their role in the issue and their contribution to the shared outcome,
• identify the lead department, where appropriate, and
• direct readers to the lead department's performance report.

Reporting on the safety and security of Canadians is incomplete

1.59 As the government has stated on a number of occasions, the events of September 11, 2001 demonstrate the need for an extensive, government-wide response to terrorism. Parliamentarians and Canadians need to know how departments and agencies are working together in a horizontal manner to help protect Canadians from organized criminal and terrorist threats. They also need to know the value they are getting for government expenditures, such as the additional $7.7 billion allocated in the December 2001 federal Budget to this issue over the next five years.

1.60 In order to measure the results of departmental contributions to this particular horizontal area, it would be useful to clearly define what these activities are supposed to achieve. Current descriptions of activities in this area at the government-wide level are sketchy. The Government of Canada's Web site states that many government departments and agencies work together to keep Canada safe but it does not identify these organizations. Information is also lacking about departmental responsibilities related to organized crime—a key issue discussed in the 2001 Speech from the Throne.

1.61 Government-wide reporting on the results of these activities is being developed. In Canada's Performance 2002, the President of the Treasury Board identified "security from organized criminal and terrorist threats" as one horizontal area of government activity. The report refers the reader to the performance reports of the relevant departments. The Treasury Board Secretariat Web site identifies the departments that contribute to helping protect Canadians from organized criminal and terrorist threats, and provides Web links to their performance reports.

1.62 Since government-wide documents provide only general references to horizontal issues, we expected to find more information on the participants and the results of their activities in this horizontal area in the departmental performance reports. Our rating model identifies two opportunities for departments to address horizontal issues in their performance reports: identify their role in the initiative and identify their contribution to the shared outcome.

1.63 Eight of the nine reports reviewed list some departments' key partners for their activities that address organized criminal and terrorist threats. However, the details vary significantly. Transport Canada simply lists all of its partners, without identifying the horizontal issue. Only three reports identify which department leads a particular horizontal issue. The Department of Justice provides information on horizontal, government-wide issues in a table that names the issue, the lead department, its partners, and the department's activities.

1.64 Although we did not rate the Privy Council Office's performance report, we noted that it addresses issues related to the security of Canadians. It states that, as part of the aftermath of September 11, 2001, it "led and co-ordinated the interdepartmental response to the crisis through the creation of formal co-ordination mechanisms and enhanced direct contact between personnel in different federal departments and agencies." This report does not mention which departments were involved in the horizontal area or what their responsibilities were.

1.65 None of the performance reports indicate the contributions their partners have on the shared outcome. Results of activities designed to help protect Canadians from organized criminal and terrorist threats are not reported. Only the Solicitor General Canada refers to an evaluation of the Integrated Proceeds of Crime Initiative, but the report does not provide the actual results of its contribution to the shared outcome of that initiative. A few departments report on some of the challenges that they face in working with other departments to produce shared results.

1.66 We noted that the Treasury Board Secretariat, in its Departmental Performance Reports 2002 Preparation Guide, requires departments to reflect the contribution their partners make to help them achieve their strategic outcomes. However, this requirement does not really go beyond asking them to identify their partners. This means that overall, there is little collective or systematic reporting to Parliament on the various initiatives aimed at increasing Canadians' security and safety or on who is accountable for these efforts and their results.

1.67 In our December 2000 Report, Chapter 20, Managing Departments for Results and Managing Horizontal Issues for Results, we recommended that the Treasury Board Secretariat should play a stronger leadership role in ensuring that horizontal issues are reported. We continue to support this view.

1.68 Recommendation. The Treasury Board Secretariat should strengthen its guidance and actively encourage departments to ensure that their performance reports

• identify significant horizontal areas,
• identify the lead department, where appropriate,
• identify the expected outcome for each area, and
• indicate how the department's activities and outputs contribute to the horizontal area's shared outcomes.

Treasury Board Secretariat's response: The Treasury Board Secretariat will continue to encourage departments to report on horizontal initiatives. We will be strengthening our guidance on the preparation of departmental performance reports with respect to programs and initiatives that cut across departmental boundaries.

Conclusion

1.69 This is the first time that we have used our model more extensively to rate performance reports. While we have identified some promising practices and challenges faced by the nine departments, overall, we found that their progress in improving performance reporting to Parliament remains too slow. Our ratings also showed that the importance of reporting on horizontal issues is not well recognized.

1.70 These findings are not new. In fact, they continue a trend of audit observations. In 1988 we concluded that reports to Parliament did not provide a fully satisfactory basis for accountability. In 1992 we again indicated that these reports did not provide the necessary breadth of information. In 1997 when the government reviewed its Expenditure Management System, we stated that progress in the area of performance reporting to Parliament was insufficient. In our 2000 Report, Chapter 19, Reporting Performance to Parliament: Progress Too Slow, we concluded that, "At the present pace it would take too many years for good performance to become routine."

1.71 In our 2000 report, we listed factors that contribute to the current state of reporting. These are as follows:

• frequently, the basic principles of good reporting are not understood or applied;
• performance reporting takes place in a political environment; and
• there are no incentives available for good reporting practices, or sanctions applied for those that are bad.

1.72 Various players can contribute to overcoming these obstacles. Of course, the departments should continue to work on improving their performance reports. The Treasury Board Secretariat should also continue to help them by challenging departmental performance reports and providing leadership and opportunities to increase their understanding and acceptance of the principles and practices of good performance reporting.

1.73 Parliamentary reviews of performance reports would also encourage better reporting. Parliamentarians, through various standing committees, could demand clarification and explanations of performance expectations and information. They could also challenge departments and agencies.

1.74 Our Office will continue to update and use the rating model as one tool to help encourage the improvement of performance reports to Parliament. By recognizing promising practices and identifying areas that should be improved, we hope to contribute to ensuring better reporting to Parliament.

Government's response: The government remains strongly committed to ongoing improvements to reporting to Parliament. In support of this, we welcome the efforts of the Auditor General to assess departmental performance reports in a systematic manner, identify strengths and weaknesses, and provide advice on how improvements can be made. Departmental performance reports reflect the responsibility of each minister and his or her department to account to Parliament for their performance against planned outcomes. We believe the findings, including the identification of promising practices, will help guide departments and agencies as they continue to improve their reporting to Parliament.

As indicated in the 2003 Budget, in order to reinforce accountability and transparency in public reporting, the government will continue to improve the relevance, timeliness and clarity of the information it provides to Parliament. This includes making greater use of electronic reporting on expenditures and the results achieved by government programs and activities. The government will continue to work with parliamentarians, parliamentary committees and the Auditor General to identify opportunities to improve reporting to Parliament. We will draw on the results of these discussions and the findings of this chapter, and on other sources, as we renew our guidance to departments on performance reporting.

In particular, the recommendation that the Treasury Board Secretariat actively encourage departments to provide more fulsome reporting on significant horizontal initiatives, the Secretariat will continue to strengthen its guidance over the coming reporting period to underscore the importance of reporting on these horizontal initiatives.

About the Audit

Objective

The objective of this audit is to rate departmental performance reporting by

• determining the overall quality of departmental performance reports, and providing Parliament and Canadians with information about departmental performance; and
• determining if horizontal issues are properly identified in performance reports, using a shared outcome as an example.

Scope

We selected the performance reports of departments and agencies that have mandates that contribute to the horizontal area of helping protect Canadians from organized criminal and terrorist threats. This horizontal issue is one of those identified in the communities theme in the President of the Treasury Board's Canada's Performance 2002 report.

We excluded the Canadian Security Intelligence Service and the Office of the Communications Security Establishment Commissioner because they are not required to prepare performance reports. The Customs program of the Canada Customs and Revenue Agency is also excluded, because although the Agency prepares a departmental performance report, it is one of the three agencies that report to Parliament through an annual report that this Office assesses separately.

Grouping similar organizations together enables us to compare common issues and challenges. Although our model is designed primarily to assess performance reporting at the department level, it also takes horizontal reporting into account.

Approach

In this audit, we assessed nine performance reports against our rating model.

These reports covered the period ending 31 March 2002. They were tabled in the House of Commons on 7 November 2002.

In the rating model, each of the five criteria is represented by a continuum that has five levels or stages of development. These are basic, fair, good, very good and excellent. An exceptional performance report would achieve an excellent rating in each criterion.

The approach developed in our April 2002 Report, Chapter 6, A Model for Rating Departmental Performance Reports requires raters to validate each other's ratings. With this in mind, our audit team used the model to rate the reports, the results of which were confirmed by experts and the Office audit team for each department and agency we examined. We did not audit each department's systems and procedures for producing the information included in their reports.

Audit team

Assistant Auditor General: Maria Barrados

Principal: Barry Leighton
Director: Yves Genest

Doreen Deveen
Denis Jobin
Catherine Livingstone
Albert Melanson
Paul Pilon

For information, please contact Communications at (613) 995-3708 or
1-888-761-5953 (toll-free).

Definitions:

An outcome is an external consequence that can be attributed to an organization, program, etc. and that is considered to be significant in relation to its commitments. Outcomes may be described as immediate, intermediate, final/end, direct or indirect, intended or unintended. (Back)

An output is a direct product or service that is produced through program activities and delivered to a target group or population. (Back)
(Treasury Board Secretariat)

An activity is an operation or work process that is conducted by an organization and that is intended to produce specific outputs, for example, products or services. An activity is the primary link in the chain through which results are achieved. (Back)
(Treasury Board Secretariat)