8.8 As the exhibit indicates, a basic function of the Department is to assist and support the Minister of Finance in his role as the key policy minister for economic and fiscal matters. The Department provides this assistance and support in the context of the Minister's Cabinet responsibilities and accountability to Parliament.
8.9 The Department plays a central, strategic role within the federal government. Its functions include preparing the Budget and providing economic policy advice to the government. The Department exercises central agency functions in the oversight and co-ordination of those activities of departments and agencies that affect the economy. In addition to its policy advisory and central agency roles, the Department has a number of direct responsibilities related to policies and programs. These responsibilities carry the concomitant obligation to account to Parliament for results achieved relative to established objectives. The Department's planned expenditures of $66 billion account for 43 percent of planned federal expenditures of $152 billion for 1997-98.
8.11 Exhibit 8.2 outlines areas where the Department has direct policy and program-related responsibilities. It also identifies cases where those responsibilities are shared with other departments. The policy and program areas listed in Exhibit 8.2 cover a wide range of activities and account for a very substantial level of expenditure, most of it statutory. Statutory expenditures are authorized by the Act of Parliament that establishes the program, but are not subject to annual scrutiny and review as part of the parliamentary supply process.
8.12 The most substantial of the expenditure programs relate to the management of the public debt, with an estimated budget of $46.0 billion for 1997-98, and the Federal-Provincial Transfer Payments program, with an estimated budget of $19.4 billion and a further $12.6 billion in tax transfers. Transfers under various statutory authorities include equalization payments, the Canada Health and Social Transfer and a variety of other transfers to provincial and territorial governments.
8.13 Among the Department's other policy and program-related responsibilities are those related to the regulatory framework for the Canadian financial services sector, and representation of Canada's interests in various international organizations, including the International Monetary Fund, the World Bank and the European Bank for Reconstruction and Development. In addition, the Department plays a central role in the development, management and implementation of Canadian international trade and investment policy. Among other things, the Department leads or directs the government on import policy, in particular on tariff and trade remedies, and is responsible for related domestic legislation and regulations.
8.14 The Department is responsible for the development and evaluation of federal taxation policies and legislation, including various preferential tax provisions. Many of these provisions reduce corporate and personal tax liabilities. By providing assistance or incentives to targeted groups of individuals and businesses, the provisions are intended to lead to the achievement of specific objectives (including goals in areas such as social, cultural, economic and regional policies). These so-called "tax expenditures" include provisions such as tax exemptions, deductions, deferrals and tax credits.
8.15
Finally, the Department has certain
responsibilities under the Canada Pension Plan (CPP).
The CPP is a joint responsibility of the federal and
provincial governments and is funded by its own direct
premium revenue. Changes to benefits, to the rate of
contributions, or to investment provisions require an Act of
Parliament and supporting orders-in-
council from at least two-thirds of the included provinces,
representing at least two-thirds of the population of all
included provinces. The Department is responsible for
leading negotiations with the provinces, and all other
aspects of the statutory reviews that result in
recommendations for changes to the CPP. In the past, the
Department has also been responsible for the
administration of the CPP Investment Fund. As of April
1998, the CPP Investment Board is responsible for
investing new CPP funds in a diversified portfolio of
securities, although the Department remains responsible
for the administration of the existing portfolio of loans to
the provinces and (with provinces) for the legislative
framework for CPP investment policy.
8.16 In view of the extent and consequences of the Department's policy and program responsibilities (involving a variety of instruments such as transfers, legislation, regulation and selective tax measures), effectiveness measurement and reporting are crucial. Information on results is needed to account to Parliament and Canadians for what has been achieved relative to established objectives, as well as to help improve future performance.
8.18 The government has embedded in policy the need for information on results to support decisions, improve the way government works and provide accountability. The 1994 Treasury Board Manual on Review, Internal Audit and Evaluation (Review Policy) seeks to ensure that the government has timely, relevant and evidence-based information on the performance of its policies, programs and operations, and that it uses this information to improve their management and cost-effectiveness as well as to account for results.
8.19 Under the Review Policy, deputy heads of departments are accountable for, among other things, ensuring that departmental programs and policies are functioning as intended, that appropriate review mechanisms are being used, and that review information is used in decision making and accountability reporting.
8.20 The improvement of information on performance to aid decision making and facilitate accountability is a key feature of the government's revised expenditure management systems. As well, in his 1996 Annual Report to Parliament (Getting Government Right: Improving Results Measurement and Accountability), the President of the Treasury Board strongly emphasized the importance of evaluating the actions of government and reporting publicly on the results (see Exhibit 8.3 ).
8.22 Public reporting of effectiveness information also serves Parliament in the exercise of its scrutiny and legislative functions. The government develops initiatives and undertakes expenditures subject to the approval of Parliament. Public accounting for the results achieved with the expenditures and authorities approved is a vital step in parliamentary control. This requires making the objectives of policies and programs, and the results achieved relative to those objectives, known to Parliament and to the public in a timely and meaningful manner. The information needs to be clear, concrete and understandable, and accessible to all parliamentarians. Although individual accountability documents cannot always include all the information that might be appropriate, they can provide clear references to other sources.
8.23 Parliament has maintained its interest in results information. As we noted in Chapter 3 of our 1996 Report, the Standing Committee on Public Accounts reaffirmed its interest in program evaluation in its November 1994 Sixth Report. Further, a recent report of the Standing Committee on Procedure and House Affairs (The Business of Supply: Completing the Circle on Control, April 1997) emphasized that effectiveness information is central to many of the key public debates of the day ( Exhibit 8.3 ).
8.26 In 1993, at the request of the Standing Committee on Public Accounts, we carried out a special audit of changes that had been introduced in the organizational approach to effectiveness measurement in the Department's Tax Policy Branch (The Revised Approach to the Evaluation of Tax Measures at the Department of Finance, March 1993). Our 1993 audit concluded that evaluations produced by the Branch under the new organizational approach met established measurement and analysis standards for the evaluation questions they addressed, whereas evaluation studies under the old organization did not consistently do so. We also found that important questions were addressed in the studies produced under the new approach.
8.27 We raised concerns, however, regarding the approach of the Tax Policy Branch to establishing priorities for effectiveness measurement. We also noted that fewer resources were being dedicated to evaluations and urged that an adequate effectiveness measurement capacity be maintained within the Branch. Our findings in 1993 were tentative because the organizational changes were recent. The present audit was therefore designed to follow up on the observations we reported in 1993.
8.28 Further details about the approach and scope are presented at the end of the chapter in About the Audit .
8.31 We found that the Department's statements of its own responsibilities were not clear in two of the areas we examined. In addition, where responsibilities are shared with other departments and agencies, we found a general lack of clarity about the organizations involved and the manner in which responsibilities are shared among them.
8.32 The Department shares responsibilities with other departments and agencies in many of the policy and program areas we included within the scope of this audit. In none of these areas, however, do the documents we reviewed consistently identify the other organizations involved or how their responsibilities relate to those of the Department of Finance. As a result, the accountability frameworks for these policy and program areas are not always clearly communicated to Parliament.
8.33 The failure to articulate clearly the accountability frameworks for the policies and programs we reviewed was particularly evident in Estimates documents. For example, although the Department shares some responsibilities with Foreign Affairs and International Trade Canada and Revenue Canada in the development, management and implementation of Canadian trade and investment policy, its 1997-98 Estimates documents make no mention of this. We recognize that there are limitations on the amount of information that can be provided in any one document. However, we believe that, as a minimum, identification of the departments and agencies involved, with cross-references to other sources of related information, is essential to provide Parliament with a clear perspective on accountability.
8.34 One of the two cases where we noted that statements of the Department's own role were not clear relates to Finance's responsibilities for the Canada Pension Plan (CPP). We found that the Estimates documents provide only limited information on the Department's responsibilities. In addition, they do not make any reference to the fact that information on the nature and extent of the Department's responsibilities can be found in the CPP annual report.
8.35 The other case where we found that Parliament did not receive clear information about the Department's responsibilities relates to the regulatory framework for the financial services sector. The 1997-98 Estimates documents indicate that the Department is responsible for "the functioning of and regulatory framework for Canadian financial markets and institutions." Although the Department acknowledges its responsibility for policy development related to the functioning of Canadian financial institutions, as well as for the regulatory framework that applies to them, it does not in fact have responsibility for the broad functioning of financial markets. The Department told us that it intends to clarify the statement of its responsibility in this regard in future Estimates documents.
8.36 The Department of Finance should ensure that the information it provides regularly to Parliament clearly and consistently identifies its own responsibilities, the departments and agencies with which it shares responsibilities and the manner in which the responsibilities are shared.
Department's response: The Department already does this. The 1998-99 Main Estimates provide more substantial information on the CPP as suggested. In addition, the audit suggests that the role of the Department in CPP is not well known. However, the Department's role has been made clear in documents, speeches and even a special web site. There were statements in every Budget presented to Parliament starting in 1994; a special CPP web site was created (winter 96); a Report on Consultations was published (June 96); a federal/provincial agreement was reached on measures to make the CPP sustainable, followed by a speech in Parliament and the tabling of draft legislation (February 97); and finally, legislation was introduced, a new actuarial report was tabled in Parliament, and a press release was issued summarizing the bill (September 97). Further, the recently concluded federal/provincial review included a number of measures to significantly improve information reporting and stewardship for the CPP.
8.38 It is important here to distinguish between operational objectives and policy or program objectives. The Department's operational objectives may vary from year to year - for example, in dealing with specific intentions or targets in planning, consultation or negotiation related to some aspect of policy development or program implementation. Although such information is useful in identifying operational activities to be performed and milestones to be reached, it is the policy or program objectives (the things to be accomplished as a result of the policy or program) that are central to establishing the basis for accountability to Parliament. It is in relation to these objectives that Parliament needs information on results.
8.39 We examined a range of public documents to determine whether they provide Parliament with good information on policy and program objectives. It is important for objectives to be articulated not only clearly, but also consistently across information sources.
8.40 We found that objectives were both clear and consistently stated for the Public Debt program, the Federal-Provincial Transfer Payments program and the International Financial Organizations program.
8.41 With respect to international trade and investment policy, we examined the Department's Estimates documents, the Departmental Outlook, the Federal Regulatory Plan, and the publication titled Finance Canada - Structure and Role. With the exception of some recent Estimates documents, which made limited reference to "improving Canadian competitiveness" through a variety of specific policy reform and consolidation measures, none of these documents included any statements of objectives for the international trade and investment policy responsibilities of the Department.
8.42 We noted that the Estimates documents contain neither information on program objectives for the Canada Pension Plan nor directions to other sources where such information is available. The Estimates documents also do not provide statements of objectives for the regulatory framework for the financial services sector. In addition, we found that two public documents issued by the Department (Enhancing the Safety and Soundness of the Canadian Financial System, February 1995; and 1997 Review of Financial Sector Legislation: Proposals for Changes, June 1996) do not provide clear statements of the objectives of the regulatory framework.
8.43 The Department provides Parliament with a special annual report on tax expenditures. Our findings on that report are presented in paragraphs 8.62-8.64.
8.44 The Department of Finance should provide Parliament with clear and consistent statements of objectives for the full range of its policy and program-related responsibilities.
Department's response: The Department already does this. However, it agrees that, wherever feasible, it will examine its procedures and provide more public information on the broad policy objectives and the extent to which they are being met.
8.46 We found significant gaps in the public reporting of results relative to objectives in all the areas included within the scope of our audit. In most cases, reporting focussed on specific areas of operational activity or progress, but did not provide information on the achievement of objectives.
8.47 In the Department's 1997 Performance Report, the information provided for the Public Debt program went furthest in reporting on program effectiveness. In this case too, however, some additional information would have strengthened the report. For example, the report describes indicators of the efficiency of Canada's financial markets and states that Canada's performance in terms of those indicators compares favourably to that of other countries with highly developed capital markets. We believe the report would have been more useful had it included the actual measurements - including comparisons with other countries, and changes in those measures over time.
8.48 The Department told us that it did not provide detail in the Performance Report because it is a summary overview of performance. It noted, however, that more detail had been provided in recent Debt Operations Reports. In these circumstances, a reference to the additional detail in Debt Operations Reports would be essential to assist readers of the Performance Report.
8.49 Where programs are subject to legislative review, we found that some information on results is generated in that context. However, based on an examination of three reviews, we have concluded that legislative reviews do not eliminate the need for the Department to undertake systematic and disciplined measurement and reporting of effectiveness. Indeed, we believe the timely reporting of effectiveness information would help inform public and parliamentary debate and enhance the legislative review process (paragraphs 8.76-8.96).
8.50 The findings of our current audit about deficiencies in effectiveness measurement and reporting are consistent with the findings of a number of audits undertaken by this Office from 1992 to 1997. The policy and program areas examined in these earlier audits cut across a wide range of the Department's responsibilities.
8.51 Summary findings relating to effectiveness measurement and reporting from these earlier audits are outlined in Exhibit 8.4 . At the time the audits were undertaken, there was only one case - the management of the public debt - where we were able to report that an evaluation was under way.
8.52 The situation regarding the effectiveness measurement of tax expenditure provisions is more positive. Although a number of the audits outlined in Exhibit 8.4 identified shortcomings in measuring the effectiveness of tax expenditures, during the current audit the Department identified a range of initiatives it believes to be related, in whole or in part, to effectiveness measurement. However, in view of the materiality and the importance of the objectives of many tax expenditures, we concluded in this case too that providing accountability information to Parliament on the extent to which objectives have been achieved deserves more attention. A more detailed discussion of the Department's approach to measuring and reporting the effectiveness of tax expenditures is presented in paragraphs 8.59-8.75.
8.53 In assessing the Department's commitment to measuring and reporting effectiveness, we also reviewed the Department's submission in the 1997 Annual Report to Parliament by the President of the Treasury Board (Accounting for Results, October 1997). Among other things, that report is intended to identify the results that government departments are committed to providing for Canadians as well as the methods they will use to demonstrate the results achieved.
8.54 We noted in Chapter 5 of our 1997 Report (Reporting Performance in the Expenditure Management System) that the Report of the President of the Treasury Board is an important step in providing accountability information to Parliament. The value of the information, however, depends on the nature of the results commitments and indicators proposed. We found that the results commitments of the Department of Finance in the 1997 President's Report are very general in nature (for example, "secure social programs"). Moreover, the means by which the results are to be demonstrated do not speak clearly and unambiguously to achievements related to policy or program objectives. For example, the Department's commitment to secure social programs is to be demonstrated by "implementation of new programs" and "improvements to existing social programs".
8.55 The Department of Finance should review the priority it assigns to accounting to Parliament for the effectiveness of its policies and programs and take steps to address the shortcomings we have identified.
Department's response: The Department already makes great efforts to achieve this objective. As the Department's overall response at the end of the chapter makes clear, the report is misleading in its conclusions on the Department's efforts. The Department undertakes reviews, analysis, and ongoing monitoring, and publishes a wide range of evaluations in a variety of forms, such as Budgets, consultation papers, press releases, regular reports, etc. The audit ignores this and, instead, focusses only on a narrow range of publications in order to conclude that effectiveness reporting is lacking. In fact, effectiveness reporting, like effectiveness measurement, is a large part of the Department's mandate and a key priority. If the wide range of publications produced by the Department were examined for effectiveness measurement, it would be apparent that the Department does, in fact, give priority to accounting to Parliament for the effectiveness of its programs and policies.
The audit ignores the fact that decisions always have to be made given a very real resource constraint and that, even with the best will in the world, it is not possible to continuously evaluate, in a formal sense, all policy and make this supply of information on effectiveness measurement public. The audit fails to examine whether, given this constraint, an increase in formal evaluation studies would provide the best ``value for money" for Canadians.
8.56 Effectiveness information can support accountability and change. A legislative review of the Canada Pension Plan in 1996 showed that a number of major deficiencies had evolved over time. As Exhibit 8.5 indicates, among the changes proposed were strengthening the program's stewardship, as well as accountability to Parliament and to the public.
8.57 We examined two additional cases (one within the Department of Finance and the other within Human Resources Development Canada) where there was evidence of benefits accompanying the measurement and reporting of effectiveness. The results, as outlined in Exhibit 8.6 , suggest that effectiveness information was a contributing factor in bringing about changes in program design and subsequent policy development.
8.58 In both case examples - the Cape Breton Investment Tax Credit (evaluated by the Department of Finance's Tax Policy Branch), and the former Unemployment Insurance Program (evaluated by Human Resources Development Canada) - effectiveness measurement and reporting were accompanied by significant developments affecting the programs. These developments in large part reflected, or were consistent with, the directions indicated by the effectiveness information that had been produced. It is worth noting that in both cases the information was publicly reported and was subject to considerable public scrutiny and debate.
8.60 Using the tax system to deliver government programs provides for less control over program and resource use than using direct government expenditures. Management procedures, including effectiveness measurement, need to recognize and respond to this characteristic. In addition, the need to report results to Parliament merits particular attention because the process of parliamentary control and accountability for tax measures differs from other government programs.
8.61 Tax expenditure provisions are not subject to regular annual scrutiny and debate as are those programs that go through the parliamentary supply process. Neither are they automatically subject to periodic legislative reviews. The enabling legislation for tax measures does not normally contain sunset provisions. Specific tax expenditure provisions are usually introduced as part of the Budget and result in later amendments to the Income Tax Act .
8.62 Annual reports on tax expenditures. In recognition of these characteristics, the Department of Finance publishes annually a list of tax expenditures, along with estimates of the tax revenue forgone for each one. We noted that this report has not been published on a regular basis. Of the past five reports, two were published in August, two in December and one in February. We believe that a regular publication schedule, co-ordinated with the cycle of the expenditure management system, would enhance the report's potential as an accountability vehicle for Parliament.
8.63 The Department's annual tax expenditure reports provide historical estimates, based on a sample of taxpayer returns and other statistical data, of a range of individual tax measures that the Department indicates could be considered to be tax expenditures. The 1997 report provides historical data for various periods, depending on the tax system and the last year for which data are available. For the personal income tax system, data are provided for 1992, 1993 and 1994; for the corporate income tax system, for 1992 and 1993; and for the Goods and Services Tax, from 1992 to 1995. In addition, the report provides projections, beyond the last historical year, to 1999.
8.64 The tax expenditure report includes descriptions of the many provisions, and, in the case of the personal tax system, categorizes them in a way that gives a broad indication of their area of impact (for example, employment, education, income maintenance and retirement). The report does not, however, provide information about the specific objectives that each provision is designed to achieve. Moreover, the report provides no information on the effectiveness of tax measures in achieving their objectives. The Department told us that it is planning to change the nature of the tax expenditure reports to include information from evaluations and analyses of tax measures, and that it is considering including the specific objectives of each tax expenditure in the 1998 report.
8.65 Objectives of tax expenditures. We found that the objectives of some of the tax expenditure provisions we examined in more detail were not clear. The Department states that tax expenditures are put in place to achieve a variety of economic, social and other objectives. We therefore expected to find that the Department would have identified their objectives in these terms, as well as the manner in which they could be expected to contribute to the policy objectives concerned. Among the 15 specific provisions we examined, we found cases where objectives were not clearly stated.
8.66 For example, the Department told us that there is no public statement of formal objectives for the non-taxation of employer-paid private health and dental benefits. In the case of the Goods and Services Tax rebates for municipal bodies and related boards and commissions, the Department stated that the purpose is to provide continuity in the exemptions from sales tax under the previous federal tax system. In our view, this provides no indication of the actual results to be achieved by the rebates or of linkages to the government's wider policy objectives.
8.67 The Department of Finance should:
Regarding the publication of summary effectiveness information, it is anticipated that the tax expenditure report will include further analysis and evaluation of specific tax expenditures and summary effectiveness information. It should be recognized that such analysis and reporting does not need to be done on an annual basis since most tax expenditures are not changed from year to year.
With respect to the need for regular publication of the report on tax expenditures that is co-ordinated with the cycle of the expenditure management system, the Department agrees and plans to publish the tax expenditure report in the spring of each year.
8.68 Measuring the effectiveness of tax expenditures. The Department's 1997 tax expenditure report indicates that there are more than 150 provisions that could reasonably be regarded as tax expenditures. The annual revenue costs of individual provisions range widely, from a few million to many billions of dollars. In light of such differences, as well as differences in their purposes, we recognize that not all the tax expenditure provisions warrant the same degree of attention in terms of effectiveness measurement and reporting.
8.69 The Tax Policy Branch told us that it uses a variety of techniques that contribute to effectiveness measurement. In addition to carrying out some in-depth evaluations of specific provisions, the Branch monitors the operation of individual provisions on an ongoing basis and undertakes internal analyses and assessments of issues brought to its notice through sources that include individual taxpayers, parliamentary committees, Revenue Canada and the Department of Justice. These analyses result in many changes to the tax system each year.
8.70 Other techniques include external reviews. For example, in 1996 the Department established a Technical Committee on Business Taxation. The Committee's mandate is far-ranging, including assessing the fairness of business taxation and considering ways in which it might contribute more to job creation and economic growth. As the Committee's work was still under way when we completed this audit, we were unable to determine the extent of the contribution it will make to measuring the effectiveness of tax expenditures.
8.71 Accounting to Parliament for the achievement of objectives. We recognize the importance of external reviews, as well as the ongoing monitoring and adjustment of tax measures to deal with operational problems as they arise. We do not believe, however, that these activities are sufficient to fulfil the Department's obligation to account regularly to Parliament and Canadians for the results achieved relative to the policy objectives established for specific tax expenditures. In our view, reporting on the achievement of objectives is central to providing accountability to Parliament, and should not be limited to situations where the government has concluded that certain provisions need to be changed.
8.72 In this context, we assessed the extent of the information available to Parliament over the eight-year period 1990-97 on the achievement of the objectives of various tax expenditure provisions. We found that Parliament has received such information largely as a result of tax expenditure evaluations, and that the output of such evaluations has been modest. Over the 1990-97 period, only five tax expenditure evaluations were completed, and, when we completed our audit, two were under way. By way of comparison, we note that there are about 150 tax expenditure provisions, of which 14 involve more than $1 billion annually in forgone tax revenue. Although the value of forgone tax revenue is not the only relevant criterion in this regard, it is frequently an important one.
8.73 The Department's view is that major studies or evaluations are generally only appropriate in situations where there may be concern about effectiveness. The Department notes that, based on the application of a value-for-money criterion, it adopts a variety of approaches as problems are identified through the monitoring of tax expenditures. These approaches may range from immediate remedial action to major evaluations. The Department's position is that it publishes information in documents such as the Budget Plan, consultation papers and press releases on the effectiveness measurement and analysis that leads to proposed changes. The Department notes, however, that there are limits on the extent to which it is possible to provide background analysis in all cases.
8.74 Approach to effectiveness measurement. As described in paragraph 8.27, our 1993 audit of effectiveness measurement in the Tax Policy Branch raised concerns about the adequacy of the resources dedicated to the function. We note that the Branch has recently taken action to increase the level of effort devoted to effectiveness measurement. The Branch started 1997-98 with 6.6 full-time-equivalent staff and a contract budget of $250,000 allocated to the function. During the year, four additional analysts have been hired. In addition, the Department told us that the approach to evaluation has been changed so as to produce a greater number of short-term evaluations (including, but not limited to, tax expenditure evaluations) that will allow more timely reporting of results and a better-informed public debate on tax policy issues. These are welcome developments.
8.75 Our 1993 audit of effectiveness measurement in the Tax Policy Branch also raised concerns about the adequacy of the Branch's planning process in targeting priority areas for effectiveness measurement. We reported at that time that the Branch had developed a set of criteria to help select particular areas for effectiveness measurement but had not applied the criteria consistently and clearly. Our current audit confirmed that the criteria have been maintained, and that they were applied in developing the Branch's evaluation plans for the 1995-97 period. We noted, however, that the implementation of these plans had fallen significantly behind the schedules established.
8.77 Many of the Department's areas of policy and program responsibility have been the subject of legislative reviews. We examined three specific reviews. Our aim was to identify the extent to which, in the course of the reviews, the Department of Finance informed Parliament about the objectives of the policies and programs and the extent to which those objectives had been achieved by existing approaches.
8.78 The legislative reviews we selected were of the equalization component of the Federal-Provincial Transfer Payments program, the regulatory regime for Canada's financial sector and the Special Import Measures Act . In all cases, the review process involved parliamentary scrutiny of proposals for change; we examined the effectiveness information that was available to support this parliamentary scrutiny.
8.79 Equalization. Chapter 8 of our 1997 Report (Department of Finance - Equalization Program) notes that little information is regularly provided to Parliament on the workings of the program. In particular, the Report states that the Department's Estimates documents have neither reported on the program's performance nor provided any indication of where parliamentarians can get such information.
8.80 As our 1997 Chapter 8 also notes, the review of equalization legislation is complicated by the provincial dimension. The process currently involves federal and provincial officials engaged in discussions behind closed doors, and there are concerns that Parliament is often not included in the deliberations in a timely fashion. The chapter calls for opening up this process for considering changes to such a complex, fundamental program by involving Parliament in a meaningful way.
8.81 The lack of regular reporting on program performance identified in our 1997 audit underlines the need for Parliament to receive information on program results within the context of the five-year cycle of legislative review and renewal. We therefore examined the information that the Department provided to Parliament to support the legislative review process leading to the 1994 renewal of the equalization legislation. We found that the information provided addressed some key aspects of the program's success in ensuring that provinces have sufficient revenue-raising capacity to provide reasonably comparable levels of public services at reasonably comparable levels of taxation. We believe that supplementing such information by considering a broader range of program effects (for example, possible inter-program effects in relation to other federal-provincial transfers) would further help support Parliament's involvement in the legislative review process.
8.82 Regulatory framework for the financial sector. We examined a range of documents to assess the role of effectiveness information in the legislative review of the regulatory framework for the financial sector. The material we examined included the recent audit work reported by this Office (Chapter 5, 1995 Report, Office of the Superintendent for Financial Institutions - Deposit-taking Institutions Sector), along with the Public Accounts Committee's report on that chapter and the Department's response. We also examined the Twenty-first Report of the Standing Senate Committee on Banking, Trade and Commerce (1995), and the Department's report (1997 Review of Financial Sector Legislation: Proposals for Change), along with relevant testimony and background documents.
8.83 Our 1995 Report indicated that although the government had produced several policy papers and reports dealing with different aspects of the regulatory framework, there had been no evaluation of the effectiveness of the framework to assess the extent to which its objectives had been achieved. In that Report, we noted that we had been unable to find a satisfactory answer to the question - Is the government achieving its objectives for the financial sector?
8.84 Following a hearing on our 1995 Report, the Standing Committee on Public Accounts in its Seventeenth Report, dated 20 November 1995, recommended that the government immediately carry out a comprehensive evaluation of the system for supervising and regulating federally regulated financial institutions. This was to be in preparation for the review of the legislation in 1997.
8.85 In its response to the Public Accounts Committee, the Department agreed that it was important to assess the relevance and effectiveness of existing systems and programs. However, the Department advocated a consultative review of the legislation rather than a program evaluation. It noted that, among other things, such a review would examine in more general and judgmental terms the effectiveness of the complex policy regime for the financial services sector than would a program evaluation. In addition, the Department stated that the complexity and rapid evolution of the financial sector would make it difficult to establish reliable empirical measures of the effectiveness of the regulatory framework. The Department also expressed the view that such measures of effectiveness would add little to the review process.
8.86 One conclusion of our audit is that, in terms of accountability to Parliament and Canadians in general, there is a strong case to be made for supporting legislative review processes with whatever information can be developed on the achievement of program objectives. This is so even if effectiveness measurement is not always able to produce definitive findings. In this case, we based our conclusion in large part on an examination of the Department's key report, 1997 Review of Financial Sector Legislation: Proposals for Change. This was an important consultative document in the context of the legislative review and represented the culmination of extensive prior work.
8.87 This report states that the message from the legislative review process is that the financial sector legislation is generally working well. However, the report provides no direct evidence to support this conclusion. In particular, the report does not explicitly identify the objectives of the regulatory framework and does not provide direct information on the extent to which objectives are being achieved. As noted, the Department stated in its response to the Public Accounts Committee that the rapid evolution of the financial sector, coupled with the short time that the legislation has been in place, makes measuring the achievement of specific program objectives difficult.
8.88 In our view, without developing and reporting effectiveness information to the extent possible, it is equally difficult to conclude as positively as the Department has done about the legislation's success. We believe it would be difficult for Parliament to assess the reliability of such a positive conclusion without clear evidence of the extent to which the objectives of the regulatory framework established under the legislation are being achieved, or an indication as to why such information cannot be provided.
8.89 The Department had in fact provided a preliminary assessment of the extent to which the regulatory framework was meeting its objectives in a background document to the deliberations of the Senate Standing Committee on Banking, Trade and Commerce. However, the Senate Committee concluded in its 1995 report that it was not possible to be certain at that point about the extent to which the regulatory framework was successful in achieving its objectives. The Department told us that its background material was instrumental in facilitating the Committee's deliberations and in shaping additional background work for the 1997 review of the legislation. We believe the Department's report on the 1997 review would have been more useful, both as a consultative document and as a means to provide accountability to Parliament, had it explicitly identified the regulatory framework's objectives and directly addressed their achievement.
8.90 The Department continues to be of the view that, because of the array of competing factors in the environment in which financial institutions and financial markets operate on a daily basis, the efficiency and effectiveness of the regulatory framework for the financial sector cannot be measured by evaluating discrete regulatory policies. It firmly believes that the sunset clauses in the financial institution statutes, which ensure that legislative reviews are conducted every five years, are the best mechanism to keep Parliament informed of the performance of the financial sector in relation to broad policy objectives. The Department notes that these reviews involve widespread consultations with stakeholders who provide considerable feedback on how the broad policy objectives are being met and what areas need to be addressed.
8.91 In December 1996, the Minister of Finance announced the establishment of a Task Force on the Future of the Canadian Financial Services Sector. This initiative is part of the government's effort to ensure that the regulatory framework for the sector keeps pace with the changing environment in which it operates. The Task Force's enquiries extend broadly to the public policies affecting the sector and its report is to be submitted to the Minister of Finance by September 1998. At the present time, therefore, the extent to which its work will involve measuring and reporting on the effectiveness of the current regulatory framework is not known.
8.92 The Special Import Measures Act. The Special Import Measures Act ( SIMA ) provides for the application of anti-dumping and countervailing duties on dumped or subsidized imports that are found to cause material injury to domestic producers. The Act was subject to legislative review in 1996 by the House of Commons standing committees on Finance and on Foreign Affairs and International Trade.
8.93 The decision to proceed with a legislative review was prompted, in part, by the findings from Chapter 19 of our 1992 Report, Department of National Revenue - Customs and Excise - Special Import Measures Act. In that audit, we noted that the Department of Finance, the department responsible for policy and legislation, had not completed an evaluation of the program since the Act came into force in 1984. We pointed out the need to measure the impacts and effects of the SIMA program, including both positive effects and possible negative effects, such as the extent to which protection may be detrimental to consumers, possibly increasing production costs and the ultimate prices of consumer goods. We recommended, among other things, that the Department complete a formal evaluation of the import policy framework. In its response published in our Report, the Department agreed with the recommendation.
8.94 In view of the Department's commitment, in this audit we examined background material prepared by the Department of Finance to assist the parliamentary committees during the 1996 legislative review. We found that the Department had provided a wide range of material, including some relating to the experience of other countries, with respect to the incidence, design and implementation of special import measures. As part of this material, the Department discussed the general impacts of these trade remedies, but provided no measurement of the actual extent of the impacts on Canadian consumers or on the Canadian economy as a whole.
8.95 Our examination of the joint report of the parliamentary committees that carried out the 1996 legislative review showed that the process led to important assessments of the continued relevance of the program and ways in which its operational design could be improved. In terms of the recommendations we made in 1992, however, we believe the SIMA legislative review would have benefited further if the Department had addressed more directly the effectiveness measurement concerns we raised at that time. This would have provided Parliament with a clearer perspective on the impacts of the program. The Department indicated that the Minister chose to have the legislation reviewed by a parliamentary committee.
8.96 Our assessment of the three cases described above confirmed that legislative reviews have many strengths - including timeliness and transparency. We concluded, however, that they need to be complemented by specific effectiveness measurement and reporting processes. Separate processes are required to provide Parliament with timely information on the results that policies and programs are achieving relative to their objectives. In addition to serving accountability, we believe such information enhances the quality of legislative reviews by leading to better-informed public and parliamentary debate on performance issues and helping assess the need for change.
8.99 In the policy and program areas we examined, effectiveness measurement is not clearly identified as an element in the Department's overall corporate planning.
8.100 We found that only the Tax Policy Branch has made efforts to plan its effectiveness measurement activities systematically, taking account of the Department's upcoming corporate priorities and the accompanying need for information on program results.
8.101 The Department of Finance should ensure that there are clear links between its overall corporate planning process and its effectiveness measurement function.
Department's response: It is already the case that effectiveness measurement is an overall part of the corporate planning process. Effectiveness measurement pervades the work of the Department and is intrinsic to what the Department does. In this way, the linkage between effectiveness measurement and the corporate planning process is established and maintained. This was demonstrated by examples such as the Tax Fairness document published as part of the 1996 Budget.
8.103 We noted that in contrast with the Department of Finance, Human Resources Development Canada, with approximately $55 billion in annual expenditure programs, currently has over 30 full-time-equivalent staff and approximately $7 million in contract resources for effectiveness measurement. We recognize that any comparisons have to be treated with considerable caution because of differences in the nature of the policies and programs involved. However, we believe that the striking difference in resource allocation raises questions about the adequacy of the Department of Finance's capacity to measure and report on the achievement of objectives in those areas where it has direct policy or program responsibilities.
8.104 Clearly identified effectiveness measurement resources are currently located in two of the Department's branches - the Tax Policy Branch and the Financial Sector Policy Branch. In the former, specific full-time-equivalent staff resources are identified; in the latter, effectiveness measurement activities are carried out largely under contract. No resources are specifically allocated to effectiveness measurement activities in the three remaining policy and program branches. Two of these (Federal-Provincial Relations and Social Policy, and International Trade and Finance) have clearly identified policy and program responsibilities. The Federal-Provincial Relations and Social Policy Branch noted, however, that it devotes significant resources to reviewing the Canada Pension Plan and to assessing how accurately the Equalization Program formula measures provincial fiscal capacities - which it sees as a key indicator of the program's effectiveness.
8.105 Whatever the Department's capacity to plan and undertake effectiveness measurement, an equally important question is whether that capacity is organized within the Department to maximize its potential contribution.
8.106 We expected that the organization of the effectiveness measurement function would reflect its importance as a key tool for management and for providing accountability to Parliament. As such, the function would be designed to ensure objective, independent and timely analysis, and would be organized in a way to sustain its role and importance within the overall departmental structure.
8.107 We found that the Department's current approach to effectiveness measurement results in a function that is partial in coverage and not well integrated with the overall corporate management structure. The function is clearly identifiable within only a limited number of the Department's policy and program branches that form the "front-line" of its activities.
8.108 Approaches to effectiveness measurement in other organizations. For purposes of comparison, we looked at selected characteristics of some organizations where we were told that effectiveness measurement and reporting play prominent and positive roles. The organizations we selected were the World Bank, the International Monetary Fund and Human Resources Development Canada.
8.109 We selected the first two because of the key role that results information, analysis and reporting play in their operational activities. We included Human Resources Development Canada because the Department has announced that it is developing a results-driven accountability system.
8.110 In making comparisons between the effectiveness measurement functions in these organizations and the Department of Finance, we focussed on three areas: the organizational structure of the effectiveness measurement function; the attention given to effectiveness measurement in corporate planning; and the reporting of results.
8.111 The results of our analysis are summarized in Exhibit 8.8 . All the comparison organizations, while diverse in nature, have a great deal in common with respect to how the function is treated.
8.112 In all three organizations, the effectiveness measurement function is comprehensive in its coverage and strategic in focus. Further, in each, the function reports either organizationally or functionally at the highest level within the organization. In contrast, we assessed effectiveness measurement in the Department of Finance as partial in program coverage, excluding important program areas, and not closely connected to the highest levels within the organization.
8.113 The Department's response to effectiveness measurement and reporting requirements is reactive rather than strategic. The Department has on occasion responded to audit findings concerning shortcomings in effectiveness measurement and reporting by indicating a willingness to undertake measurements in specific areas if directly requested to do so by Parliament. This response presupposes that Parliament can assess beforehand, and without adequate information, the degree of importance that should be accorded to various program or policy outcomes. In our opinion, this is unrealistic.
8.114 Further, as noted earlier, the function is not clearly identified as an element in the Department's overall corporate plans. In contrast, effectiveness measurement figures visibly in the corporate planning processes of the three comparison organizations and is a significant factor in their management and accountability regimes. There is increasing emphasis in these organizations on the reporting of results achieved relative to program objectives.
8.115 The Department of Finance should ensure that its approach to effectiveness measurement is consistent with the need to maximize its contribution to meeting the Department's obligation to account to Parliament for results.
Department's response: Both effectiveness measurement and reporting are part of the everyday work of the Department of Finance and are an essential part of the basic mandate of the Department. Work done in the Department that takes the form of analysis, review and ongoing monitoring (e.g. the Excise Act Review and the CPP Review) is ignored, even though the audit recognizes these as legitimate forms of effectiveness measurement. The Department prints evaluation information in a variety of forms, including Budgets, consultation papers, press releases, etc., but the chapter takes almost no account of these.
8.117 We found that systematic measurement of effectiveness relative to policy and program objectives, and public reporting to account for the achievement of objectives with the funds and authorities approved by Parliament are limited.
8.118 Priorities and plans for effectiveness measurement are not established within the context of the Department's corporate planning process, and the resources allocated specifically to the function are limited. Overall, we found the function to be partial in coverage and not clearly integrated with the overall corporate management structure.
8.119 In view of government policy and Parliament's expectations for improved information on results, we believe that the Department needs to review the options available to it for strengthening effectiveness measurement and reporting and then select and implement the most appropriate option.
Department's overall response: In responding to the conclusions of this audit, there are two general comments to be made.
- First, the Department does undertake a considerable number of evaluations but they are ignored because the audit takes much too narrow and mechanistic a view of what effectiveness measurement and reporting are. This is because of the audit's singular focus on so-called ``formal evaluations" - detailed, longer-term, major research undertakings - to the virtual exclusion of any other type of effectiveness measurement, which could be relatively short, but equally effective. Examples of the latter are the analyses, presented in Budget chapters, underlying the basis for a change in policy.
- Second, the audit raises the question of whether the Department should do more ``formal evaluations". In answering that question, it must be recognized that to do so would require diverting resources away from other policy analysis and development and less formal evaluations. In striking the balance between ``formal evaluations" and other critical work, the Department must be guided by the principle that Canadians must get ``value for money". Disturbing this balance would imply going counter to this fundamental principle.
As the examples in the Department's responses to specific recommendations in the chapter make clear, it would appear that a whole range of evaluations has escaped the attention of the auditors.
The objectives of our audit were:
Our scope included components of all the Department's major policy and program areas except those that relate to privatizing certain Crown corporations, to the Special Program (payment of Canada's equity interest in the Hibernia oil development), and to Domestic Coinage (production and shipment of coins). The estimated 1997-98 expenditures for the programs we excluded represent less than one percent of the Department's total expenditures.
With the exception of the Tax Policy Branch, our audit covered the period from 1992 to 1997. In the case of the Tax Policy Branch, we followed up on a special 1993 audit of the changes that had been introduced in the organizational approach to effectiveness measurement in that Branch.
Criteria
The Department of Finance should have in place procedures and practices that enable it to:
The audit focussed on the measurement and reporting of effectiveness in those areas where the Department has direct policy and program-related responsibilities. We placed particular emphasis on public reporting because of the need for Parliament and the public to be informed clearly and regularly of the results of policies and programs, as well as of the nature and extent of the Department's accountability for them.
We sent a questionnaire to departmental branches with direct policy and program-related responsibilities. The questionnaire was designed to obtain information on those responsibilities, the planning and organization of effectiveness measurement, and effectiveness measurement undertaken and publicly reported. We conducted interviews with departmental officials to obtain further information and clarification. In addition, we examined the Department's submissions to Parliament in Part III of the Main Estimates for the period 1991-92 to 1997-98, government Budget Papers produced during the same period, departmental submissions to standing committees, and White Papers and legislative review reports.
Finally, we reviewed material on effectiveness measurement activities of other organizations and interviewed selected individuals from those organizations.
Audit Team
Assistant Auditor General: Maria Barrados
Principal: Henno Moenting
Director: Jim Blain
Sophie Chen
Glenn Wheeler
For more information, please contact Henno Moenting.