Follow-up of Recommendations in Previous Reports

Main Points

12.1 Progress by the government in taking corrective action is uneven. In the departments' and agencies' responses to our audits, they committed to taking action and correcting deficiencies. In our follow-up work, we found many cases where they have made reasonable progress. But in other cases, progress is more limited. We encourage departments and agencies to complete the action necessary to resolve the issues raised.

12.2 Citizenship and Immigration Canada and the Immigration and Refugee Board are actively taking steps to address the issues raised in our 1997 audit. However, progress related to receiving refugee claims and handling failed claims cannot be fully assessed until the proposed legislation is passed and the ensuing regulations issued. Citizenship and Immigration Canada could not provide information on removals of failed refugee claimants that would indicate whether its processes were more effective.

12.3 National Defence is making slow progress addressing concerns that we raised in our 1999 audits of the management of hazardous materials and the risk of fraud and abuse by employees. In the case of hazardous materials management, the Department has not yet completed its national-level environmental management system. Therefore, little improvement in compliance can be expected before 2004 when this project is complete. Similarly, the Department has not yet fully implemented the Defence Ethics Program—the cornerstone of its efforts to promote ethical conduct and eliminate fraud and abuse.

12.4 In our follow-up of our 1998 audit of comprehensive land claims, we found that fundamental issues related to the current treaty negotiation process remain. These include the difficulty of achieving certainty and the length of time to reach agreements. Progress in these areas depends on the political will, co-operation, and good faith of all parties concerned. Although Indian and Northern Affairs Canada has taken action to address our recommendations, progress in evaluating the impacts of settlements and in capturing and reporting the complete costs of settling and implementing land claims is slow.

12.5 The government has not addressed our concerns about management oversight that we raised in our 1998 and 1999 audits on contracting for professional services using sole-source contracts. However, it has issued a new policy on advance contract award notices and has acted on our recommendations related to training. The Treasury Board Secretariat continues to reject our recommendation that departments with significant levels of sole sourcing be required to conduct annual assessments of their compliance with the regulations.

12.6 The Canada Customs and Revenue Agency has addressed most of the recommendations from our 1999 audit of the Underground Economy Initiative and continues to work on three other recommendations. These include providing additional performance indicators on the impact the Initiative has had on reducing the underground economy, improving selection of audit files, and reporting the additional tax resulting from underground economy audits.

12.7 The Department of Foreign Affairs and International Trade acted on all the recommendations raised in our 1999 audit of the delivery of capital projects in four missions. It has improved its planning capacity for capital projects and strengthened its guidelines for the planning of projects. The Department has also completed a comprehensive report on steps being taken to improve the management of property in the Department.

12.8 The Patented Medicine Prices Review Board has made good progress in addressing our recommendations from our 1998 audit, in particular for those areas most directly within its control. Several of the recommendations involved reviewing the Patent Act or regulations and consulting with Industry Canada and Health Canada. The Board has taken action on most of those recommendations as well.

Introduction

12.9 We make recommendations in each of our audits to focus attention on correcting current or future problems and to improve the management of government. We encourage management of the entities that we audit to respond to us in writing, stating whether they agree with our recommendations and how they plan to take corrective action. In turn, we publish their responses in our report, so that Parliament and the public can judge their commitment. Approximately two years after an audit, we return to determine what corrective action the entity has taken. This work forms the basis of our follow-up report.

12.10 It is important to clarify what a follow-up is, and is not. Apart from a few unique situations, it is not a second audit of the same issues. Rather it is a report on what management tells us, or can demonstrate to us, about the progress it has made toward addressing our recommendations since our initial audit. We do not exhaustively seek or examine additional evidence to support or refute what management has told us, but we do review its claims for reasonableness and report to Parliament accordingly.

12.11 This chapter contains our follow-up report on 21 chapters reported between 1995 and 1999. We also discuss our follow-up work on other chapters (Exhibit 12.1).

12.12 Many entities made satisfactory progress in taking corrective action. However, there are also many instances where delays resulted in the government not achieving as much as it could. Examples where progress is slow include the National Defence's management of hazardous materials and Indian and Northern Affairs Canada's evaluation of the impacts of settlements and reporting of the full costs of comprehensive land claims. For some long-standing issues, such as the settlement of land claims, progress is slow because it depends on the political will, co-operation, and good faith of other involved parties.

12.13 This will be the last year that we publish a chapter of follow-up reports in this format. We are currently revising what we follow up and how we report our findings. We will continue to track departmental progress in taking corrective action toward our recommendations but will focus more of our attention on reporting significant issues to Parliament.

Federal Transportation Subsidies—The Western Grain Transportation Act Program (1995, Chapter 6)

Assistant Auditor General: Hugh McRoberts
Director: Diamond Lalani

Background

12.14 In 1983 the Western Grain Transportation Act was enacted to facilitate the transportation, shipping, and handling of Western grain to export markets. The Western Grain Transportation Act Program was subsequently developed as one of the major federal direct subsidy programs for surface freight operated by the federal government. The Program assigned specific roles to the Grain Transportation Agency and the National Transportation Agency. The Act and the Program worked together with other legislation, government agencies, producers, railways, and private interests to constitute Canada's grain handling and transportation system.

12.15 In 1995 we audited the Western Grain Transportation Act Program and reported our observations and recommendations in our May 1995 Report, Chapter 6. This year we conducted a follow-up to determine the progress made by the Department since our last audit.

Focus of the follow-up

12.16 The scope of our review was restricted to identifying and analyzing the changes that have occurred since the end of our 1995 audit. Our work involved reviewing documents and discussing them with management. While we assessed the documents for reasonableness and consistency, we did not seek independent, corroborative evidence to verify the information in them.

Conclusion

12.17 Overall, the Department has made satisfactory progress in dealing with our recommendations related to the grain handling and transportation system. Since our 1995 audit, the Department has taken a number of initiatives that have led to major reforms to the system; these include a comprehensive review of the grain handling and transportation system and consultations with stakeholders.

12.18 The Department indicated that it has developed a program design to monitor the grain handling and transportation system and the impacts of the reforms on stakeholders, and it has engaged an independent third party monitor to implement the design.

Observations

Changes in the grain handling and transportation system since our 1995 audit

12.19 Since 1995, the Western Grain Transportation Act has been repealed and grain traffic is now covered by the Canada Transportation Act. The role of the National Transportation Agency was limited to the calculation of annual adjustments to the maximum rate; the Grain Transportation Agency became part of Transport Canada and was renamed the Western Grain Transportation Office for a one-year transition period before the industry assumed responsibility for allocating grain cars.

12.20 On 10 May 2000, the Minister of Transport, together with the Minister of Agriculture and Agri-Food and the Minister responsible for the Canadian Wheat Board, announced the government's intention to introduce measures to increase efficiency and cut costs in Canada's grain handling and transportation system. He stated that the reforms to be implemented by 1 August 2000 included the following:

• the establishment of a revenue cap that provides for an annual estimated $178 million reduction in railway revenues, which represents an estimated 18 percent reduction in grain freight rates from 2000-01 levels;
• the phase-in of Canadian Wheat Board tendering for logistical services for grain shipments through the ports of Vancouver, Prince Rupert, Thunder Bay, and Churchill, from at least 25 percent of their volume in 2000-01 to a minimum of 50 percent in 2002-03; and
• amendments to the Canada Transportation Act to facilitate the transfer of branch lines to community-based short lines and to simplify the Final Offer Arbitration process.

12.21 Bill C-34 (an Act to amend the Canada Transportation Act) that became effective 26 July 2000, with some provisions becoming effective 1 August 2000, has brought into effect changes to Canada's grain handling and transportation system, with an objective of more flexibility in pricing for grain movements.

12.22 The Minister also announced that an independent third party will provide a transparent and continuous monitoring program to assess the overall efficiency of the grain handling and transportation system, including the impact of changes on farmers, the Canadian Wheat Board, railways, grain companies, shippers, and ports.

Demand peaking

12.23 In 1995 we recommended that the Department study the costs and benefits associated with the way the Western Canadian grain transportation system responded to demand peaking.

12.24 The Department has not done any cost-benefit analysis in this area. However, it believes that the major reforms to the grain handling and transportation system will encourage a more commercial system, including more flexible policies for setting grain freight rates and a more commercial approach to car allocation. The new policy environment provides industry participants with a more market-based signal and alternatives to address demand peaking.

Car cycle times

12.25 In 1995 we recommended that the Department measure and report on grain hopper car cycle times on a timely basis.

12.26 In 1999 the Department implemented amendments to the Carriers and Transportation Undertakings Information Regulations that provide access to more information related to grain handling and transportation, including information on railway car cycle times. The Department indicated that it has received car cycle information related to the 1999-2000 crop year, which it is currently reviewing. The Department has also developed a program design that is intended to monitor the grain handling and transportation system and that includes elements pertaining to car cycle times.

Reviews of the program

12.27 In 1995 we recommended that the Department, along with the National Transportation Agency, take steps to ensure that the government has the information, expertise, and resources necessary to carry out any work it will be responsible for under the 1998 and 1999 reviews of the Western Grain Transportation Act Program announced in the 1995 Budget.

12.28 The Department indicated that the staff from the Canadian Transportation Agency and the Department provided information and assistance to the reviews that were conducted.

Revenue Canada—Child Tax Benefit and Goods and Services Tax Credit Programs (1996, Chapter 19)

Assistant Auditor General: Shahid Minto
Principal: Ronnie Campbell

Background

12.29 In the five years since our audit, changes have been made to both the Child Tax Benefit (CTB) and the Goods and Services Tax Credit (GST Credit) programs. The CTB was introduced in 1993 as a tax-free monthly payment to help families with low to moderate incomes. It was folded into an enriched program called the Canada Child Tax Benefit (CCTB) in July 1998. The CCTB program is the vehicle used to provide the federal portion of a joint federal, provincial, and territorial government initiative called the National Child Benefit. Like the former CTB, the CCTB is based on the number of children in the family and the level of family income.

12.30 The Goods and Services Tax Credit was introduced in 1990 to help families and individuals with low to moderate incomes offset all or part of the GST they pay. In the three provinces that have since combined their provincial sales taxes with the GST to create a harmonized sales tax, the GST Credit continues to apply. Both the CCTB and the GST Credit are administered by the Benefit Programs Directorate of the Canada Customs and Revenue Agency (CCRA).

12.31 In our 1996 audit, we noted that Revenue Canada (now CCRA) had not tested the level of awareness of the CTB and GST Credit programs and did not have profiles of the specific information needs of participants. As well, it had not measured the take-up rates to ensure that the programs were reaching all those entitled to receive them. The audit also found inconsistency in primary caregiver decisions and a lack of controls to protect against fraud and abuse and to ensure data accuracy. There was also a lack of service standards and monitoring of overpayments and no formal evaluation of the CTB and GST Credit by the Department of Finance. These observations still apply, despite the changes in these programs. In response to our audit, Revenue Canada prepared a five-point action plan that was appended to our 1996 chapter.

Focus of the follow-up

12.32 Our follow-up work involved reviewing status reports provided by CCRA and the Department of Finance on their progress in implementing our 1996 recommendations. Our observations are also based on a review of other documents provided to us and interviews held with officials of the Benefit Programs Directorate.

Conclusion

12.33 Given that it has been five years since our audit, we expected that the CCRA would have completed its action plan to address our concerns and recommendations. While the Benefit Programs Directorate has carried out a number of activities to address our observations and recommendations, several important issues remain to be addressed. These include developing participant profiles, obtaining complete and accurate knowledge about the CCTB and GST Credit take-up rates, strengthening the monitoring of overpayments, and quantifying the extent of inaccuracy of the database. As well, the Department of Finance has yet to evaluate whether the programs are meeting their intended objectives.

Observations

Activities under way to increase awareness of programs, but participant profiles not yet developed

12.34 In 1996 we observed that Revenue Canada had not tested the national level of awareness of the programs and did not have profiles of participants in the programs and their specific information needs. In addition, we found that the Department could do more to inform families of their legal obligation to report changes that could affect benefit payments, such as changes in address, marital status, and number of eligible children.

12.35 The Directorate undertook a number of activities in response to our audit. It monitored inquiries from the public to determine the nature of CCTB calls, and it conducted a client satisfaction survey of the CCTB application process. It also conducted focus groups on the wording of CCTB and GST Credit notices, and it revised client contact letters to remind recipients to report changes affecting benefit payments.

12.36 As well, the Directorate developed communication strategies for the CCTB and the GST Credit. Many of the activities in the strategies are under way, such as sending targeted mail-outs to clients to remind them of their obligations and providing program information to new immigrants. However, as part of these strategies, the Directorate still needs to develop profiles of existing and potential clients and their specific information needs.

12.37 In 2000 the CCRA conducted a national telephone survey to provide baseline information on various programs. The survey showed that 57 percent of respondents were aware that the CCRA manages the CCTB program, and 69 percent were aware that it manages the GST Credit program. However, management has indicated that the results were not conclusive because the survey questions did not separate awareness of the programs from the CCRA's role in managing them.

12.38 Although many actions have been taken, the national level of awareness of these programs is still unknown and profiles of potential and current participants have not yet been developed.

Complete and accurate information needed for take-up rates

12.39 The take-up rate is a statistic that represents the proportion of all individuals for whom the benefits are intended who actually receive them. In 1996 we recommended that the Department periodically measure and report on the take-up rate for both the CTB and the GST Credit programs.

12.40 In 1997 the Benefits Program Directorate measured the CTB take-up rate using 1995 and 1996 birth data made available by three provinces. The results indicated that approximately 95 percent of entitled children were enrolled within a year of their birth. However, these results could not be projected to yield national results due to a lack of information from other provinces and a lack of data on immigrants, emigrants, and deceased children.

12.41 In 1998 the Directorate conducted a study to determine the CTB take-up rate for a sample of children who received landed immigrant status in 1996. However, the integrity of the results was compromised because it was not possible to confirm that the children were in Canada and therefore eligible for the CTB.

12.42 Also in 1998, the Directorate approached the Privacy Commissioner for permission to obtain provincial and territorial information on the births and deaths of children. This information would enable it to determine the CCTB take-up rate on an ongoing basis, rather than negotiating periodically with the provinces for information. However, the Privacy Commissioner deemed that the release of this information would be an unwarranted invasion of privacy.

12.43 The Directorate has established a recent take-up rate for the GST Credit. Two studies, involving potential recipients from the 1995 and 1999 tax return years, showed that 98.5 percent and 98.2 percent of entitled taxpayers respectively claimed the credit. However, it is not clear that they claimed it for the correct number of children. The most recent study revealed a substantial discrepancy in the number of children recorded for the GST Credit compared with the CCTB.

12.44 In our view, complete and accurate information on take-up rates is still needed. The Directorate plans to explore other avenues to monitor take-up rates in the future.

Steps taken to ensure consistency in primary caregiver decisions

12.45 Where two or more applicants claim they are entitled to child tax benefits for the same child, the CCRA must determine who the primary caregiver is, since only that person is eligible for the CCTB benefit. In 1996 we noted that there was little quality control in this area. We recommended that uniform policies and monitoring practices for review of such decisions be established.

12.46 In response, the Directorate revised the CCTB application form to include custody information, improved the standard questionnaires used to verify the child's primary caregiver, and updated the operating manual to provide better guidance to field officers.

12.47 Despite these steps, a 1999 internal audit reported that staff were concerned about the effectiveness and efficiency of the revised questionnaire and procedures. As a consequence, the Directorate made additional changes and developed a training package to ensure consistency in the application of the procedures.

12.48 Management has informed us that the monitoring of primary caregiver decisions occurs as part of a broader review of application processing that is conducted in field offices, although specific results relating to these decisions are not available.

Overall accuracy of database not known

12.49 In 1996 our analysis suggested that there were more children on the tax benefit roll than the entitled child population overall, including some deceased children. In addition, we found that benefits were being processed when there was incomplete or inconsistent information, particularly relating to marital status. We recommended that Revenue Canada institute an appropriate control environment and enforcement regime to protect against fraud and abuse and to ensure data integrity.

12.50 In response to our concerns, the Directorate conducted two studies in 1997. One study concluded that the enrolment of non-existent children in the CTB program was not a problem to the extent that additional action was required. The second study found that 30 percent of deaths of children enrolled in the CTB program in Alberta and British Columbia go unreported and, in some of these cases, CTB payments could continue until the child reached 18 years of age. The Directorate then increased its efforts to educate clients about their obligation to inform Revenue Canada about deceased children. There has not been a follow-up study to determine the impact of these efforts.

12.51 The Validation and Controls Section was created in 1997 to deal with incomplete and inconsistent information. Revenue Canada developed a three-year quality assurance and compliance strategy and created a comprehensive database to target files for review based on the probability of mispayment and the dollars at risk.

12.52 The quality assurance and compliance strategy included plans to separately select statistically valid samples of files to determine compliance levels as well as causes and amounts of overpayments or underpayments. A random sample of files was chosen for 2000. Files that had recently been worked on were screened out. Preliminary findings for the remainder show that 68.4 percent required no adjustment, 7.6 percent were adjusted for marital status or child entitlement, 6.7 percent are still in process, and 17.3 percent could not be verified due to lack of response from the recipient.

12.53 In 2000-01, the Validation and Controls Section adjusted 27.8 percent of the targeted files it reviewed. For some adjustments related to an internal computer problem in recognizing marital status, management estimated that CTB/CCTB overpayments could have been as high as $1.5 million each year since 1993, with additional provincial and GST Credit overpayments. Many more files in the inventory have yet to be adjusted.

12.54 The Validation and Controls Section has not quantified the total value of overpayments, underpayments, and potential tax loss that it has detected as a result of its work. Nor has it projected its random sample findings to provide an overall assessment of the degree of inaccuracy in the database. Management is considering an online project management system to provide this type of information in the future.

Trends in overpayment of benefits uncertain

12.55 In 1996 we noted that the Department had only begun to analyze overpayments in the CTB program, and no information was available to track GST Credit overpayments. We recommended that it develop improved systems for the control and collection of known overpayments.

12.56 In 1996 management indicated that CTB overpayments were $69 million. However, in March 2000 the CCRA realized that a programming error had existed since 1993, and the receivables could have been $122 million. Management also informed us that at April 2001, CCTB receivables were $116 million. The magnitude of the undetected error and the delay in detecting it raises continuing concerns about the extent of monitoring of overpayments.

12.57 The Directorate has done no analysis of GST Credit overpayments, although they were estimated to be $20 million in 1996. Until recently, GST Credit overpayments were transferred to the individual accounting system, where they could not be separately identified. As of July 2001, overpayments are separately tracked.

12.58 Despite the above-noted weaknesses, the Directorate has taken some positive steps to reduce overpayments. These include reducing the number of contingency payments that are made while waiting for tax returns to be received and processed, automatically offsetting CCTB overpayments by tax refunds, and automatically transferring debts to the collections system, resulting in more proactive collections action.

Published service standard for application processing is now in place

12.59 In 1996 we reported backlogs in the processing of CTB applications and poor access to telephone lines. Program officials informed us that applicants could expect to wait two to three months from the time their application was received by the CCRA until they received a cheque. In addition, 90 percent of callers using the benefit phone lines received a busy signal. We recommended establishing standards of service for the benefit programs and reporting performance in achieving those standards.

12.60 Since our report, the Agency has published a service standard for application processing time. It states that a CCTB payment, or explanation as to why no payment is forthcoming, will be issued before the end of the second month following the month of receipt of a completed application. This standard was met 90 percent of the time in 1999 and 98 percent of the time in 2000. The Directorate has informed us that additional service standards are being developed.

12.61 Access to phone lines has improved since 1996, with the introduction of messaging systems and call centres using interactive voice response systems. Callers now receive a busy signal only 62 percent of the time. Further improvements in accessibility are expected with the recent implementation of national traffic routing of CCTB and GST Credit calls.

Department of Finance has not formally evaluated CCTB and GST Credit programs

12.62 Although the Child Tax Benefit was put in place in 1993 and the GST Credit in 1990, we noted five years ago that neither program's performance had been evaluated against program objectives. When the CTB was replaced by the Canada Child Tax Benefit and the collaborative federal/provincial/territorial National Child Benefit initiative, the three partners made a commitment to monitor and evaluate the National Child Benefit on an ongoing basis.

12.63 Since then, National Child Benefit progress reports have been published for 1999 and 2000. These reports examined changes over time in a set of indicators, such as the number of children living in poverty in Canada and the number of families leaving welfare and earning employment income. However, they concluded that it is difficult to know the degree to which the changes are attributable to the National Child Benefit. This can be determined only by carrying out an extensive evaluation. Such an evaluation is now scheduled for completion in the spring of 2002. It will include an assessment of federal child benefits as well as provincial and territorial contributions.

12.64 The Department of Finance has not yet conducted a formal evaluation of the GST Credit program to ensure that it is meeting its intended objective.

Citizenship and Immigration Canada and Immigration and Refugee Board—The Processing of Refugee Claims (1997, Chapter 25)

Assistant Auditor General: Shahid Minto
Principal: John Hitchinson

Background

12.65 In our 1997 chapter, we concluded that a thorough review of the refugee determination process was needed. The process did not quickly grant Canada's protection to claimants who genuinely needed it, and it did not discourage those who did not need or deserve Canada's protection from claiming refugee status.

12.66 Since then, the government has introduced new legislation on immigration and refugee determination. The House of Commons passed Bill C-11, the Immigration and Refugee Protection Act, on 13 June 2001. At the time of our follow-up, the Senate was considering the Bill.

Focus of the follow-up

12.67 We assessed progress on the recommendations made in our 1997 chapter, specifically on the three key steps of the refugee determination process: receiving refugee claims, determining refugee status, and handling failed claims. Our observations are based on interviews with officials of Citizenship and Immigration Canada and the Immigration and Refugee Board, and a review of documents provided to us.

Conclusion

12.68 Progress related to receiving refugee claims and handling failed claims cannot be fully assessed until the proposed legislation is passed and the ensuing regulations have been issued. However, we note that both Citizenship and Immigration Canada and the Immigration and Refugee Board are actively taking steps to address the issues we raised.

12.69 The Immigration and Refugee Board has made satisfactory progress in addressing most recommendations related to refugee determination. However, Citizenship and Immigration Canada could not provide information on removals of failed refugee claimants from the country that would indicate whether its processes were more effective; the Department is still developing a National Case Management System that will help address this issue. Both organizations still need to improve the information they provide to Parliament.

Observations

Receiving refugee claims

12.70 In 1997 we recommended that Citizenship and Immigration Canada review the mechanisms used in applying the eligibility criteria set out in the Immigration Act. Under Bill C-11, the decision on eligibility must be made within three working days after the claim is received, or the claim is automatically referred to the Immigration and Refugee Board. The Department is working on a project to implement a new screening process. Until the proposed legislation becomes effective and the project is completed, we cannot assess progress toward addressing the recommendation.

12.71 We also recommended that Citizenship and Immigration Canada and the Immigration and Refugee Board co-operate to establish a common strategy for ensuring that all information relevant to the processing of refugee status claims is collected in a timely fashion. In our view, both the Department and the Board have made satisfactory progress toward increasing their co-operation. The Administrative Framework Agreement between the Department and the Board has been followed by agreements on sharing of information and on co-ordination of priorities.

12.72 In 1997 we found that the majority of persons claiming refugee status at ports of entry in Canada arrived from a country other than one where they may be subject to persecution. We also found that the "safe third country" provision of the legislation to safeguard the system against an influx of claimants who did not need Canada's protection had never been implemented. The proposed legislation maintains that provision. Negotiations with the United States to apply the provision were suspended in December 1997. Since then, no formal negotiations have taken place.

Selection of Board members

12.73 We recommended that the government ensure that the selection process for Board members provide greater certainty that appointments or reappointments to the Immigration and Refugee Board are based on the qualifications needed to respond to the complexity and the importance of the task. In 1997 the Minister requested that the advisory committee created in 1995 to assist in the selection of Board members recommend mechanisms to be put in place to improve the process.

12.74 The Ministerial Advisory Committee developed screening tools to evaluate the candidates according to established criteria. The Committee had five meetings in 2000 to recommend candidates to the Minister and two meetings by the end of June 2001. Between January 2000 and June 2001, the Board received more than 300 applications for Board member positions. The Committee recommended 115 candidates to the Minister, and 33 were appointed to the Board.

12.75 In 1999 the Immigration and Refugee Board implemented a new performance appraisal program for Board members. One of the purposes of this program is to provide a basis for the members' continuing tenure with the Board. Recommendations for renewals are no longer the responsibility of the Ministerial Advisory Committee but that of the Performance Review Committee of the Board. The Review Committee contributes to the reappointment process by overseeing the appraisal process that provides a report to the Minister at the end of a member's term as advice on reappointment.

12.76 We noted that the turnover rate for Board members has decreased since 1997, from 20.9 percent to 9.4 percent in 2000. New appointments are generally made for two years, and the Board seeks the reappointment of members for five years. In April 2001, there were 196 Board members and 19 vacant positions. This was an improvement from 1997 when 28 positions were vacant.

12.77 As shown in Exhibit 12.2, the number of refugee claims received by Citizenship and Immigration Canada has increased in recent years. Management at the Department also confirmed that trend for the current fiscal year. The inventory of cases at the Immigration and Refugee Board is also increasing. We remain concerned that the Board may not have the necessary complement of staff to deal with an increasing inventory.

Informed decision making

12.78 In 1997 we noted that the information available to Board members did not always foster informed and equitable decisions. The Department and the Board have taken a number of measures to address this issue. In 1998-99, the Immigration and Refugee Board implemented the National Learning Framework and customized training programs for both new and experienced Board members. Since implementation of the framework, 24 new members and 22 more experienced members have been trained. The Board organized issue analysis sessions in 2000 and 2001 to enhance the quality of decisions by promoting greater consistency in the treatment of similar cases. More sessions are planned for the coming year.

12.79 The Board created national geographic networks based on the six major world regions from which refugee claims are received. These networks are intended to facilitate the exchange of information and viewpoints among members, refugee claim officers, and researchers. Country-specific analyses of differences in acceptance rates of claimants from the same country are also produced. The Board has undertaken an inter-regional exchange of members to deal with high variances in acceptance rates.

12.80 Additional resources have been obtained to create a team responsible for quality assurance. Although the need for consistency must be reconciled with adjudicative independence, the Department and the Board can work to identify areas where decision makers can adopt a consistent approach without impinging upon the independence of individual decision makers.

Case processing

12.81 In 1997 we also noted that there were no clear directives governing the order of priority for processing cases and that each region set its own priorities. Personal information forms were received late, and postponements and adjournments were a major cause of Board delays. Since then, the Board has introduced a number of initiatives to improve its case management. A priorities co-ordination agreement was signed with Citizenship and Immigration Canada. Postponements and adjournments have been reduced by 41 percent since 1997-98. In March 2001, Personal Information Forms were received 37 days after the claim, compared with 47 days in 1997. Overall, the average processing time was reduced from 13 months in 1996-97 to 9.6 months in 2000-01. However, the continuing increase in the number of claims may result in longer processing times in the coming years. In addition, the Board is developing a case management system and expects to implement it in the spring of 2002.

Organizational climate

12.82 In 1997 we observed that most Board members and refugee claim officers described the organizational climate as negative or variable. Since the audit, the Immigration and Refugee Board has promulgated a vision, mission, and values statement in the organization. The National Learning Framework integrates learning for Board members and refugee claims officers. Teamwork is promoted throughout the organization, and this approach is particularly reflected in the national geographic networks.

Handling failed refugee claims

12.83 We recommended that Citizenship and Immigration Canada ensure that the risk-of-return review be within the scope of the objectives set for the Post-Determination Refugee Class in Canada, and that the review be carried out in an efficient and timely manner. A new process is being developed as a result of Bill C-11. Therefore, we could not assess progress on this recommendation.

12.84 We noted that more rigour was needed in evaluating applications for permanent residence on humanitarian and compassionate grounds. In December 1999, Citizenship and Immigration Canada issued a guideline on the handling of humanitarian and compassionate cases, and it provided training to staff. A coding system was implemented to track cases.

12.85 In 1997 we noted that the Department was experiencing a great deal of difficulty carrying out removals of failed refugee claimants. The Department continues to work on a co-ordinated approach with other government departments and agencies to improve co-operation from countries in issuing travel documents and accepting the return of their citizens. To date, the Department has negotiated bilateral removal arrangements with eight countries. However, factors we observed in 1997 that made it difficult to carry out removals quickly are still present.

12.86 We also noted in 1997 that the Department had insufficient information to manage removals adequately. The National Case Management System, a centralized database that provides management information and case-tracking support for enforcement activities, is currently operational in the Atlantic, Montreal, Toronto, and Vancouver offices. However, the system is not yet available to all offices across Canada. Consequently, the Department still lacks information that would help it determine whether its removal processes have become more effective.

Information to Parliament

12.87 In 1997 Citizenship and Immigration Canada and the Immigration and Refugee Board submitted Estimates to Parliament that did not provide complete and relevant information on the processing of refugee claims. Our review of the Department's 1999-2000 Performance Report showed that the Department still needed to put more emphasis on outcomes of activities related to the determination of refugee status. The Board's most recent report, for 1999-2000, showed that the Board needed to account more specifically for its achievements in enhancing the quality of decisions and promoting greater consistency in the handling of similar claims.

Expenditure and Work Force Reductions in the Public Service and in Selected Departments (1998, Chapters 1 and 2)

Assistant Auditor General: Maria Barrados
Principal: Kathryn Elliott

Background

12.88 In 1998 we published two chapters on expenditure and workforce reductions in the federal public service. These chapters reported the results of our audit of the departure incentive programs approved by the government. In Chapter 1 we examined the role of central agencies in applying the reductions across the federal government. Chapter 2 dealt with the ways that certain departments had carried out the reductions.

12.89 Expenditure reduction. Our 1998 audit examined the extent to which announced targets for expenditure and workforce reductions have been or are likely to be achieved. We concluded in Chapter 1 that the government was likely to meet the expenditure reduction targets of Program Review.

12.90 Savings. We examined the extent to which the costs of workforce reduction and initiatives had resulted in savings. In Chapter 1 we concluded that it was not possible to measure the precise extent to which incentives contributed to savings. We emphasized the difficulty of consolidating and reconciling information originating from various sources on workforce reductions and on costs associated with departure incentives. We also discussed a few aspects of the development and implementation of the cost-effectiveness test known as "payback," which is similar to the concept of "return on investment."

12.91 Recommendation. We recommended that the Treasury Board Secretariat work with departments to determine the costs of final workforce reduction and final reductions in salary and wage expenditures and ensure that Parliament is fully informed.

12.92 Response. The Standing Committee on Public Accounts agreed with our findings. At the end of Program Review, the President of the Treasury Board committed to providing Parliament with a detailed report on the final costs of workforce reduction and the final reductions in salary and wage expenditures.

Focus of the follow-up

12.93 Our follow-up focussed on the above-noted recommendation. We examined the files that the Treasury Board Secretariat used to support its final report, and we had discussions with Secretariat representatives. We excluded from our review the part of the recommendation applying to departments.

Conclusion

12.94 In the spring of 2001, the Treasury Board Secretariat gave us a draft report on the costs and expenditure reductions of the various departure incentive programs. In the fall of 2001, the Treasury Board Secretariat tabled this report with the Public Accounts Committee.

12.95 In our opinion, the information provided by the Treasury Board Secretariat is a reasonable report to the Public Accounts Committee on the number of recipients and the cash incentive payments. However, measuring the extent to which these incentives contributed to ongoing savings continues to be difficult.

Observations

Savings: a matter of point of view

12.96 In its report to the Public Accounts Committee, the Treasury Board Secretariat indicates that about 60,000 employees left the federal public sector under a special departure program during the target period (1995-96 to 1998-99). Of these employees, about 44,000 were from the federal public service.

12.97 The Secretariat reports that the departure incentive programs cost $4.2 billion and indicates ongoing annual savings of $3 billion since 1999. In 1998, we noted the following difficulties in trying to measure the extent to which incentives had contributed to savings:

• tracking of expenditure reductions over a four-year period during which the expenditure base had changed significantly; and
• deficiencies in the design and application of payback and the reporting structure.

12.98 In its report to the Public Accounts Committee, the Treasury Board Secretariat maintains its position that savings will continue to accrue in the future. In our opinion, the Secretariat's estimates do not completely capture the net savings, as we pointed out in 1998. However, the weaknesses of the systems and the changes in government programs make it difficult to improve the estimates.

Department of Finance—Effectiveness Measurement and Reporting (1998, Chapter 8)

Assistant Auditor General: Maria Barrados
Principal: Barry Leighton

Background

12.99 In 1998 we reviewed the extent to which the Department of Finance undertook effectiveness measurement and reported publicly on the achievement of objectives in areas where it had direct responsibilities for policies and programs. We also examined the extent to which the Department had the capacity, structures, and procedures to measure adequately the effectiveness of its policies, programs, and operations.

12.100 We found at that time a lack of clarity in information regularly provided to Parliament on the Department's policy and program responsibilities. We observed that improvement was needed in providing Parliament with clear statements of policy and program objectives and with results achieved relative to objectives. We concluded that the Department's effectiveness measurement practices were not closely integrated with the overall corporate management process, thereby resulting in only partial coverage of programs.

Focus of the follow-up

12.101 In our follow-up work, we reviewed reports prepared by the Department on the progress it had made in responding to our observations and recommendations. This included an action plan prepared in response to a Public Accounts Committee request. We also conducted interviews and examined relevant documents.

Conclusion

12.102 The Department of Finance has made some improvements, including better communicating to Parliament of its responsibilities and clarifying its programs and policy objectives. We found a significant improvement in the information the Department provides on the objectives and results of tax expenditures. Limited progress has been made in advancing its effectiveness measurement and reporting in other responsibility areas, and it has yet to improve the information it provides to Parliament on the effectiveness of many of its programs and policies.

12.103 The Department has recently established an Internal Audit and Evaluation Committee and designated a Director responsible for this function. Other efforts to increase its capacity for effectiveness measurement are still at an early stage. The integration of effectiveness measurement with the Department's overall corporate planning is not yet fully implemented.

Observations

Accountability to Parliament for results

12.104 Communicating responsibilities to Parliament. In 1998 we recommended that the Department ensure that the information it provided regularly to Parliament clearly and consistently identify its own responsibilities, the departments and agencies with which it shares responsibilities, and how the responsibilities are shared.

12.105 Our follow-up found that the Department has made some progress in clearly identifying its own responsibilities in public documents. In particular, we noted in 1998 that statements were not clear on the Department's role for the Canada Pension Plan (CPP) and the regulatory framework for the financial services sector. Our follow-up found that in both cases the Department has clarified its role in both its reports on plans and priorities and its performance reports.

12.106 In those cases where the Department shares policy and program responsibilities with other departments and agencies, these key public documents often do not identify the other organizations involved or how responsibilities are shared. We found few references in these documents to sources of additional information on programs and policies.

12.107 Information to Parliament on objectives. In 1998 we recommended that the Department provide Parliament with clear and consistent statements of objectives for the full range of its policy and program responsibilities. Our follow-up found that the Department has made some progress in providing clear statements of its objectives and these statements were consistent in various public documents. Work remains to be done to ensure that objectives reflect overall program goals rather than operational or activity-level outputs.

12.108 We followed up on three areas where we did not find objectives stated clearly and consistently in 1998. Our analysis of public documents, including the reports on plans and priorities and the performance reports, indicated some improvement in the statements of objectives for international trade and investment policy, the CPP, and the regulatory framework for the financial services sector. However, the stated objectives for these programs tend to focus on operational or activity-level outputs rather than on policy or program goals. We did note an improvement in the consistency of objectives across public documents.

12.109 Providing effectiveness information to Parliament. In 1998, we found significant gaps in the public reporting of results relative to objectives. In most cases, reporting focussed on specific areas of operational activity or programs but did not provide information on the achievement of objectives. We recommended that the Department review the priority it assigns to accounting to Parliament for the effectiveness of its policies and programs and take steps to address the shortcomings we identified.

12.110 As in our previous audit, we found that the Department provided the most effectiveness information for the Public Debt program. As well, in key documents the Department refers to sources of additional information for this program. As in 1998, we noted a lack of measures of Canada's performance that would allow comparisons with other highly developed capital markets. In our 2000 Report, Chapter 8, Department of Finance—Managing Canada's Debt: Facing New Challenges, we observed that the Department needed to establish a more complete range of performance measures to ensure that it has a sound basis for assessing how well the program is doing at meeting its objectives. The Department states that it is working to address these concerns as outlined in its 1999-2000 Debt Management Report. For the remaining program and policy areas, we found limited improvement in the reporting of effectiveness information.

12.111 Information on the objectives and results of tax expenditures. In 1998 we recommended that the Department identify clear objectives for each tax expenditure in terms of its intended contribution to economic, social, or other objectives; report the objectives to Parliament in the annual report on tax expenditures; report summary effectiveness information in the annual report on tax expenditures where available; and establish a schedule for publishing the annual report on tax expenditures that complements the expenditure management cycle.

12.112 Our follow-up found that the Department has made significant progress in implementing this recommendation. Starting with the subsequent annual report on tax expenditures in 1998, the Department has worked to clarify statements on objectives of individual tax expenditures. Our review of the tax expenditure provisions we examined in 1998 found that a majority now has clearly stated objectives. In addition, the tax expenditure reports include summary effectiveness information for those expenditures evaluated. We noted that in the period since our last audit only three tax expenditures had been evaluated.

12.113 In 1998 the Department agreed with our recommendation on establishing a regular schedule for publishing the annual report on tax expenditures that complements the expenditure management cycle. Of the past four reports, two were published in July, one in June, and one in September. The Department informed us that its subsequent work in the area led it to conclude that an early spring publication date is not feasible because some of the tax expenditure data are not available until December. It also stated that some of the data on projections rely on economic variables as they exist shortly before the February Budget. In addition, since the timing of the Budget or Economic Update fluctuates, the publication date for the report will fluctuate as well.

12.114 Legislative review processes. In 1998 we examined the effectiveness information that was available for three areas subject to legislative review. Our assessment confirmed that while legislative reviews had many strengths, they needed to be complemented by specific processes for effectiveness measurement and reporting. Our subsequent review of these three areas found that one had produced effectiveness information on a regular and public basis. More work needs to be done to ensure that effectiveness information is produced on a regular basis in addition to legislative reviews.

Effectiveness measurement capacity and corporate planning

12.115 Integration of effectiveness measurement with the Department's overall corporate planning. In 1998 we recommended that the Department ensure that there are clear links between its overall corporate planning process and its effectiveness measurement function. Our follow-up found that the Department has started to take action to address this recommendation, but that it is in the early stages of doing so. In 2001 the Department established the Internal Audit and Evaluation Committee, which includes the Deputy Minister and four assistant deputy ministers. One of its responsibilities is to ensure that the results of internal audits and evaluations are incorporated into departmental priority-setting, planning, and decision-making processes. The Committee is to approve management action plans in response to audit and evaluation reports. It plans to meet three to four times a year and as needed.

12.116 Department's approach to effectiveness measurement. In 1998 we expressed concern about the Department's capacity to meet its responsibilities for effectiveness measurement and reporting. We recommended that the Department ensure that its approach to effectiveness measurement meet its obligation to account to Parliament for its results.

12.117 The Department has undertaken work to strengthen its capacity to measure effectiveness. In 2001 it established the position of Director of Internal Audit and Evaluation with responsibility to co-ordinate and facilitate evaluation activities across the Department. It is also developing a plan for undertaking evaluation work, which it expects to be completed by the end of 2001. The number of staff specifically allocated to effectiveness measurement work has remained constant since our 1998 audit. Only the Tax Policy Branch has personnel dedicated to evaluation and its own work plan. Although other branches do not have dedicated personnel, they undertake effectiveness measurement on an ad hoc basis using mostly contract resources.

Indian and Northern Affairs Canada—Comprehensive Land Claims (1998, Chapter 14)

Assistant Auditor General: Maria Barrados
Principal: Joe Martire

Background

12.118 The extent of Aboriginal claims to land and resources is illustrated in Exhibit 12.3. Under the government's 1986 Comprehensive Land Claims Policy, the purpose of settlement agreements is to provide certainty and clarity of rights to ownership and use of land and resources.

12.119 Settlement agreements are generally referred to as modern-day treaties. They cover a wide range of issues such as title to certain lands, wildlife harvesting rights, participation in land and resource management, economic development, and self-government arrangements. At July 2001, 14 claims had been settled; this leaves 71 in various stages of negotiation, of which 51 relate to land in British Columbia. However, not all Aboriginal groups that have unresolved land claims have joined the established treaty process. For example, in British Columbia, about 70 bands have not entered the treaty process.

12.120 The 14 settled claims involve approximately 62,000 Aboriginal people, full ownership of over half a million square kilometres of land, a financial package of about $2 billion, and other considerations. Indian and Northern Affairs Canada plans to spend about $340 million in 2001-02 to negotiate and implement comprehensive claim settlements.

12.121 In September 1998, we reported to Parliament on the settlement, implementation, and reporting of comprehensive land claims. The audit concluded that resolving comprehensive land claims is a difficult challenge for all those involved and that there was a need to resolve a number of key issues. Specifically, our observations and recommendations included the need to take the following action:

• Enhance the achievement of certainty by addressing differing expectations relating to rights, capacity, and implementation of claim settlements.
• Expedite the treaty process and ensure adequate rigour in determining the nature and amount of assets to be included in the settlement.
• Strengthen implementation plans, improve reporting, and conduct evaluations of settlement impact.
• Report the complete costs of reaching and implementing settlements and compare them with relevant budgets.

12.122 The standing committees on Aboriginal Affairs and on Public Accounts held hearings in October and November 1998 on the results of the audit and reaffirmed our recommendations. In particular, the Standing Committee on Public Accounts stressed the need for information on outcomes achieved and complete costs.

Focus of the follow-up

12.123 In our 1998 audit, we examined four claim settlements finalized between 1984 and 1993, including several aspects of their implementation. We also discussed with the British Columbia Treaty Commission certain issues applicable to it.

12.124 Two claims have been finalized since the 1998 audit: the Nisga'a Treaty in B.C. and the Tr'ondek Hwech'in Agreement in the Yukon. Our follow-up work did not review these settlements in detail. We discussed with departmental officials the progress achieved in addressing the issues raised in our 1998 audit. We also reviewed written assertions and a sample of related documentation provided by the Department.

Conclusion

12.125 The Department is taking action to address most of our observations and recommendations. However, it needs to accelerate its efforts in evaluating the implementation of settlements and in capturing and reporting the full cost of reaching and implementing settlements.

12.126 We also noted that fundamental issues remain related to the current treaty negotiation approach, namely the difficulty of achieving certainty and the length of time to reach agreements. Progress in these areas depends on the political will, co-operation, and good faith of all parties concerned.

12.127 Achieving certainty in settlements continues to be a significant challenge because governments and Aboriginal groups have different expectations of what a modern treaty is and does. The slow pace of claims settlements is costly to all parties, both financially and in terms of foregone economic development. For example, as at 31 March 2001, outstanding loans to First Nations to fund their participation in negotiations totalled approximately $550 million. For a few First Nations, the amount of their negotiation loans now approaches 50 percent of expected settlement amounts and has become another impediment to settlement. In its 2001-02 Report on Plans and Priorities, the Department states that it expects to conclude up to 15 more final agreements over the next three years.

Observations

Achieving certainty continues to be a significant challenge

12.128 In 1998 we reported that certainty can mean different things to different parties and that there are many factors that affect the achievement of certainty. We recommended that the Department further address differing expectations of the parties relating to rights, capacity, and implementation to enhance the achievement of certainty concerning ownership and use of lands and resources.

12.129 Fundamental differences in the parties' aims for land claims have been recognized for many years. The 1985 report of the task force to review comprehensive land claims policy (commonly known as the Coolican report) noted, "The federal government has sought to extinguish rights and to achieve a once-and-for-all settlement of historical claims. The Aboriginal peoples, on the other hand, have sought to affirm the Aboriginal rights and to guarantee their unique place in Canadian society for generations to come."

12.130 The government's 1986 Comprehensive Land Claims Policy recognizes that Aboriginal groups have objected to blanket extinguishment of all Aboriginal rights in exchange for the benefits provided through settlement agreements; it states that "alternatives to extinguishment may be considered provided that certainty with respect to lands and resources is established."

12.131 In December 1997, the Supreme Court decision in the case of Delgamuukw v. The Queen ruled that where Aboriginal title is established, it is the right to exclusive use and occupation of land; further, governments must conduct meaningful consultations and sometimes obtain consent and pay compensation for activities that infringe on Aboriginal title. This ruling increased Aboriginal expectations concerning access to lands and resources, government recognition of Aboriginal title, and consultation and compensation.

12.132 In its 1997 response to the report of the Royal Commission on Aboriginal People, Gathering Strength — Canada's Aboriginal Action Plan, the Government of Canada undertook to discuss its comprehensive claims policy and process with Aboriginal, provincial, and territorial partners in order to respond to concerns about the existing policy. Policy review discussions were held between Indian and Northern Affairs Canada and the Assembly of First Nations (AFN).

12.133 In May 2000, the AFN proposed that a panel of experts be established to review the 1986 Comprehensive Land Claims policy and to make recommendations for an alternative. In July 2000, the Minister of Indian and Northern Affairs responded that, in his opinion, the existing policy is sufficiently flexible to accommodate the concerns of First Nations. According to the Department, the best way to resolve outstanding issues of Aboriginal rights and title and to address capacity and implementation issues is at individual negotiating tables where the unique circumstances of each claim are taken into account.

12.134 In 1998 the governments of Canada and British Columbia and the First Nations Summit of B.C. launched a Tripartite Review of the British Columbia Treaty Process. In April 2000, the three parties agreed to "A Statement on Certainty Principles for Treaty Negotiations in British Columbia." In October 2000, the First Nations Summit proposed a new approach for achieving certainty in treaties to the federal and provincial ministers. This Reconciliation Model for Treaties proposes that treaties recognize, affirm, and protect Aboriginal title and do not include any language that expressly or implicitly extinguishes Aboriginal title and rights. Rights not specifically included in the treaty would continue, but a First Nation would agree to only assert, exercise, and enforce its rights and title as provided in the treaty. The B.C. government does not believe that such a "non-assertion" model would bring the degree of certainty the province requires for treaties.

12.135 We have observed that the parties at individual negotiating tables continue to seek legal wording for rights provisions that are acceptable to all. For example, the certainty provisions used in the Nisga'a Final Agreement refer to modifying any Aboriginal rights the Nisga'a may have and defining the rights the parties have agreed the Nisga'a may exercise. According to the Department, any Aboriginal rights other than as set out in the Agreement are released. At another negotiation table, three independent lawyers were asked to make recommendations to the negotiator on how to best satisfy everyone's concerns about certainty provisions; no recommendation has yet been made.

12.136 The degree of certainty achieved through land claims agreements ultimately depends on the satisfaction of all parties with the results realized. Achieving certainty continues to be a challenge as parties' expectations of treaties remain significantly different. Finding ways to achieve certainty that are acceptable to all parties is critical to expediting the treaty process and ensuring that the terms of any treaty that is negotiated will be honoured.

Claims settlements continue to be slow

12.137 Since our 1998 audit, two additional settlement agreements have been finalized; five agreements-in-principle have been signed or initialled, of which two have not yet been ratified; and eleven framework agreements have been concluded. Nevertheless, in our view, the rate of progress will need to accelerate significantly if the Department is to achieve the expected results included in its 2001-02 Report on Plans and Priorities. In that report, the Department stated that it expects to conclude up to 15 agreements-in-principle and up to 15 final agreements over the next three years.

12.138 We noted that the time taken to reach 12 of the 14 modern-day agreements signed ranged from 6 to 25 years. For example, the Nisga'a Agreement, which came into effect on 11 May 2000, took 24 years to settle and consisted of approximately 1,100 pages of documentation spread over seven agreements. The Nisga'a Agreement is the first treaty to contain both self-government and land claims arrangements in one agreement. We further noted that 11 of the 14 signed agreements are located north of the 60^th parallel. In the north, the federal government owns the land and has constitutional authority, populations are smaller, and Crown land is more available. By contrast, most of the ongoing negotiations are in the south. They require the participation of provincial governments that control most of the lands and resources, and involve the interests of more non-parties.

12.139 Indian and Northern Affairs Canada has taken some steps that could contribute to expediting the negotiation process. In December 1998, a three-year Negotiation Preparedness Initiative with funding of $29 million was approved. The Initiative was intended to enhance the capacity and expertise of Aboriginal groups that have asserted a land claim to prepare for negotiations on land and resource components of comprehensive claims settlements. Furthermore, in order to provide benefits to First Nations earlier in the process and to protect land and resources during negotiations, treaty-related measures were introduced in British Columbia in 1999. The measures can provide benefits that will be included in an anticipated final agreement or provide funds to First Nations to conduct studies and undertake economic analyses of particular areas of interest, such as forest management.

Evaluation of settlement impact is needed

12.140 Our 1998 audit recommended that Indian and Northern Affairs Canada, together with other settlement partners, ensure that implementation plans contain adequate details on time frames, project costs, and cost sharing. We also recommended that the Department address issues relating to fragmentation, completeness, and timeliness of reporting on implementation, and, together with other settlement partners, perform timely evaluations of settlement impact.

12.141 Settlement agreements reached after 1986 require implementation plans that identify certain activities to be undertaken and payments to be made to implement settlement agreements. These plans are intended to ensure that all parties have a common understanding of how the agreement will be implemented and to secure their commitment to meet their obligations. The Department believes that implementation plans need to be flexible planning tools in order to avoid the need to revise the plan when operational circumstances change. In response to our recommendation, the Department is revising its 1989 handbook for implementation negotiators and managers.

12.142 In 1998 we found that the Land Claims Obligation System (LCOS), used by the Department to report on the status of federal obligations, tracked only activities and processes, not results produced and costs incurred. In addition, the information in LCOS was too general to be used to assess the status of certain obligations.

12.143 In response to this finding, the Department is developing a new monitoring system, the Treaty Obligation System, as a possible replacement for LCOS. The Department informed us that the provincial government and the Nisga'a Nation have agreed to use this new system as a common database to monitor the Nisga'a Implementation Plan. According to the Department, the Treaty Obligation System is not expected to be fully operational for the Nisga'a settlement until late 2001 at the earliest.

12.144 In its response to the Public Accounts Committee, the Department indicated that it would develop an evaluation framework to evaluate the social and economic benefits of settlements and to consider including such information in its Performance Report. The Department stated that it expected to have made significant progress in developing a multi-year plan to evaluate settlements by December 1999.

12.145 The Department has conducted studies of the key expectations of various parties, including performance indicators and measurement strategies. However, it has not yet developed an evaluation framework and is not certain when the first evaluation will be completed. The Department has informed us that it is exploring various options for including an evaluation requirement in the negotiation of new agreements.

More work needed to report complete costs of reaching and implementing settlements

12.146 Our 1998 audit recommended that the Department report to Parliament the complete costs of reaching and implementing settlements and compare them with relevant budgets. The Public Accounts Committee concurred with our recommendation and called for the Department to include in its annual Performance Report complete costs of reaching and implementing land claims settlements; these costs should include the potential value of assets in settlement agreements, consolidated costs of other federal departments, and the costs of implementation projects and revenue and resource sharing. In its reply to the Committee, the Department pointed out that some of the Committee's suggestions would require time to implement because consultation with and agreement of other settlement partners and the development of systems to gather claims-related costs would be needed.

12.147 The Committee's recommendation on cost reporting has not yet been implemented. Although the Department is developing a report format that would provide details on its own expenditures, it has no plans to report the costs of other government departments; however, it plans to provide electronic links to other departments' financial information, as appropriate. Departmental officials believe that other government departments are responsible for reporting on the funds allocated to them by Parliament and that gathering such information would be a logistical challenge.

12.148 We continue to believe that Indian and Northern Affairs Canada should provide a leadership role in reporting the complete costs of reaching and implementing settlements. In our view, this information would enable Indian and Northern Affairs Canada, Parliament, and the public to understand the types of activities and costs related to comprehensive land claims.

Patented Medicine Prices Review Board (1998, Chapter 17)

Assistant Auditor General: Maria Barrados
Principal: Patricia MacDonald

Background

12.149 In 1998 we examined the work the Patented Medicine Prices Review Board undertook to regulate the prices of patented medicines in accordance with relevant legislation and regulations. We also reviewed whether the Board reported to Parliament complete and reliable information on the extent to which its price review activities affect prices of patented medicines, price trends for all drugs, and research and development spending by pharmaceutical patentees.

12.150 We found that the Board had contributed to the containment of patented drug prices and had taken steps to assess the impact of federal price regulation. However, other factors also had contributed to the containment of prices, and we were concerned that the Board's estimates of its savings to the Canadian health care system were overstated.

12.151 We identified issues pertaining to the legislative framework that needed to be reviewed. We observed that the Board needed to ensure that its price review decisions were clear and transparent, and to identify cost-effective means to check the accuracy of price information submitted by manufacturers. Finally, we observed that improvements were required in the reporting of drug price trends and pharmaceutical research and development expenditures.

Focus of the follow-up

12.152 In our follow-up work, we reviewed status reports prepared by the Board on its progress in responding to our observations and recommendations. These included an action plan prepared in response to a Public Accounts Committee request. We also conducted interviews and examined relevant documents.

Conclusion

12.153 The Board has made good progress in implementing our recommendations, in particular those most directly within its control. It has taken action to improve the measurement of its activities and to enhance the public's understanding of its role and scope of jurisdiction. The Board has made some progress in checking the accuracy of price information submitted by patentees. As well, it has initiated action to ensure the clarity and transparency of its price review decisions. The Board has also explored opportunities to improve co-ordination of scientific reviews of new drugs and reviewed its reporting of drug price trends.

12.154 Implementing several of our recommendations involved reviewing the Patent Act or regulations, and this required the Board to consult with Industry Canada and Health Canada. The Board brought these recommendations to the attention of both departments and has taken action on most of them. The Patent Act has been amended to allow for the Minister of Health to enter into agreements with provinces respecting distribution of funds collected through voluntary compliance undertakings. Work on addressing recommendations related to price factors and comparisons with prices in foreign countries is ongoing. An assessment of the continued relevance of reporting pharmaceutical research and development expenditures was undertaken, with no changes being made. However, no action has been taken on clarifying the jurisdiction of the Board over patented medicines whose patents are dedicated for public use. We encourage the Board to complete its implementation of our recommendations.

Observations

Impact of federal regulation of patented drug prices

12.155 Board's measurement of the effectiveness of its activities. In 1998 we recommended that the Board improve its approach to measuring the effectiveness of its activities, taking into account other factors that have an impact on patented drug prices. Our follow-up found that while the Board has not attempted to estimate savings from its activities since our audit, it has taken steps to improve its approach to measuring effectiveness.

12.156 In October 2000, the Board's Executive Committee finalized an evaluation plan that served as a starting point for identifying evaluation priorities. The plan calls for four evaluations to be completed in 2001-02 and 2002-03. Evaluation priorities are to be reviewed annually and to be included in the Board's Strategic Plan. As well, in 1999 the Board signed a Memorandum of Understanding with the Minister of Health to provide analyses and reports on expenditure trends, price levels, interprovincial price comparisons, and cost drivers facing public drug benefit plans. To date, the Board has produced several studies on these issues.

Legislative framework

12.157 Board's role and scope of jurisdiction. In 1998 we recommended that the Board continue to identify opportunities to enhance public awareness of its responsibilities. Our follow-up found that the Board completed a consultation with stakeholders, begun at the time of our audit, and in late 1998 it released the Road Map for the Next Decade. This document provided an outline of the measures the Board was to take to address concerns raised, including the development of a new communications plan. The Board plans to report on the evaluation of its Consultation Policy and Communications Plan in 2001-02.

12.158 Price factors. In 1998 we recommended that the Board, in consultation with Health Canada and Industry Canada, bring to the government's attention the need to review the legislation and regulations to examine the relevance of the price factors as well as the use of comparisons with prices in the foreign countries listed in the regulations. Our follow-up found that the Board has undertaken work to address this recommendation, and while the legislation and regulations have not been amended, some changes have been initiated. However, more work is needed to fully address this recommendation.

12.159 In December 1998, the Board brought this recommendation to the attention of Health Canada and Industry Canada. Health Canada indicated that this issue was being addressed by the Federal/Provincial/Territorial Task Force on Pharmaceutical Prices. As well, in 1999 the Board established the Working Group on Price Review Issues to examine the use of United States Department of Veterans Affairs formulary prices in the international price comparison, the price review process for new patented drugs, and the price guidelines for category 3 drugs.

12.160 The Working Group issued its report on the first issue in late 1999, and the Board implemented the report's recommendations to take into account the prices charged to the United States government in calculating U.S. prices for comparison purposes. The policy was effective commencing January 2000, and it is subject to a two-year transitional period. The Working Group's report on the second issue was completed in November 2000, and the Board conducted a broader consultation in 2001 on its specific proposals to implement the report's recommendations. Finally, the Working Group has begun its consideration of the review and analysis of the price guidelines for category 3 drug prices and expects to complete this work in 2002-03.

12.161 Patent dedication. We recommended that the Board bring to the attention of the government the need to review the legislation and regulations to clarify the jurisdiction of the Board over patented medicines whose patents are dedicated for public use. The Board told us that it had brought this issue to the attention of the government. It stated that it is not aware whether the government has considered the need to clarify the legislation and it has not pursued this issue further. The Board also stated that it has not established, and does not currently plan to establish, criteria for investigating complaints about the pricing of drug products with patents dedicated for public use as they are expected to be rare and it is difficult to foresee all possible eventualities. The Board told us it received no complaints in 1999 and 2000.

12.162 Voluntary compliance undertakings. In 1998 we recommended the review of legislation and regulations regarding distribution of money collected through voluntary compliance undertakings. Our follow-up found that in 1999 the Patent Act was amended to allow the Minister of Health to enter into agreements with provinces respecting the distribution of funds collected through voluntary compliance undertakings.

12.163 Pharmaceutical research and development expenditures. We recommended the review of the continued relevance of the requirement for reporting pharmaceutical research and development expenditures. In response, Health Canada conducted an assessment of the means of obtaining and reporting this information. The assessment observed some weaknesses in the research and development data reported by the Board, but it concluded that the reporting of this information was still relevant. The government has not made any statutory or regulatory changes, nor has it requested the Board to make changes to the research and development reporting requirements.

Review of prices of patented medicines

12.164 Clarity and transparency of price review decisions. In 1998 we recommended that the Board ensure that its price review guidelines are rigorously applied, or that they are revised if considered inappropriate. The Board should also ensure that in cases where exceptions are made or its guidelines are not applied, the reasons for and effects of its decisions are clear and transparent. The Board told us that the guidelines have been applied in all cases since our last audit. In 1999 and 2000, there were nine cases where prices exceeded the guidelines. Decisions on four cases were reported publicly. For the remaining five cases, it was decided, following investigations, that no further action was warranted. Although these cases were reported in aggregate in the annual report, the specifics of the results of the investigations were not made public. The Board is currently considering proposals from the Working Group to make the price review process more open and transparent.

12.165 Verification of price information. We recommended that the Board identify cost-effective means to check the accuracy of price information submitted by patentees. Our follow-up found that the Board has taken initial steps to improve the accuracy of price information. In late 1998 it published a study that examined the methods it used to verify foreign patented drug price information filed by patentees; among the observations, it found that, overall, patentees were complying with the regulations. Based on the study, the Board enhanced its verification process using the methodology developed in the study to verify the prices of all patented drug products under investigation. In addition, in January 2001 the Board's Executive Committee approved the recommendations of an evaluation of the foreign price verification process. It also included in its 2001-04 Research Agenda a plan to conduct another study to verify foreign patented drug price information filed by patentees.

12.166 Co-ordination of scientific reviews of new drugs. We recommended that the Board explore opportunities to improve the co-ordination of its work with Health Canada in performing scientific reviews to categorize new patented medicines. In response to our recommendation, the Board examined this issue through the Working Group on Price Review Issues. The Working Group concluded that there was little work that could be done to link the priority review process at Health Canada with the Board as the respective reviews are undertaken for different purposes and it would be difficult to reconcile the different approaches.

Reporting of drug price trends

12.167 In 1998 we observed weaknesses in the Board's Non-Patented Medicine Price Index (NPMPI), derived from Statistics Canada's Industrial Product Price Index (IPPI). We recommended that the Board work with Statistics Canada to improve the reporting of drug price trends. After our audit, the Board decided to stop reporting the NPMPI. It also formed a task force with Statistics Canada to reach a better understanding of the indices used to measure price changes. The task force concluded that the indices do not measure the same thing. The task force's report confirmed the decision to stop publishing the NPMPI.

Electronic Commerce: Conducting Government Business via the Internet (1998, Chapter 19)

Assistant Auditor General: Doug Timmins
Principal: Nancy Cheng

Background

12.168 A technology "Blueprint" document was developed in 1994 as a strategic framework for renewing government services using information technology. As one of its initiatives, the government committed to making electronic commerce its preferred way of doing business by 1998. Following the September 1997 Speech from the Throne that introduced the "connecting Canadians" agenda, the government also committed to becoming a model user of the information highway by 2000.

12.169 In 1998 we examined the government's progress in using the Internet for internal operations and service delivery. We concluded that progress had been made in three key areas: a public key infrastructure project for secure electronic commerce, the legal framework, and common technology infrastructures for interoperability among departments and agencies. Nevertheless, we observed several key risks and made recommendations to the government to address them.

Focus of the follow-up

12.170 Our follow-up consisted of reviewing a status report prepared by the Treasury Board Secretariat on the actions taken to address our 1998 recommendations. We also reviewed supporting documentation provided to us by the Secretariat and we held discussions with its officials. We obtained information about the Government On-line initiative but we have not audited its activities.

Conclusion

12.171 The launching of the Government On-line initiative in 2000 has changed the context substantially for our 1998 observations and recommendations. Our recommendations continue to be relevant and significant, but the required time to implement them and the complexity of some of them have changed. In summary, we observed good progress in many areas, such as developing on-line business applications and a government-wide strategy for advancing electronic commerce. Yet, much work remains to put in place common technology infrastructures to support electronic service delivery across government.

Observations

Providing leadership and sponsorship to advance electronic commerce

12.172 In 1998 we were concerned that electronic commerce for government lacked strategic direction. We recommended that its goals be defined and that senior sponsorship and a strategic plan be put in place.

12.173 The October 1999 Speech from the Throne continued with the theme of connecting Canadians. It stated: "By 2004, our goal is to be known around the world as the government most connected to its citizens."

12.174 In early 2000, the government announced a framework for Government On-line (GOL) as its new strategy for electronic commerce. The framework set the direction for delivering services electronically to Canadians at the time and place of their choosing and established parameters for implementation by 2004. It included general timelines, roles and responsibilities of departments and agencies, and oversight by the Treasury Board Secretariat.

12.175 The GOL agenda provided the focus and strategic direction that the electronic commerce initiative lacked. The President of the Treasury Board was appointed lead minister for GOL; the Chief Information Officer Branch of the Secretariat oversees and co-ordinates GOL activities centrally for 28 core departments and agencies.

12.176 We noted good progress in the development of the Government of Canada Web site. In 1998 we observed that the site was static and not user-oriented. Under GOL, significant efforts have been made to re-orient the site. In adopting a citizen-centric approach, the government anticipated needs from a user perspective. Through clustering and the use of three gateways, the site now channels searches to the respective links and sources of information.

12.177 In 1998 we also recommended that the Treasury Board Secretariat keep an inventory of electronic commerce initiatives in departments and agencies for sharing good practices and lessons learned. Through the GOL initiative, development of those initiatives was tracked and reported in the government Intranet as pathfinder projects.

Using public key infrastructure to support secure electronic commerce

12.178 In 1998 we recognized the progress made by the government in developing a management framework for establishing a public key infrastructure (PKI), including contracting and developing a technical product and drafting model certificate policies and certification practice statements. To advance the use of PKI in government, we recommended that the government involve business and program managers in identifying and developing applications requiring secure data communication, that the issue of certifying the public be addressed, and that attention be paid to project management risks for the technical product.

12.179 As part of GOL, departments and agencies have been identifying programs and business processes to be put on-line. As a result, business and program managers have been engaged in planning business initiatives and working with information technology professionals to develop applications. The Treasury Board Secretariat brought to our attention several applications that make use of PKI to provide secure data communication, such as secure messaging in government and the Canada Education Savings Grant system for communicating with financial institutions.

12.180 The 1998 audit underscored some project management risks, including the technical complexity involved in cross-certification among departments and agencies and with the public. Subsequently, the contract for the technology product was successfully completed and the final product has been delivered to the government. In 2000 a government-wide licence was obtained for the use of necessary software products in implementing a public key infrastructure. Through its secure messaging project, the government has successfully demonstrated the feasibility of cross-certification among departments and agencies.

12.181 The issue of certifying the public poses a more significant challenge, and its complexity increased as a result of the GOL agenda. Since the audit, several departments have been issuing digital certificates to some public users. In addition, the government has conducted a series of consultation sessions with private sector representatives to elicit their input on authentication services. The Treasury Board Secretariat advised us that a model for centrally managed PKI services to provide the public with digital certificates to meet GOL requirements is under development. More efforts will be required before the government will be in a position to certify the public on a larger scale, particularly in the context of GOL.

Addressing potential legal liabilities

12.182 In 1998 we also raised the issue of potential government liability through the use of the Internet to provide services or carry out transactions, especially where no contractual relationship exists and third parties choose to rely on government information. We recommended that potential liability issues be identified and addressed as the government introduces new electronic commerce initiatives.

12.183 The PKI policy was approved in 1999. It provides a governance and management structure that outlines departmental responsibilities and liabilities relating to the use of digital signatures in government. The Treasury Board Secretariat advised us that it also plans to use features of the technical product as well as notices and subscriber or service delivery agreements to mitigate risks associated with potential legal liabilities. As departments and agencies implement electronic initiatives under GOL, they will need to consider legal liability issues and take steps to manage them.

Building common technology infrastructure to support electronic commerce across government

12.184 In the 1998 audit, we noted the importance of interoperability across departments and agencies in providing seamless access to government services. We recommended that efforts in identifying and adopting technical standards be accelerated to support common technology infrastructures and provide the basis for interoperability, and that PKI standards be put in place when the cryptography product becomes ready for deployment.

12.185 The government referred to the development of its strategic information management and information technology (IM/IT) infrastructure initiative as a response to our recommendation. This initiative included the adoption of a federated architecture approach to technology infrastructure and a process that facilitates planning, design, co-ordination, and implementation of technology elements to support government-wide IM/IT capabilities.

12.186 As we noted in the past, progress in advancing information technology standards in government has been slow. The Treasury Board Secretariat indicated that it is launching a revitalization of the Treasury Board Information Technology (TBIT) standards to support common technology infrastructures. It is also preparing a revised approval process for IM/IT standards to better meet the needs of departments and agencies, including an accountability regime.

12.187 Much work remains to be done in the area of common technology infrastructure and the standards required for supporting it. Moreover, a recent study commissioned by the Secretariat to examine issues relating to the implementation of a government-wide public key infrastructure identified a number of organizational and technical issues to be addressed.

12.188 Further attention and increased oversight in developing common technology infrastructures and establishing and enforcing standards are needed to fully support seamless access to government services. The ability to interoperate across government, especially among those departments and agencies with common beneficiaries and stakeholders, is essential for the Government On-line initiative.

Transport Canada—Investments in Highways (1998, Chapter 25)

Assistant Auditor General: Hugh McRoberts
Director: Diamond Lalani

Background

12.189 Transport Canada has been the federal government's key arm for investing in provincial and territorial highways through a series of ad hoc contribution programs and several multi-year cost-sharing agreements with the provinces and territories.

12.190 In 1998 we audited investments in highways and reported our observations and recommendations in our December 1998 Report, Chapter 25. This year we conducted a follow-up to determine the progress made by the Department since our last audit.

12.191 The Audit and Advisory Services within Transport Canada conducted a follow-up audit of the recommendations that we made for the Highway Contribution Programs and reported the results in May 2001.

Focus of the follow-up

12.192 The scope of our review was restricted to identifying changes that have occurred since the end of our 1998 audit. Our work involved discussion with management of the results of the Department's internal follow-up audit. We have placed reliance on the information presented in the Transport Canada Audit and Advisory Services Report of May 2001.

Conclusion

12.193 Overall, based on the report prepared by internal audit, the Department has made satisfactory progress in carrying out our recommendations or addressing most of the issues and concerns. Since the announcement of a new highway infrastructure program, the Department has remodelled its framework for contributing to provincial and territorial highway projects.

12.194 The Department recognizes, however, that there is still work to be done. This includes completing the Management Committee guidelines, addressing a number of provincial/territorial accounting practices, enhancing functions of the Highway Contribution Management System, and specifying the expected frequency of reporting highway data by provinces and territories.

Observations

Changes in the highway contribution programs since our 1998 audit

12.195 The Department has indicated that most of the highway contribution agreements signed in 1987, or in subsequent years, had expired or were due to expire prior to 2003.

12.196 The federal 2000 Budget proposed that new highway funding be made available from the $2.65 billion Physical Infrastructure Program. In June 2000, the Minister of Transport sought Cabinet approval of the program design and funding approach for a new Strategic Highway Infrastructure Program. In March 2001, the Cabinet accepted the Minister's recommendation and approved the highway component of the Physical Infrastructure Program. The Treasury Board approved a submission from Transport Canada to fund the Strategic Highway Infrastructure Program with up to $600 million over five years.

12.197 The Department has indicated that it now has a new investment strategy, as set out in the Strategic Highway Infrastructure Program, which consists of two major components: Highway Construction and National System Integration. This program replaces the previous 1974 National Highway Transportation Policy and defines the conditions under which the federal government will now contribute to both the proposed conventional highway construction initiatives and alternative financing arrangements, including public-private partnerships and tolling schemes.

Department's progress on our recommendations

12.198 The Department has made the following representations in the Transport Canada Audit and Advisory Services Report of May 2001 (157700-022) that was approved by the Transport Canada Audit and Review Committee on 26 June 2001.

With the necessary government approvals secured, the stage is now being set for the Department's redesigned approach to national highway funding. A new Performance and Accountability Framework will support the next generation of formal agreements with the provinces and territories and enable comprehensive program evaluation and reporting on results in 2005. An Audit Framework has also been prepared respecting the Treasury Board's new transfer payment policy that will govern both project and program audits.

The tools and methods to manage and administer the new program have been significantly improved. With respect to the 1998 audit recommendations, we conclude that the majority of issues and concerns have been effectively addressed. A comprehensive agreement template has been developed for consistent application among all provinces and territories. This agreement now calls for formal, documented project applications that must address federal program objectives and environmental impact assessments. Applications will be also subject to closer scrutiny by the federal government employing detailed screening criteria.

While the Management Committee continues as the focal point for agreement management and oversight, its role, responsibilities and procedures have been effectively strengthened. Projects approved by the Management Committee will be documented describing their nature, duration, costs and contracts. All proposed amendments to construction work, or adjustments to funding, will be subject to prior approval of the Management Committee.

We have concluded that most of the new or revised monitoring and control mechanisms affecting environmental assessments, claims payment, cost overruns, annual audits and progress reports, and project certification, directly address the concerns raised by the 1998 audits. Equally beneficial to program administration will be the approved addition of five persons to Surface Programs for the duration of the program. We also note that the means of tracking and documenting the key milestones, decisions, and finances of each agreement and project have been improved. Enhanced information management practices, and a redesigned Highway Contribution Management System, will now ensure that program integrity and due diligence are maintained.

As in 1998, the collection of relevant highway data for analysis, decision-making and reporting continues to present a challenge in 2001. At the provincial/territorial level, scarce resources and incompatible data collection practices and priorities have pre-empted the ability to provide federally requested information. However, under the new Strategic Highway Information Program (SHIP), a resolution to this data problem may be at hand. As a precondition of federal funding, provinces and territories will be required to provide minimum highway data to the federal government. Given the historic data collection problems, the Department's National Highway Information System initiative offers an opportunity to create a national database for highway information. With the support of the Council of Deputy Ministers of Transportation, and funding from Transport Canada, a feasibility study among the provinces and territories is expected to report in September 2001 on the benefits and costs of participating in this national system.

12.199 Subsequent to the approval of the May 2001 Audit and Advisory Services Report, the Department has indicated that the feasibility study will not be completed by September 2001. The work is being co-chaired by Transport Canada and Transport Quebec and will likely be considered by deputy ministers at their fall meeting next year, in September 2002.

Contracting for Professional Services: Selected SoleSource Contracts (1998, Chapter 26)
Sole-Source Contracting for Professional Services: Using Advance Contract Award Notices (1999, Chapter 30)

Assistant Auditor General: Hugh McRoberts
Director: Jaak Vanker

Background

12.200 Two complementary principles are central to government contracting: best value and open access to contracting opportunities (Exhibit 12.4). The principle of best value is to ensure that in acquiring goods or services for the Crown, the government receives the best combination of value and price. The principle of open access gives all qualified vendors a fair chance to do business with the Crown without political or bureaucratic favour. An open, competitive bidding process provides the best guarantee that both of these principles will be respected.

12.201 In our 1998 Report, Chapter 26 and our 1999 Report, Chapter 30, we conducted two audits examining aspects of the government's treatment of sole-source contracts for professional services.

12.202 In 1998 we audited a sample of 26 sole-sourced professional service contracts initiated by five departments: Correctional Service Canada, Fisheries and Oceans Canada, Health Canada, National Defence, and Transport Canada. Because our sample was a selected one, the results cannot be generalized statistically to the broader population of similar contracts. However, the cases were not chosen in any targeted or biased way that would have led to the sample being unrepresentative of the practices used in sole-sourcing contracts of the type we audited.

12.203 In our audit we found only 8 of the 26 contracts that we examined had satisfied one of the necessary conditions that make such awarding permissible. Indeed, in most instances it was not even clear from the contract file which exception the manager had invoked to justify sole-sourcing the contract.

12.204 We recommended that deputy ministers ensure that those to whom they delegate contracting responsibility fully understand the dual principles of government contracting policy (open access and best value) and be held accountable for adherence to them. In addition, we recommended that deputy ministers ensure that when contracts are sole-sourced, the circumstances are fully consistent with the provisions of the Government Contracts Regulations.

12.205 In 1999 we audited a statistical sample of 50 sole-sourced contracts for professional services that were let using an Advance Contract Award Notice (ACAN). National Defence, Industry Canada, Human Resources Development Canada, and the Canadian International Development Agency initiated the audited contracts.

12.206 We found that only 11 percent of the contracts we examined had a justification for sole-sourcing on file that complied with the conditions stipulated in the Government Contracts Regulations.

12.207 Managers are supposed to make this determination, justify it, and document it before deciding to sole-source and before posting an ACAN. However, in 89 percent of the cases we examined, the uniqueness of the contractor was either not determined at all (that is, management was fully aware that the firm selected was not unique) or was unsupported in fact. We made a number of recommendations directed at the Treasury Board Secretariat.

12.208 Both chapters were the subject of a hearing by the Standing Committee on Public Accounts and resulted in subsequent reports by the Committee—Report 28 (36th Parliament) and Report 12 (37th Parliament) respectively. The government has tabled responses to the recommendations made by the Committee in each of the reports.

Focus of the follow-up

12.209 Our follow-up comprised reviewing progress reports and supporting documentation supplied by the Treasury Board Secretariat, Public Works and Government Services Canada (PWGSC), and other departments and agencies; conducting interviews; and performing analysis to assess the reliability of the progress reports supplied to us. We specifically examined changes in policies for contracting for services that were using either an ACAN or the non-competitive approach. Our work did not extend to seeking corroboration for the assertions made to us.

Conclusion

12.210 Our recommendations in the two chapters addressed five areas: training, policy clarification, enhanced recourse, management oversight, and accountability (Exhibit 12.5).

12.211 The Treasury Board Secretariat, PWGSC, and most of the departments and agencies have described the steps that they have taken to address deficiencies identified by the Office and the government's standing committees. Their reports indicate that significant action has been taken in the areas of training and policy clarification. However, the Treasury Board Secretariat continues to reject our recommendation on enhanced recourse for disputes related to contracts that fall outside the trade agreement related to procurement.

12.212 The government states that it has taken steps to improve management oversight and accountability for contracting through Results for Canadians and the new Internal Audit and Active Monitoring Policy. In addition, individual departments have taken steps to improve their practices in these areas. However, in our view, the government's actions have not adequately addressed our concerns.

Observations

Professional Development and Certification Program under way

12.213 To ensure that contracting authorities fully understand the objectives of government contracting, the Professional Development and Certification Program for procurement specialists was recently launched. The program is composed of two components: continuous learning and professional certification.

12.214 Sixteen courses that address the critical skill requirements have been developed and are now available. These courses cover Government Contracts Regulations as well as policy considerations that form the government's contracting management framework.

12.215 The professional certification component is under study.

12.216 According to the Treasury Board Secretariat, this new program complements departmental in-house courses and the ongoing training carried out by the Secretariat and key partners including PWGSC; the program is also in keeping with the government's Modern Comptrollership initiative.

Policy clarification

12.217 In 1999 there was a lack of guidance to departments on how the challenge process for ACANs was to be conducted. We recommended that this issue be addressed. The new Advance Contract Award Notice policy and the Best Practices—Guide for Managers is now available on the Treasury Board Secretariat Web site at www.tbs-sct.gc.ca.

12.218 As part of the Procurement Reform initiative, adjustments to the ACAN policy have been made and are communicated in the Best Practices—Guide for Managers. The new practices include the following:

• principles to guide the review of a supplier's statement of capabilities;
• clear timelines for posting the notices;
• clear language in the notices that improve suppliers' understanding of the work to be performed, the process, and how they may proceed;
• guidance to improve the quality of information received from suppliers;
• elimination of words that may have discouraged suppliers from expressing their interest in work identified in an ACAN; and
• guidance to ensure that, in the event that a statement of capabilities is rejected, that decision will not be made by the same official(s) who originally decided to proceed through the ACAN process.

12.219 In addition, a number of key departments have issued their own procedural clarifications. It is too early to assess the extent to which, under the revised policy, ACANs may play a role beyond adding transparency to what we view as an otherwise non-competitive process.

Enhanced recourse

12.220 For contracts that fall outside one or more of the trade agreements, the potential supplier's recourse on issues related to the procurement process is to appeal back to the very department whose actions are the object of the appeal. The supplier may also follow the potentially very costly route of taking the matter to the courts.

12.221 The Treasury Board Secretariat believes that the current avenues of recourse are sufficient and continues to reject our recommendation that an independent dispute resolution mechanism be established.

Management oversight

12.222 The results of both audits led us to believe that closer management oversight of the contracting process, particularly as it related to the decision to sole-source, was needed. We made a number of recommendations to encourage this.

12.223 Most departments have a process by which some of their contracts are subject to review by a management Contract Review Committee. Generally, however, this committee reviews only contracts let directly by the department. For most departments and agencies, their major contracts are let by PWGSC, the contracting authority for the largest number and value of service contracts. In most instances, such contracts are not subject to review by the departmental Contract Review Committee, and PWGSC has no corresponding structure. Apart from normal management supervision, PWGSC has a limited review mechanism in place, but it applies only to the very largest contracts.

12.224 In general, government managers believe that accountability for contracts issued by PWGSC is unclear and, accordingly, management oversight of these contracts is weak. Where PWGSC is the contracting authority, most departments do not carry out further reviews of the solicitation procedure used and hold PWGSC accountable for the sourcing decision. Although contracting officers at PWGSC may seek additional information from the originating department in support of the sourcing decision, they in turn view the department as being responsible for that information and its relevance to the decision. The result is that both PWGSC and the originating department work on the assumption that the other is exercising the appropriate oversight.

Accountability

12.225 The Treasury Board Secretariat agrees that departmental management is responsible for ensuring compliance of their contracting procedures with the Government Contracts Regulations and with our recommendations that departmental management be held accountable for this result. However, accountability requires regular and credible information on performance. In the area of contracting, such information is not currently available. Thus, to ensure that some key performance information would be available, we recommended that in departments with significant levels of sole-sourcing (more than 50 contracts for over $25,000 per year), the Treasury Board Secretariat require that internal audit conduct annual assessments of the compliance of these contracts with the regulations. The Secretariat rejects this recommendation, claiming that it is neither appropriate nor cost-effective. We have been informed that in some departments internal audit has planned ad hoc audits of contracting.

Revenue Canada—Underground Economy Initiative (1999, Chapter 2)

Assistant Auditor General: Shahid Minto
Principal: Barry Elkin

Background

12.226 In our 1999 audit of the Underground Economy Initiative in Revenue Canada (now the Canada Customs and Revenue Agency), we made several observations and recommendations. These focussed on the need to accurately report the tax impact of the Initiative's enforcement activities, such as audits; the need to maintain a balanced approach that incorporates activities to promote voluntary compliance; and the need to involve other federal departments and other partners in sharing information to better deal with the underground economy.

12.227 In particular, we observed that over the first five years of the Initiative the income tax impact attributable to the detection of unreported income by the Initiative's audits was difficult to determine, but it was likely significantly less than the $300 million reported.

Focus of the follow-up

12.228 Our follow-up examined the commitments made in the Agency's response to the 1999 audit recommendations. We also reviewed the status report prepared by the Underground Economy Audit Directorate on the progress in addressing our recommendations up to September 2001, as well as other internal reports provided to us. We conducted limited interviews in Ottawa with the Manager of the Underground Economy Audit Directorate.

Conclusion

12.229 Since our 1999 audit, the Agency has continued to work on addressing the recommendations. The Initiative continues to operate at approximately the same level of resources as in the period covered by the audit, including about 770 audit staff (25 percent of the Agency's entire Small and Medium Enterprises compliance resources). The Initiative also continues to carry out the same types of activities, namely audits and activities to promote and assist compliance of businesses and the public. Improvements have been made in broadening the number of sectors for audit attention, maintaining and strengthening the activities to promote voluntary compliance, and continuing to provide assistance to business in maintaining good books and records.

12.230 Three recommendations continue to need attention by the Agency. These include the following:

• identifying and reporting additional performance indicators to indicate improvements in overall compliance by taxpayers;
• clearly identifying and reporting the additional tax received as a result of the detection and reassessment of unreported income; and
• targeting high-risk files for audits of unreported income.

12.231 The third recommendation is of particular concern since 60 percent of the Underground Economy Initiative audits fail to reveal any additional unreported income. The current level of additional income being reassessed may indicate either that the Initiative has had a positive effect on improving compliance behaviour or that the selection of files for audit can still be improved.

Observations

Activities to promote compliance

12.232 In 1999 we noted that it was important to complete the social research to determine whether social marketing activities could be a valuable tool to combat the underground economy. The Agency completed the research but at the time of our follow-up had not carried out a broad-based social marketing campaign. The research did confirm the desirability of communicating messages through the home renovation sector to consumers, encouraging them not to participate in the underground economy. Fifteen hundred contractors participated in seminars presented in 2000-01 by the Canadian Home Builders Association and the Agency to assist them in delivering this message to consumers. A decision is still to be made on undertaking a further consumer campaign in the home renovation sector.

12.233 Activities to promote voluntary compliance by businesses were part of the multi-dimensional and balanced approach of the Agency's strategy to deal with the underground economy. In 1999 we noted that the number of community visits had declined. Over the past two years, the Agency has strengthened these activities. It has carried out many more community visits and implemented a national program to assist business taxpayers in meeting their obligations to keep good books and records. The Agency has also conducted on-site visits to help companies implement the Contract Payment Reporting System.

Work continues on identifying data sources

12.234 External data sources are a key to identifying unreported income and we recommended that the Agency collaborate with other governments and other partners to access these sources. Since the audit, additional data have been received from provincial and municipal governments while ensuring that the Access to Information and Privacy Act is respected. The Agency is continuing to work on identifying other sources of data on unreported income; however, there is still an outstanding problem in matching these data to tax return information because of the lack of a personal identifier on the data.

Sector profiles completed

12.235 Auditors need good background information on the industry sectors that they are auditing. The Agency has addressed our recommendation to complete the sector profiles for the four industry sectors initially focussed on by the Initiative. It has also produced 15 new sector profiles. We further recommended that the Agency reconsider the Initiative's audit focus and include high-risk taxpayers in all sectors. Currently, 50 percent of the underground economy resources are still dedicated to the construction/home renovation and hospitality sectors. Since March 2000, each tax service office has been able to utilize the other 50 percent to address up to three other high-risk sectors as long as they meet certain criteria.

Reporting of additional income and tax

12.236 In 1999 we attempted to identify the tax results from the Underground Economy Initiative as distinct from other ongoing activities or those carried out by other agencies. In particular, we were interested in clearly identifying the additional unreported income identified and the income tax reassessed by the underground economy audits. However, we found that this was difficult since the Agency's systems did not record these data. The audit estimated that the additional tax for the first five years would be less than the $300 million that the Agency reported. We recommended that the additional income and the additional tax from the underground economy audits be clearly recorded and reported by the Agency.

12.237 Since 2000-01 all offices are identifying the additional unreported income detected by the underground economy audits and it is now reported on a national basis. The unreported income for 2000-01 was $200 million. The tax resulting from this additional income is not yet separately reported in the Agency's performance information. The estimated tax could be $34 million.

Selection of audit files

12.238 Based on our review of the underground economy audit files, we recommended that the targeting and selection of audit files be improved. The Agency informed us that it has continued to enhance its targeting of high-risk files for unreported income by computer-assisted risk assessment and data matching, the use of informant leads, and new reporting systems. However, our review of the performance data provided for 2000-01 indicates that only 40 percent of the completed audits have identified any additional unreported income.

Better performance indicators needed

12.239 Since the activities in the Underground Economy Initiative were broad-ranging and somewhat specialized, we recommended that performance indicators be developed to report the results of these activities. For example, since some activities are meant to encourage compliance, information on changing attitudes to compliance may be important to report. The Agency agreed in its response that a wide range of performance indicators was needed to measure and report on the performance of the Initiative activities and on their long-term effects on compliance.

12.240 The performance indicators currently used for reporting results of the Initiative's activities cover fiscal impact, efficiency, quality service and client education, responsible enforcement, and visibility/education. The indicators do not monitor change in compliance behaviour, which is the objective of the non-audit activities of the Underground Economy Initiative. It is therefore not clear whether compliance behaviour has improved since the start of the Initiative.

Statistics Canada—Managing the Quality of Statistics (1999, Chapter 3)

Assistant Auditor General: Maria Barrados
Principal: Barry Leighton

Background

12.241 In 1999 we examined the extent to which Statistics Canada assesses the adequacy of quality management systems and practices in its statistical programs to ensure that it has the information needed to manage and report on quality. We also reviewed Statistics Canada's self-assessments in four selected surveys to determine whether they provide the Agency with reasonable assurance of the adequacy of quality management systems and practices. Finally, we looked at whether Statistics Canada appropriately informs users about data quality and the methodology used to develop the statistics.

12.242 We found that the Agency employed a number of formal quality assessment mechanisms, but that individual programs did not apply them consistently. We concluded that the mechanisms did not provide systematic, transparent information on the adequacy of quality management systems and practices in the Agency's statistical programs or on the level of quality achieved.

Focus of the follow-up

12.243 In our follow-up work, we reviewed documentation provided by Statistics Canada on actions taken in response to our recommendations. We also conducted interviews with Agency management.

Conclusion

12.244 Statistics Canada has made good initial progress in implementing all of our recommendations. Since 1998 Statistics Canada has better integrated its approach to managing quality in its statistical programs. As part of this effort, it has developed a revised quality assessment mechanism that will be applied to all programs. It has also taken steps to implement recommendations from its 1998 self-assessment of the Uniform Crime Reporting Survey (UCR) in order to correct weaknesses it identified. In addition, the Agency has begun to include more information on statistical quality in its Performance Report. Finally, Statistics Canada has revised its Policy on Informing Users of Data Quality and Methodology and started to apply this policy. However, the Agency needs to maintain momentum to ensure that the initiatives begun to correct the weaknesses our 1999 audit identified are completed.

Observations

Corporate assessment of quality

12.245 Application of formal assessment mechanisms. In 1999 we recommended that Statistics Canada ensure that formal quality assessment mechanisms are applied consistently so that they provide systematic information on the adequacy of quality to management systems and practices in individual statistical programs and, to the extent possible, information on the quality achieved. Statistics Canada has developed a revised formal quality assessment mechanism designed to be applied consistently to all statistical programs. The Integrated Program Reporting (IPR) process consists of three levels of assessment: an Annual Consolidation of Indicators (ACI); a Biennial Program Review (BPR); and a Quadrennial Program Review (QPR). The ACI is the responsibility of the Corporate Planning Division, which prepares tables from central systems such as human resources and financial systems. The BPR and the QPR are more detailed reviews of the programs prepared by managers. For 2000-01, the Agency planned to complete BPRs or QPRs for 21 of 49 statistical programs; it completed nine (five BPRs and four QPRs). Of the remaining 12, 10 (six BPRs and four QPRs) are being finalized, and 2 were postponed until 2001-02. According to Statistics Canada's current schedule, reviews of approximately 40 percent of programs will be completed by the end of 2001-02.

Statistics Canada's self-assessment of four surveys

12.246 Wider application of self-assessment technique. In 1999 we recommended that Statistics Canada review the potential for a wider application of an enhanced self-assessment technique as one component of the formal processes it uses to assess quality. Statistics Canada has acted on this recommendation. The BPR and QPR of the Integrated Program Reporting process incorporate self-assessment techniques similar to those used as part of the Agency's 1998 self-assessment process.

12.247 Self-assessment of Uniform Crime Reporting Survey. In 1999 we observed that the recommendations of the Agency's self-assessment of the Uniform Crime Reporting Survey (UCR) were more important than the self-assessment suggested and deserved the attention of senior management. This survey is currently in a period of transition. Prior to 1988, the UCR survey consisted of aggregated data of incidents of crime (UCR1). Since then, the Canadian Centre for Justice Statistics has been working closely with police forces and vendors of police records management systems to enable forces to transmit incident-based data (UCR2). National coverage of UCR2 is at approximately 50 percent. As a result, UCR2 does not constitute a representative survey of Canada. The self-assessment of the UCR survey done as part of the 1999 audit recommended specific analyses to determine the impact of imputation of missing data. Statistics Canada has acted on this recommendation and developed monthly and annual imputation rates.

12.248 Implementation of UCR2 has depended largely on accessibility of information technology. Affordability of computer systems and amalgamation of jurisdictions have accelerated the implementation of UCR2. There are two large forces, the OPP and the RCMP, and a number of very small forces that are not yet included in the UCR2. Statistics Canada told us that inclusion of the OPP and the RCMP in the UCR2 will add approximately 40 percent to the total survey coverage. The OPP has implemented a UCR2-compliant records management system in most of its jurisdictions and is currently sending test data to Statistics Canada to verify the system. The OPP personnel responsible for implementing the new system stated that yearly UCR2 data from all OPP jurisdictions should be available for 2002. The RCMP expects to implement a UCR2-compliant system over the next five years.

Reporting performance to Parliament

12.249 In 1999 we recommended that Statistics Canada improve the coverage and content of information on statistical quality in its annual Performance Report to Parliament by drawing on quality information available from internal assessment and reporting systems. We found that Statistics Canada's 1999-2000 Performance Report outlines the components of quality defined in its Quality Assurance Framework and uses this framework to discuss the quality of selected programs. The Report systematically reviews several products for relevance, accuracy, timeliness, accessibility, interpretability, and coherence. These six dimensions of information quality define a product's "Fitness for Use," one of the four aspects of performance described in the Framework.

Informing users about data quality and methodology

12.250 In 1999 we recommended that Statistics Canada ensure that its Policy on Informing Users of Data Quality and Methodology is applied consistently across products and dissemination media. We also recommended that, in informing users of data quality and methodology, Statistics Canada use clear and standardized headings in all printed and electronic documents for sections that provide information on data quality and methodology or that refer the user to a source where such information can be found. We further recommended that the Agency establish a flexible disclosure framework to define the contents of the data quality and methodology section in individual statistical products and related documents, to ensure that the intent of the Policy is met. The framework should allow for variations in the depth and breadth of the information provided, depending on the program, product, or medium of dissemination.

12.251 Our follow-up observed that Statistics Canada has taken steps to address these recommendations. It has created an Integrated Meta Database (IMDB) that is intended to serve as its flexible disclosure framework. Data from existing metadata repositories were converted into the IMDB in November 2000, and active follow-up began January 2001. Between January and the end of July, 46 percent of the 366 active surveys were updated to conform with the Policy on Informing Users of Data Quality and Methodology. This represents 78 percent of the monthly surveys, 50 percent of the quarterly surveys, and 35 percent of the annual surveys. Also, the IMDB has been updated to satisfy the Policy for nine out of the ten mission-critical surveys. Statistics Canada expects to have updated IMBD information on 95 percent of surveys by the end of 2001-02, with the exclusion of information on concepts and variables. Information on concepts and variables measured will be added to the IMDB and upgraded during a later phase of the development.

12.252 The Agency's Methods and Standards Committee revised the Policy on Informing Users of Data Quality and Methodology to include the use of standardized headings covering quality and methodology. The IMDB will enable Statistics Canada to use clear and standardized headings in all printed and electronic documents for sections that provide information on data quality and methodology. An internal audit on the extent to which statistical products comply with the Policy on Informing Users of Data Quality and Methodology is currently being planned and is expected to be completed by 2001-02.

Integrated approach to managing quality

12.253 In 1999 we recommended that Statistics Canada make its quality-related policies, guidelines, and systems more coherent and cohesive by reconciling differences in definitions of quality and co-ordinating the requirements for internal documentation and external reporting. Statistics Canada has reconciled differences in definitions of quality by consistently applying its six-dimensional definition of product quality, which assesses the product's Fitness for Use. As mentioned earlier, Statistics Canada is incorporating the definitions of quality from the Quality Assurance Framework into the IMDB specifically for the purposes of co-ordinating the internal documentation and external reporting.

12.254 We also recommended that Statistics Canada co-ordinate the development of the Integrated Meta Database with other quality-related initiatives and take steps to ensure the ongoing completeness and reliability of the database. The Agency is currently integrating the IMDB with other quality-related initiatives. Specifically, when performing the Annual Consolidation of Indicators (ACI), the Corporate Planning Division plans to generate assessments of quality that are standardized across programs, using the data stored in the IMDB.

12.255 We further recommended that Statistics Canada assign to a corporate focal point the responsibility for promoting an integrated, consistent approach to developing and implementing quality-related initiatives throughout the Agency, including the assessment and reporting of quality. Statistics Canada has made organizational changes to create corporate focal points of responsibility for quality-related initiatives. At the time of our 1999 audit, the responsibility for evaluations rested primarily with managers of individual programs. Since then, several changes have been made: the Corporate Planning Division is responsible for all ACIs; the mandate of the Methods and Standards Committee has been revised to make it responsible for ensuring that the Agency's quality management processes are appropriate, coherent, and used; and all Quadrennial Program Reviews are presented to the Program Evaluation Committee.

Agriculture Portfolio—User Charges (1999, Chapter 11)

Assistant Auditor General: Doug Timmins
Principal: Ellen Shillabeer

Background

12.256 Our 1999 audit looked at the management of user charges in three organizations in the Agriculture Portfolio—Agriculture and Agri-Food Canada (the Department), the Canadian Grain Commission, and the Canadian Food Inspection Agency. Within the Department, we examined user charges in the Prairie Farm Rehabilitation Administration, Market and Industry Services Branch, and the National Income Stabilization Account program of Policy Branch. We looked at only those fees for services provided to users outside the government.

12.257 At the time of our audit, these organizations had made progress toward meeting the requirements of the government's 1997 policy on user charges, but they were losing momentum, and user charges were not being updated as programs changed. The organizations did not meet all of the requirements of the policy. They needed to improve their management of user charges in a number of areas, including costing their services, assessing the impact of fees, establishing formal appeal and dispute resolution processes, and integrating user charges into their strategic planning.

12.258 As noted in the audit, the Minister of Agriculture announced a moratorium on new or increased user charges in November 1996. In the spring of 2000, the Minister extended the moratorium for all three organizations—to the end of 2002 for Agriculture and Agri-Food Canada and the Canadian Food Inspection Agency and 2004 for the Canadian Grain Commission. In their response to our chapter and in subsequent parliamentary committee hearings, the organizations committed to taking action, within the limits imposed by this moratorium.

12.259 Parliamentarians have continued to express interest in the issue of user charges, and the agriculture committee of the House of Commons has held hearings on this topic. The House of Commons Standing Committee on Finance has also examined cost recovery issues and originally tabled its report in the House in June 2000.

Focus of the follow-up

12.260 We conducted our follow-up by reviewing status reports, provided by each of the organizations, which describe actions they took to address our recommendations. This was supplemented by further review of additional documentation and by interviews with departmental officials. We also reviewed the government response to the Report of the Standing Committee on Finance.

Conclusion

12.261 The organizations have made limited progress in the two years since our audit—in part, because of the government's continuing moratorium on new or increased fees. They also hesitated to take action before the Treasury Board Secretariat finished a review of the government's Cost Recovery and Charging Policy. However, even in areas where action was possible, such as improving their understanding of the costs of providing their services, we found limited progress.

12.262 We will continue to monitor the actions taken by these organizations in coming years to determine their progress on our recommendations.

Observations

Barriers to taking action

12.263 All three organizations indicated that the continuing moratorium has had an impact on their ability to make changes in their management of user charges. For example, since the moratorium does not allow for new or increased fees, the organizations have not had the opportunity to demonstrate an improvement in the information they provide to the public on the likely impact of proposed fees. The moratorium has also affected other aspects of the management of user charges, such as the responsiveness of fee structures, the inclusion of user charges in strategic planning, and appeal and dispute resolution processes. However, there are other areas where these organizations could be doing more to prepare for the eventual end of the moratorium; we explain these areas further in the following sections.

12.264 In addition to the moratorium, the organizations cited a second barrier to taking action; they are awaiting a revised policy on user charges from the Treasury Board Secretariat. The Secretariat's review of the government's 1997 Cost Recovery and Charging Policy was originally scheduled to be completed before 2000, but the review of the policy and any changes to it are not expected to be in place before early 2002. Departmental officials told us they are reluctant to implement changes that could be contrary to the new policy.

Analysis of current situation lacking

12.265 In our 1999 audit, we considered the moratorium period as an opportunity for these three organizations to reassess their current position and plan improvements. They have done this, but only to a very limited extent. While the moratorium prevents them from introducing new fees and severely limits their ability to change existing fees, they could have better identified changes that are needed to prepare for when the moratorium is lifted.

12.266 In its published response to our original audit, the Canadian Food Inspection Agency committed to focussing its efforts on simplifying its fees. This has not happened and the number of fees remains at about 1,600, although the Agency has reiterated its intention to simplify its fee structure. Also, while the Agency continues to acknowledge that there are known problems with its user charges, such as different commodity groups paying different proportions of the costs of providing similar services, the Agency has not conducted any formal analysis to determine how the fees need to be changed. It has developed a draft cost recovery policy and strategic plan that outline the Agency's plans for costing and consultations before resuming consultations on user charges in 2003. The Agency has also hired an economist to conduct economic impact assessments. However, it is too early to determine the results of these efforts.

12.267 We noted in our audit that the moratorium had prevented the Canadian Grain Commission from taking action on the recommendations of its 1999 program review to better align its fee structure with its underlying costs. Since the moratorium has continued, this is still the case. However, the government has taken this situation into account and approved $83 million in additional funding for the Canadian Grain Commission to ensure that their mandatory fees remain frozen until 2004.

12.268 One branch of Agriculture and Agri-Food Canada has made progress despite the moratorium. The Prairie Farm Rehabilitation Administration has completed relevant studies for two of its major programs. The first study, focussing on the southwest Saskatchewan irrigation projects, determined the proportion of benefit the public receives from the service and the portion of the private benefit that should be recovered from the users. The second study developed an economic framework for cost recovery in the branch's community pastures program, including the optimum cost recovery level for this program.

Minor improvement in costing methods

12.269 In 1999 we emphasized the need for solid costing practices in order to know whether user charges are set at appropriate levels. All three organizations have made significant changes to their accounting systems, which now have the capacity to track and implement more sophisticated costing methods. The Canadian Food Inspection Agency indicated that it has done some work on improving its resource allocation tracking system and activity tracking system. The Canadian Grain Commission and Agriculture and Agri-Food Canada continue to work on their systems. However, efforts to date have been focussed on getting their new systems up and running, and the ability of these new systems to capture and analyze costing information has not been fully activated. More effort is needed in this area.

Consultations stalled between the Canadian Food Inspection Agency and users

12.270 Our 1999 audit found that, in general, the consultation process between the Canadian Food Inspection Agency and users had improved over the previous five years. During the moratorium the Agency has continued to hold consultations with stakeholders to share information on program developments; however, consultations on user charges have stalled. Thorough consultations are required for progress in areas such as identifying changes needed in fee structures and developing further service standards; yet the Agency has no plans to conduct in-depth, sector-specific consultations with industry on user charges before 2003. Both the Department's Prairie Farm Rehabilitation Administration and the National Income Stabilization Account program have continued to hold regular consultation sessions. The Canadian Grain Commission has also held annual meetings with its stakeholders.

Recent work on cumulative impact

12.271 Led by Agriculture and Agri-Food Canada, the organizations have made good progress in addressing two recommendations on the cumulative impact of user charges. In 1998 the Department examined the impact of selected federal cost recovery initiatives on the agri-food sector. Building on this work, the Department is now completing a study that outlines an approach for assessing the cumulative impact of the various proposed fees for programs in the agriculture and agri-food sector.

Little progress on service standards

12.272 In 1999 we noted that, with few exceptions, the organizations did not use service standards and/or measures of client satisfaction as management tools to assess the extent to which their services were meeting customer expectations. We found that little has changed in the past two years in all three organizations. Service standards are the exception, not the rule. And those standards that are in place are not being rolled up to an organizational level. This is an area that could be addressed during the moratorium.

Some work on guiding principles

12.273 In 1999 we highlighted the importance of guiding principles for establishing user charges—both to ensure consistency across an organization and to provide user groups with important information. This is an area where we expected progress. While both Agriculture and Agri-Food Canada and the Canadian Food Inspection Agency have begun to develop a framework and a strategic plan for the management of user charges, these initiatives are still in the draft stages and have not yet received senior management approval. The Canadian Grain Commission continues to apply the guiding principles it has had in place for some time, but has been unable to fully apply these principles to its mandatory services because of the moratorium.

12.274 Members of the three organizations agreed in August 2000 that the most effective way to approach the challenge of improving the management of user charges is to work together to develop guiding principles, a generic framework, and general guidelines for managers. Agriculture and Agri-Food Canada drafted a framework for the management of cost recovery that is intended to be applied by the three organizations. This approach recognizes that it is the same users (primarily farmers and industry in the agriculture and agri-food sector) who pay fees to the three organizations. While the Canadian Grain Commission and the Canadian Food Inspection Agency have, in principle, agreed to proceed with the development of this framework, progress has been slow while the Department awaits the results of the Treasury Board Secretariat policy review.

National Defence—Hazardous Materials: Managing Risks to Employees and the Environment (1999, Chapter 13)

Assistant Auditor General: David Rattray
Principal: Peter Kasurak

Background

12.275 In 1999 we examined how National Defence managed hazardous materials such as flammable substances, corrosive products, and ammunition. The Department used over 6,000 hazardous products and estimated that at least a quarter of its 80,700 full-time employees came into frequent contact with hazardous chemicals.

12.276 Although departmental policies, plans, and procedures at the national level indicated an awareness of the legal framework and best practices for hazardous materials management, implementation at the base and unit levels was inconsistent. We found shortcomings in compliance with laws and regulations, in application of audit protocols and methodologies, and in application of compliance management systems aimed at promoting continual improvement.

12.277 The Department responded positively to our recommendations and indicated it would take action to address the concerns raised in our audit.

Focus of the follow-up

12.278 The objective of our follow-up was to determine the status of our recommendations and identify actions taken and improvements made by the Department. We interviewed departmental officials and reviewed key documents and reports.

Conclusion

12.279 The Department has taken steps to address our concerns. It has developed action plans for each recommendation and has made some progress. However, it has carried out few recommendations fully, and it has revised and extended many of the completion dates for its action plans.

12.280 Personnel shortages have contributed to the slow progress. As well, the Department's main focus has been the preparation of its second sustainable development strategy and its environmental management system at the national level. Officials indicated that these two documents will provide much of the direction for departmental progress in addressing our concerns.

Observations

Information systems are improving

12.281 We recommended that National Defence provide its employees with safety and environmental information on hazardous materials, and that it implement inventory systems that contain standard information on hazardous materials.

12.282 The Department has taken steps to update and improve the information it provides to personnel. A project is under way to amalgamate workplace information management systems into a Web-based system that will also incorporate a centralized database for current and historical Material Safety Data Sheets for hazardous materials.

12.283 The project includes a cross-functional team to identify minimum standard requirements for the Web-based system. This system is intended to allow employees to access a preferred products library of less-hazardous products and to directly link them to the Departmental Inventory Management System for accurate and up-to-date inventory information.

12.284 Training courses for the Workplace Hazardous Materials Information System (WHMIS) have been updated and the tracking system has been standardized. These courses provide training on identifying and interpreting hazardous information and symbols. The new tracking system ties the training records to the Department's National Training Database, personnel files, unit employment record files, and Base Safety Officer files.

Slow progress in monitoring effluent

12.285 Our audit identified the need for an assessment of liquid effluent and air emissions at National Defence bases to ensure that hazardous materials were not contaminating the environment.

12.286 The Department's second sustainable development strategy, dated December 2000, commits the Department to issuing national guidance on effluent monitoring, developing and carrying out an effluent monitoring plan for storm sewers, and periodically testing discharges to municipal sewers for conformance with local standards. The strategy also commits the Department to developing a national air emissions strategy.

12.287 The Department participates in an interdepartmental committee that, among other things, is addressing a possible gap identified by National Defence related to federal and provincial air emission regulations.

12.288 The Department has monitored effluent at some bases through base-level funding or through a headquarters-based program, the Corporate Environmental Program (CEP). The testing that was conducted touches on some of our concerns and other monitoring programs that the Department is addressing.

12.289 The CEP, which funds a variety of environmental projects, will cease to exist at the end of the 2002-03 fiscal year. When this occurs, environmental projects and programs will be at risk of not succeeding unless they are appropriately funded through some other means. In an effort to help address this problem, the Department is conducting a study to evaluate future environmental risks and liabilities. It will then be able to develop strategies to mitigate or eliminate these risks and liabilities.

Compliance management remains weak

12.290 Our audit found widespread, frequent, and recurring instances of non-compliance with legal and policy requirements, and a lack of information on overall compliance with those requirements. In addition, the management systems to ensure compliance at the base and unit levels did not promote continual improvement. We recommended that the Department have consistent audit and inspection programs at headquarters, and consistent methodology for annual audits at the base level. We further recommended that bases and units establish performance measures for hazardous materials compliance that measure continual improvement toward a target of full compliance.

12.291 The Department has drafted a national-level environmental management system (EMS) based on the ISO 14001 environmental management system standard. The EMS contains principles that commit National Defence to continual improvement and to compliance with relevant legislation and regulations and with other requirements to which the department subscribes. The EMS also commits the Department to developing audit protocols and conducting audits on a three-year cycle at the command level.

12.292 The national level EMS serves as a guide for command-level environmental management systems, which in turn serves as a guide for base-level systems. The target date for the development and implementation of the first full cycle of environmental management systems across the Department is not until March 2004.

12.293 Core requirements for audits and inspections were not yet developed or included in the national-level EMS. Consequently, command-level and base-level audit and inspection protocols have not become standardized for comprehensiveness and consistency. Some officials noted that a key component of the audit and inspection process must include third party audits, whereby the audit team is made up of personnel from outside the group being audited. Third party audit results are considered less biased and more objective.

12.294 As part of the EMS process, organizations must identify their legal and other requirements. The Department's environmental policy, the Code of Environmental Stewardship, commits the Department to meeting and exceeding the letter and spirit of all federal environmental laws and, where appropriate, being compatible with municipal, provincial, territorial, and international standards. Our audit noted that National Defence did not define the term "where appropriate." Consequently, it was not clear in what circumstances the Department's actions would be compatible with municipal, provincial, territorial, and international standards.

12.295 The Department has only recently initiated efforts to develop and issue a policy that clarifies federal, provincial, and municipal regulations, and standards applicable to departmental activities on federal lands. The final policy is expected in the spring of 2002.

12.296 Although progress is slow in addressing our recommendations, the Department intends to establish a departmental definition of full compliance. This will provide the needed guidance to subordinate environmental management systems on the extent to which bases and wings are to meet provincial, municipal, and other requirements applicable in their jurisdictions.

Revenue Canada—Goods and Services Tax: Returns Processing and Audit (1999, Chapter 16)

Assistant Auditor General: Shahid Minto
Principal: Jamie Hood

Background

12.297 The Goods and Services Tax (GST) is a multi-stage sales tax levied on taxable goods and services. The GST applies on each exchange of taxable goods and services, from the initial producer or supplier through intermediate producer or supplier to consumers. To ensure that the value added at each stage of the supply chain is taxed only once, producers and suppliers are able to recover the GST paid on their purchases of business inputs through "input tax credits." The Harmonized Sales Tax (HST) is collected in three of the Atlantic provinces and is an integral part of the GST program. The objective of the 1999 audit was to assess the efficiency and effectiveness of Revenue Canada's GST returns processing and audit activities.

12.298 In 1999 we noted that Revenue Canada, now the Canada Customs and Revenue Agency (CCRA), had streamlined its GST processing operations at its Summerside Tax Centre and improved their timeliness. However, we were concerned that the standards for processing returns did not place enough emphasis on quality, accuracy, timeliness, and unit cost. Furthermore, we noted that better performance measures would help to manage interest costs and improve the service to registrants. At the time of our audit, these operations were being redistributed among other processing centres and we were concerned that efficiency gains could be lost.

12.299 We also noted that the automated procedures used to identify higher-risk GST refund requests for pre-payment review needed to be more discriminating. We recommended that Revenue Canada review its preliminary processing procedures and consider reviewing those returns that included payments. We also found that GST post-payment audits could be better planned and executed, that they took too long to complete, that audit staff were not adequately trained, and that performance could be improved with a better selection of registrants for audit. Despite the enhancements made to various systems, we were concerned about the accuracy of the analyses done to evaluate the pre- and post-payment audit processes.

Focus of the follow-up

12.300 Our follow-up consisted of reviewing the status report prepared by the Canada Customs and Revenue Agency on its progress to implement our 1999 recommendations, reviewing supporting documentation, and holding discussions with its officials.

Conclusion

12.301 The Agency has made satisfactory progress in addressing all of our recommendations, except in the GST audit program support area where much of the effort to completely address our recommendation lies in the future.

Observations

12.302 The processing of GST return and rebate claims consists of opening and sorting the mail, validating the claims for completeness, batching them by type of document, entering the relevant document information into a central processing system, and correcting errors. If the total GST charged by a registrant to its customers exceeds the input tax credits claimed, the registrant is expected to remit the difference to the Agency. This is known as a "debit return." If the input tax credits exceed the GST charged, the registrant can claim a refund from the Agency. This is called a "credit return." CCRA pays interest on claims that take more than 21 days to process. The Agency also receives claims for rebates of GST paid from, for example, visitors to Canada.

12.303 A number of automated validity checks are applied to GST credit returns and rebate claims to detect those that may not comply with legislation or that may warrant investigation for some other reason. Selected returns are electronically forwarded to local Tax Services Offices (TSOs). As part of the pre-payment audit process, an officer examines each item and decides whether to pay the amount claimed or to refer the claim to an auditor who conducts a more in-depth examination.

12.304 The Agency also does post-payment audits to find other kinds of noncompliance with the many requirements of the Excise Tax Act, and to find problems with refunds that have already been paid. Using computerized support systems, auditors in TSOs select, plan, and do GST audits that represent a high risk of non-compliance. Results of audits are reported in Agency systems.

Progress made in processing GST returns

12.305 In 1999 we were concerned that management was not taking full advantage of available information and that Revenue Canada had not updated its processing standards to include quality, accuracy, timeliness, and unit cost. We were also concerned that it had not done benchmarking activities with other large processing centres of other similar organizations to assess its performance standards, its service delivery options, and its technologies.

12.306 In the follow-up, we found that the Agency has taken steps to monitor its returns processing activities and has developed reports on its data capture and error correction activities. It also compares the efficiency and quality of those activities among Tax Centres. Standards have been established for the new GST Telefile service. Managers recognized the need for official performance indicators related to other aspects of processing such as unit cost, but these have not been developed yet. Processing activities have been benchmarked to compare efficiencies among Tax Centres, but benchmarking has not yet been broadened to include other similar organizations.

12.307 We were also concerned in 1999 that Revenue Canada was not monitoring and controlling the amount of its portion of interest charges. The amount of interest cost that has been avoided could be a valid performance indicator of processing efficiency and service to registrants.

12.308 During the follow-up, we found that the Agency has directed its managers to pay particular attention to transactions that need to be corrected and to process, as a priority, those credit returns that stand to earn interest. The Agency is promoting the Direct Deposit of refunds to reduce the time it takes for a payment to reach a registrant. The Telefile option that was introduced in April 2001 to enhance service to registrants was also put in place to reduce interest costs. The Agency is developing a new report that will differentiate between interest amounts due to slow processing and those paid as a result of audit activities. When completed, this report is expected to provide managers with the necessary information to help them monitor and control the amount of interest that they are responsible for.

Progress made in GST pre-payment audit program; debit returns continue to be examined in post-payment audit

12.309 In our 1999 audit, we were concerned about several aspects of Revenue Canada's pre-payment audit programs:

• Automated validity checks were inefficient because they were not particularly discriminating, and many returns chosen by the checks did not need to be reviewed.
• Debit returns were not being considered as part of the pre-payment audit process.
• Procedures in TSOs to allocate resources and manage the review and audit of refund claims were out of date.

12.310 In the follow-up, we found the following:

• The Agency has reviewed and revised its automated validity checks for credit returns. More changes for credit return validity checks are to be implemented in 2003 and 2004.
• The Agency is currently analyzing validity checks for rebates and changes should be implemented in phases between 2003 and 2005.
• The Agency has considered reviewing and auditing debit returns at the same time as it conducts pre-payment audits of credit returns. It has concluded that its audit selection procedures for post-payment audits are appropriate to select high-risk debit returns. These selection procedures are reviewed periodically.
• The Agency has established a new model for allocating resources to the audit program. The model identifies GST pre-payment audits as a program priority. New guidelines for managing GST pre-payment audits that require establishing dedicated GST/HST audit teams have been drafted and were being released at the completion of the follow-up.

Agency improved its GST post-payment audit

12.311 In 1999 we were concerned that Revenue Canada needed to continue to improve its audit selection systems and provide additional training to those people that use them to select post-payment audits. It also needed to enhance its efforts to reduce the duration of audits.

12.312 Our follow-up work revealed that the Agency had recently revised its Workload Development Guide to assist staff in the use of audit selection systems. The Agency was also working with Statistics Canada to improve standard industry classification codes to reflect more detailed and accurate descriptions of the major business activity. The GST/HST Enhanced Registration Review pilot project was initiated to improve the criteria used to identify potentially high-risk registrants early on in the process. The results of the project will be reviewed to ensure that the criteria are acceptable to all regions.

12.313 We also found that the Agency had reduced team sizes and had put in place specific accountabilities for TSOs to report every six months on their progress in meeting audit objectives. This report includes status reports on audits outstanding for more than six months. A focus group was also held and it made recommendations that were implemented in field offices. These included recommendations for better-defined goals and objectives, more auditor accountability, more attention to file selection, and auditor training matched to audit responsibilities. A pamphlet informing taxpayers and registrants of their responsibilities during audits was distributed to help reduce the time taken to complete them.

Change in approach to support GST audit programs

12.314 In 1999 we were concerned that timely information was not available to effectively manage the pre-payment and post-payment audit programs. It was hard to determine which activities were working well and which ones needed improvement. It was also difficult to analyze variances in audit projects.

12.315 We found that the Agency had abandoned the "power play cubes" prototype it was hoping to use to track audits as a replacement to the 1999 systems. More recently, the older reporting systems have been enhanced to include more pre- and post-payment audits. In addition, several ad hoc reports are generated by the Agency to aid in managing the credit return workload. Although Agency management thinks that the current reports are adequate for decision making, the Agency is designing a replacement reporting facility to be implemented in 2001-02. The Agency also plans to implement report card monitoring in 2001-02 to improve the reporting of actual performance results against plan. Progress has been achieved, but much of the effort to completely address our recommendation lies in the future.

Preparedness for Year 2000: Final Preparation (1999, Chapter 25)

Assistant Auditor General: Doug Timmins
Principal: Nancy Cheng

Background

12.316 In the late 1990s, public and private sectors around the world were preoccupied with the threat posed by the Year 2000 computer problem. The Year 2000 problem refers to the potential for system errors, malfunction, or failure caused by the past practice of representing the year with a two-digit code.

12.317 Our third audit on Year 2000—reported in 1999, Chapter 25—examined the progress the government was making in its final preparations. The audit focussed on seven mission-critical functions and the results of remediation and testing at six organizations responsible for them. We also examined several horizontal issues, such as the government's communication strategy and departmental contingency planning. We commented on the significant progress made by the government and offered recommendations for action to be taken beyond December 1999.

Focus of the follow-up

12.318 Our 2001 follow-up work consisted primarily of interviews with staff at the Treasury Board Secretariat and a review of documents at the Secretariat relating to Year 2000. In addition to reports of the Secretariat's Year 2000 project office, we also reviewed project completion reports for the six departments and agencies responsible for the seven mission-critical functions that we audited in 1999.

Conclusion

12.319 The follow-up showed that significant efforts were made in the period leading up to and beyond the millennium rollover to avoid system problems in 2000. Further, the government has taken steps to document lessons learned from preparing for this government-wide threat. However, limited progress has been made to ensure that departments and agencies are aware of and in compliance with government date standards, and measures to address this issue have yet to be developed and implemented. With respect to valuable information bases, steps are being taken to require departments and agencies to update and maintain them; the policies, standards, and practices are not expected to be finalized until 2003.

Observations

12.320 After 1 January 2000, the Treasury Board Secretariat led various projects to capture lessons learned from Year 2000. Some of the more significant projects are as follows:

• Departmental Project Completion reports were written and/or updated in winter 2000. Most of the six organizations that we audited in 1999 indicated to the Secretariat that they plan to maintain critical systems inventories and/or business continuity plans.
• Final business readiness interviews were conducted in spring 2000 with departments and agencies responsible for government-wide mission-critical functions. These interviews dealt with issues such as Year 2000 incident reports, project completion, harvesting benefits, the total incremental cost of Year 2000 remediation and related activities, lessons learned, and adherence to the date standards.
• A Post Implementation Assessment Report was completed in July 2000. Its purpose was to assess the management of government operations for Year 2000 issues, and it included an appendix on lessons learned.
• A Year 2000 Learning Model identifying key elements for the successful implementation of a large-scale horizontal initiative was prepared. Presentations were made to departments and agencies in fall 2000.

Government date standards

12.321 In 1999 we recommended that the Treasury Board Secretariat ensure that all departments and agencies be made aware of the existing government date standards and that departmental plans be put in place to comply with the date standards in future upgrades and maintenance of departmental systems. Standardized coding for dates provides better efficiency in data exchange between government agencies or with non-government organizations. Furthermore, the use of a four-digit year code, as required in the date standards, alleviates the need for updating and maintaining date codes that involve only two digits. Many departments and agencies used the "windowing technique" and retained a two-digit year code in their Year 2000 remediation strategy.

12.322 Government date standards have been adopted since 1988 and form part of the Treasury Board Information Technology (TBIT) Standards program. During the follow-up, we noted that there had been limited activities in recent years to update, disseminate, and monitor the use of those date standards. The Secretariat advised us that a standards manager is now in place to revise the TBIT standards to prepare for Government On-Line, a government-wide initiative to put major government programs and services on-line. The Secretariat indicated that the revised TBIT standards will include date standards with a four-digit year code.

Maintenance of Year 2000 information bases

12.323 The Year 2000 computer problem provided departments and agencies with an opportunity to better understand their critical information systems and to properly document them. Contingency planning also became a key activity in preparing for Year 2000. In 1999 we recommended that the Treasury Board Secretariat consider requiring departments and agencies to maintain and update valuable information bases that were developed to respond to Year 2000.

12.324 The Treasury Board Secretariat advised us that the Government Security Policy was under revision and that information technology security will form a key part of the new policy. In reviewing the draft documents, we noted that departments and agencies will be required to plan for business continuity. According to the Secretariat, departments and agencies will need to review their mission-critical functions and update the list of related systems as part of that planning. The Treasury Board Secretariat expects standards pursuant to the Government Security Policy to be finalized in 2003.

12.325 In preparing for Year 2000, the National Contingency Planning Group identified and defined elements of Canada's critical infrastructure. This information now rests with the Office of Critical Infrastructure Protection and Emergency Preparedness (OCIPEP), an agency created in February 2001 to develop and implement a comprehensive approach for protecting Canada's critical infrastructure.

Vigilance over potential Year 2000 pitfalls

12.326 In 1999 we noted the possibility of Year 2000 causing problems with systems and operations beyond the rollover date of 1 January 2000. We recommended that the government remain vigilant beyond 1 January 2000 to guard against potential Year 2000 pitfalls.

12.327 The government responded by keeping the Treasury Board Secretariat Year 2000 project office open until 31 March 2000. A post-Year 2000 office remained in place until 1 December 2000. The departments and agencies were made aware of all potentially sensitive dates in 2000, and measures were in place to address any potential incidents. According to the Secretariat, no incidents have been reported.

National Defence—The Proper Conduct of Public Business (1999, Chapter 26)

Assistant Auditor General: David Rattray
Principal: Peter Kasurak

Background

12.328 During the 1990s National Defence suffered from several highly publicized breakdowns in management and leadership conduct. Our Office received serious complaints of abuse of resources by Defence staff. Therefore, we undertook an audit of the Department's support of ethical conduct; its maintenance of effective internal controls, including internal audit; and its follow-up on complaints.

12.329 We identified findings that were not significant enough to bring to the attention of Parliament but that warranted action by the Department. In January 2000, we reported these matters in a management letter addressed to the Deputy Minister.

12.330 In our 1999 audit we reported that, in response to a growing need to reemphasize ethical decision making and integrity, the Department had developed a Defence Ethics Program. Overall, we found that the Department had been slow in carrying out the Program. We concluded that, unless more leadership was shown from the top, it was unlikely that a uniform approach would emerge and be carried out consistently throughout the Department and the Canadian Forces.

12.331 We also found weaknesses in departmental control systems, including internal controls, internal audit, and the military police. Audit resources assigned to the military services had declined and systematic risk analysis was not used to apply the remaining resources to the highest priorities. Managers had become concerned that the risks they were taking were too high. Local management was concerned about weaknesses in financial management systems. Departmental audits revealed an increasing risk of fraud and abuse. A case of widespread systematic abuse—secret commissions and kickbacks for diesel fuel purchases—had escaped detection.

12.332 We examined how the Department responded to complaints directed to senior management at National Defence Headquarters. We found that allegations of abuse had not always been dealt with adequately. Corrective action had been inconsistent and remedial action was slow. We recommended that the Department track all complaints and ensure that they be resolved expeditiously.

12.333 Findings reported in our management letter showed lapses in monitoring and control by the Department and the Canadian Forces. The management letter addressed values and ethics programs, environmental allowances, and provision of official residences, furniture, and hospitality and entertainment. The principal findings were the following:

• Errors were detected in reporting on the progress of the implementation of the Statement of Defence Ethics to the Minister's Monitoring Committee.
• Inconsistencies were noted in the use of official residences by senior military officials, including inadequate requirement for some residences, occupancy by staff other than those justified in the original requirement, questionable requirements based on representational duties that could be performed at other military facilities, and inadequate justification for rent reductions.
• High maintenance costs and lack of maintenance standards were found for official residences at Norfolk, Virginia and Colorado Springs, Colorado.
• Hospitality expenditures were inconsistent with Treasury Board policy on such events as holiday parties and levees.
• Pay for hazardous duty or adverse environmental conditions was awarded to staff that were not eligible for it.

Focus of the follow-up

12.334 We followed up on the chapter recommendations and those made in the management letter. We reviewed official documentation and interviewed departmental officials on the related issues.

Conclusion

12.335 The Department has made progress in carrying out the chapter's recommendations. However, the Defence Ethics Program—a key response to ethical problems—has still not been fully implemented seven years after it was launched by senior officials in 1994. Little progress has been made in carrying out the recommendations addressed to the Deputy Minister in the management letter.

Observations

Implementation of Defence Ethics Program by headquarters groups remains slow

12.336 Our 1999 audit recommended that the Department implement the Defence Ethics Program as quickly as possible. We found that progress continues to be slow. The development of the Defence Ethics Handbook has been completed and a final version has been distributed electronically throughout the Department. A formal Defence Ethics policy was promulgated only in June 2001. However, program implementation remains a concern. Two years after the tabling of our chapter to Parliament, only 3 of the 13 National Defence headquarters groups reviewed have developed acceptable plans to carry out the Ethics Program. Officials told us that they intend to rewrite these plans by November 2001.

12.337 In 1999 we found that 4 of 10 headquarters group ethics co-ordinators were reporting to middle rather than senior management and that 3 of 10 were at a more junior rank than recommended by the Ethics Program. Our follow-up found that National Defence has in many cases separated the coordinator role into two jobs: Ethics Advisory Board member and ethics coordinator. Although most Ethics Advisory Board members are at a senior level, in 7 of 13 cases the co-ordinator is still relatively junior in rank. Several co-ordinators also told us they did not have direct access to the most senior manager of their group. We are concerned that inconsistencies in separating the two roles could weaken the position of ethics co-ordinator.

12.338 The Chief Review Services has not produced program performance reporting instructions for departmental business plans. Departmental officials are attempting to develop a tool to assess the effectiveness of program implementation to complement the Baseline Survey model already in use. Development of the tool has proven to be more difficult than expected as few organizations have attempted to measure the progress made by value-based ethics programs. Nevertheless, officials anticipate completion by 2002.

12.339 The Defence Ethics Program is being slowly integrated into formal human resource management systems. We found that the Canadian Forces Personnel Appraisal System presently incorporates an assessment factor titled "Ethics and Values" for colonels and general officers. The new appraisal system for non-commissioned members and officers up to lieutenant-colonel still does not include ethics as a specific element. Officials told us that ethics would be included in the next version of the appraisal to be implemented in 2002.

12.340 The Department has undertaken additional initiatives on defence ethics. A baseline survey conducted in 1999 found that fairness was a significant issue in the Department. Discussions by National Defence staff on addressing this issue in parallel with Quality of Life activities are under way. The Department has established an ethics newsgroup and chat room to provide a forum for all staff to discuss ethical issues. It has also held a Department-wide ethics week to focus on dialogue and solicit feedback and comment on issues of concern.

Steps have been taken to address risk

12.341 The environment in which National Defence operated in 1999 was rapidly changing, and it was essential that the Department identify, manage, and control risk. We therefore recommended that risk assessments be conducted and audit resources assigned commensurate with the risks identified. We also recommended that the audits by the Chief Review Services Branch and those by the three armed services be coordinated. The Chief Review Services has developed a comprehensive risk assessment in its 2000-01 plan in co-operation with headquarters groups and the three services. The Branch has also launched a major audit of the management and control of local funds to increase audit presence in the field and has developed a fraud risk management strategy. The Audit and Evaluation Committee has tasked the Chief Review Services to develop direction to coordinate audit work of the Branch and the three services.

A strategy is being developed to address fraud and abuse of resources

12.342 Management cannot rely solely on internal audit as a control. Audit only validates the effectiveness of other controls. Our 1999 audit recommended that the Department reinstate its management checks of compliance, consistent with assessed risk, to reduce the likelihood of fraud and abuse of resources.

12.343 The Department has implemented an initiative to revitalize finance, including clarification of comptrollers' roles and responsibilities. The goal is to ensure that there are sound financial management controls and management practices at all levels of the organization. Financial policies and procedures are being developed or updated. The Department has a strategy to communicate new policies and procedures to financial administrators before they are issued and intends to follow up on their application. There is also an increased focus on training related to financial policy and the new financial information systems as well as on the development of reference material. Commands and groups have also increased their local training efforts in the areas of contracting, verification of accounts payable, and the use of acquisitions cards.

Military Police independence has improved

12.344 In response to our audit, the Department has done a great deal of work to strengthen the training of commanders to ensure the independence of the Military Police. Presiding officers are now trained and certified as qualified to perform their duties in the administration of the Code of Service Discipline. Military Police reform is being adopted. The Military Police Professional Code of Conduct and the Occupational Structure Implementation Plan (OSIP) will ensure that the highest professional standards are maintained in the Military Police Branch. The Department expects that the implementation of the OSIP will commence by September 2001.

12.345 Other recommendations and observations from the report of the Military Police Services Review Group have been carried out except for the implementation of a new Security and Military Police Information System. The first phase of this project is under way and is scheduled to be completed in early 2002. The second phase is expected to be in place by September 2003.

Department takes action on complaints more quickly

12.346 In 1999 we reviewed complaints received by our Office and referred to the Department since 1995. We also reviewed complaints that the Deputy Minister and the Chief of the Defence Staff had assigned to the Directorate of Special Examinations and Inquiries since its inception in 1992. Some complaints took two to three years to investigate. We recommended that the Department ensure that all complaints be followed up promptly and tracked, with the objective of completing them in a year.

12.347 We followed up cases begun after the completion of our audit to the end of June 2001. We found that the Chief Review Services had reduced the time taken to follow up on complaints. Staff completed 16 cases within four months on average. Another 16 cases are still open but only four have been open for more than a year. Two cases have been transferred to other agencies for further investigation.

12.348 The Chief Review Services developed various tools to ensure that proper follow-ups are carried out. Requests for special investigations are now tracked from start-up to closure, and it is now easier to verify the state of the file. A guide for the conduct of special examinations and inquiries has also been developed to provide guidance to staff.

12.349 We also recommended that the Chief Review Services follow up on corrective and remedial action and report the findings to the Deputy Minister. We found that when cases are closed, special investigation staff prepare a checklist that must be signed off by both their Director and the Chief Review Services. This checklist includes recommendations on whether follow-up action will be required.

Action on recommendations in our management letter is incomplete

12.350 Our management letter addressed to the Deputy Minister made recommendations on the use of public funds and on ethical issues.

12.351 We recommended that the Department clarify the interpretation of the hospitality policy with the Treasury Board Secretariat. We also recommended that the Department develop clear policy and ensure compliance with the intent of hospitality and entertainment policies and regulations. The Deputy Minister and the Chief of the Defence Staff have issued direction on hospitality, especially that offered inside Canada. Officials are working with the Treasury Board Secretariat and other departments to update hospitality policy for outside Canada.

12.352 In regard to our concerns about the inadequate management of official residences, staff have developed a draft policy for consideration by senior management in October 2001. The policy, if adopted, would be a good step forward. However, significant concerns about the draft policy remain. Criteria for the identification of requirements for residences appear vague and the rationale supporting the proposed retention of some residences rather than others is unclear. In addition, the Department may allow commanding officers to continue to pay for improvements and maintenance in excess of standards from their base operating budgets. We note that the Department estimates that official residences have cost much more than other base housing to maintain. An accountability structure is required that separates decision-making and expenditure authority from potential occupants of official residences.

12.353 We recommended that a central program authority be appointed for environmental allowances to clarify accountability and the lack of record keeping. An Allowance Review Project was established in January 2001 and the new policy is planned to be completed in the fall of 2002.

12.354 We also observed errors in reports to the Minister arising from staff mixing up the Canadian Forces Ethos Statement and the Defence Ethics Statement. However, the Department has taken steps to clarify the two statements and is working to integrate them further. Officials told us that progress reports are now accurate. The Canadian Forces are continuing to develop a complete concept of military ethos as part of their professional development initiatives for officers and non-commissioned members.

National Defence—Alternative Service Delivery (1999, Chapter 27)

Assistant Auditor General: David Rattray
Principal: Peter Kasurak

Background

12.355 In 1999 we audited the Alternative Service Delivery (ASD) program at National Defence; the program seeks to provide new and better ways of providing government services. Our audit objective was to determine how well National Defence was managing its ASD program and how well it had progressed toward ensuring that non-core support services were delivered to core defence activities in the most cost-effective way. We focussed on the strategic framework, the analytical framework (business case analysis), human resource management, and the contracting aspects of the program. We audited 14 completed projects initiated between 1994 and 1996.

12.356 The Department had not achieved the savings it expected. It had reduced its annual savings projections from $200 million by 1999 to $175 million by 2004. Many of the business case analyses for the 14 projects were poorly done, and personnel appeared to lack the necessary skills to undertake analyses. In addition, the government lacked an adequate framework for "partnering" with the private sector and contracting out large service programs. Further, inflexible contract arrangements in two cases resulted in payments for unused training capacity.

12.357 The Standing Committee on Public Accounts held hearings on our Report and made recommendations on improving business case analysis and developing the necessary skills and training to manage the ASD program, including the application of its management framework. The Committee recommended the development of a culture that favours an awareness of costs and a more businesslike approach. It further recommended that the Department include its ASD plans and savings targets in its annual Report on Plans and Priorities beginning in 2001-02, and that it report progress in its Performance Report beginning with the period ending 31 March 2002. Finally, the Committee recommended that the Department resolve with the Treasury Board Secretariat any remaining substantive issues related to implementation of the Financial Information Strategy and that it use the financial data to guide its ASD program.

Focus of the follow-up

12.358 We followed up on our recommendations and those made by the Public Accounts Committee. We also interviewed the National Defence Headquarters managers responsible for the ASD program, reviewed documentation, and communicated with both the Treasury Board Secretariat and Public Works and Government Services Canada. In addition, we reviewed a consultant's project report entitled the ASD Capacity Check (May 2001). The Department had initiated this project to assess the capabilities needed to deliver ASD within National Defence, to properly situate ASD within a strategic change perspective, and to ensure that ASD is clearly oriented to optimizing combat capability.

12.359 We did not review the two ASD projects begun since our original audit. Nor did we review the NATO Flying Training in Canada project, because of the ongoing implementation process.

Conclusion

12.360 The Alternative Service Delivery program will no longer be identified as a separate program. Essentially the Department intends to integrate ASD into a new program of continuous improvement. This is in part due to the negative view of ASD held by many managers and staff, the increased emphasis on performance measurement, and the need to apply continuous improvement to all departmental activities. This program will continue to emphasize the goal of developing efficient and effective programs and services. The Department has taken some action on almost all of our recommendations.

12.361 We were unable to assess our recommendation directed to the Treasury Board Secretariat. The Secretariat has yet to complete its work on the provision of guidance and training for large multi-year service contracts, including the key issues of how competition is to be addressed in situations where long-term "partnering" would be beneficial to the government.

Observations

ASD program stalls

12.362 Since we completed our original audit, only two new ASD projects have been introduced. Our interviews and the ASD Capacity Check suggest several reasons for this. The Capacity Check indicates that managers are neither committed to nor supportive of the ASD program because they feel the process is too burdensome. They are apt to avoid the process altogether. These negative views probably developed because of the threat of job loss brought on by the program's strong initial focus on cost savings and contracting out. Officials informed us that there are numerous "ASD-like" activities occurring throughout the Department, but managers simply do not want to become engaged in the full ASD process. Senior management now realizes that initial expectations were too high, the time frame for change was underestimated, and promised savings have not materialized.

ASD to be part of broader continuous improvement program

12.363 National Defence is changing its approach to ASD. The Department intends to integrate the program within its continuous improvement strategy, and ASD will no longer be identified as a separate program. Continuous improvement will apply to all activities and eventually will include the identification of service levels and costs of all services. It will also include an increased emphasis on performance measures and costs and on the success of managers in reaching pre-agreed targets. As in the ASD program, business case analysis and many of the techniques for determining the most efficient and effective way of delivering services will remain applicable. The Framework for Strategic Change within which to pursue ASD initiatives was approved by senior management in June 2001.

12.364 Although the ASD program will soon cease to exist in the form we originally audited, many of our recommendations and those of the Public Accounts Committee remain relevant. The Department's actions toward implementing these recommendations are assessed in the following paragraphs.

ASD training is now available

12.365 Both the Public Accounts Committee and the Auditor General recommended that training be provided for people responsible for evaluating and implementing ASD projects, particularly those skills associated with business case analysis.

12.366 The Canadian Forces Management Development School has developed new training modules on ASD methodology, business case analysis, and various other ASD topics. In addition, the chapter on business case analysis in the Department's costing handbook has been revised and a best practice publication on business case analysis is planned. As well, a Base Service Index is being developed that will provide standards for base services and will eventually be used to establish costs and set priorities for resource allocation.

Human resource policy established for employees affected by ASD

12.367 The Department has developed and issued a national policy on employment continuity that is to be applied to all ASD initiatives affecting 10 or more permanent employees. This policy was developed to ensure that all National Defence employees affected by ASD are treated in a fair and reasonable manner. The national policy addresses the weaknesses and inconsistencies of the case-by-case approaches used in earlier ASD initiatives.

No reference to ASD in Report on Plans and Priorities

12.368 Despite a Public Accounts Committee recommendation, the Department did not provide details in its 2001-02 annual Report on Plans and Priorities on the human, financial, and technical resources it is devoting to implementing its ASD program. Although the Department accepted the recommendation in principle, it states that ASD is to be considered part of the Department's overall continuous improvement plan. However, the Department does commit itself to reporting annually in its Performance Report on overall progress and lessons learned and applied from initiatives.

Departmental cultural change a work-in-progress

12.369 The Public Accounts Committee recommended that National Defence, under the leadership of the Deputy Minister, assign immediate priority to developing a departmental culture that favours an awareness of costs and a more businesslike approach. The Department accepted this recommendation in full and stated that it had given priority to four projects that underlie the Department's move to a more businesslike culture. Collectively these four projects are referred to as the Modern Management Agenda. The Modern Management Agenda is under the direct control of the Deputy Minister and includes the integration of management systems, the improved development and use of financial information, human resource management, and the integration of enterprise information systems.

12.370 The Department has also established an implementation plan addressing various elements of continuous improvement. This implementation plan includes the introduction of businesslike tools such as standards and performance measurement and an increased emphasis on business case analyses.

Work on procurement reform fails to address concerns about "partnering"

12.371 The Auditor General recommended that the Treasury Board Secretariat develop guidelines and training for large multi-year service contracts. We further recommended that such guidelines and training address the key issues of how competition is to be addressed in situations where long-term "partnering" would be beneficial to the government.

12.372 The Treasury Board Secretariat informed us that it is working with Public Works and Government Services Canada and National Defence on a Framework and Best Practices Guide on Long-Term Complex Contracts. As this guide has yet to be completed, we were unable to assess whether this recommendation has been addressed. These contracts must be implemented consistent with the Government Contracts Regulations and the government's contracting policy. Nevertheless we believe that guidelines and training for large multi-year service contracts addressing the issue of competition within a "partnering" context would be beneficial.

12.373 Public Works and Government Services Canada deals with the need for training related to multi-year contracts through its Benefits Driven Procurement program. However, the Department told us that Benefits Driven Procurement training does not address specifically how competition is to be addressed in situations where long-term "partnering" would be beneficial to the government. The Department's view is that it deals with competition in many ways, and its policy and procedural environment, supported by training, is applicable to all procurement strategies, not just Benefits Driven Procurement.

Department of Foreign Affairs and International Trade—Delivery of Capital Projects in Four Missions (1999, Chapter 31)

Assistant Auditor General: Shahid Minto
Principal: John Hitchinson

Background

12.374 In 1999 we noted that the number of increases in preliminary cost estimates and other problems identified by our audit seemed to indicate systemic weaknesses in the Department's planning of capital projects. We observed a lack of rigour in the Department's analysis of options in support of its recommendations to the Treasury Board. We also noted two instances where the Department had exceeded its delegated authority and Treasury Board approval was granted retroactively. We stated that the quality of reporting to the Treasury Board to explain project delays and budget increases needed to be improved.

Focus of the follow-up

12.375 Our follow-up consisted of a review of actions taken by the Department of Foreign Affairs and International Trade in response to our recommendations. It involved enquiry, discussion, and a review of various documents supplied by the Department to the end of August 2001.

Conclusion

12.376 The Department has taken action on all of our recommendations. In March 2001, the Deputy Minister submitted to the Treasury Board Secretariat a comprehensive report on steps being taken to improve the management of property in the Department. The Management Improvement Plan for the property program covers a wide range of issues, including those raised in our Report.

12.377 The Department has improved its planning capacity for capital projects and strengthened its guidelines for the planning of projects. By maintaining a sustained effort to implement the Management Improvement Plan as well as the revised guidelines, the Department can make real progress toward improving its property management.

Observations

Information to Treasury Board and use of authorities

12.378 Subsequent to our audit, the Physical Resources Bureau at the Department submitted an action plan to the Treasury Board Secretariat. The plan aimed at strengthening the rigour of the estimates of cost and the scope for preliminary project approval. The Bureau began revising project management procedures and incorporating risk management plans into Treasury Board submissions. The Bureau staff is also working with the Secretariat to produce a new Property Business Plan.

12.379 In correspondence with the Department, the Treasury Board Secretariat underlined the importance of submitting reliable estimates to obtain preliminary approval and remaining within delegated authorities. The Secretariat clarified that when significant scope changes occur, new approval must be obtained.

Guidelines for assessment of feasibility and definition of feasibility

12.380 Preliminary project approval is based on an assessment of feasibility. In March 2000, the Bureau issued new guidance on preparation of the assessment of feasibility report. The guidance addresses the points raised in our audit, including the need for proper analysis and costing of all reasonable alternatives that could potentially fulfil requirements. At the same time, the Bureau issued guidance on preparation of the definition of feasibility report, which contains the necessary analyses for obtaining effective project approval. The guidance also incorporates best management practices for this phase of the project planning.

12.381 We reviewed a few of the feasibility reports produced after the guidance had been issued. We observed that the Bureau staff had made efforts to follow the guidance but that in some instances required steps had not been completed or not documented.

12.382 Improvements in costing. The Bureau has arrangements with a number of qualified consultants to develop the costing of alternatives during the planning phase of a project. It has also provided training to its own staff.

12.383 In order to keep better track of costs during the implementation phase, the Department added a project management module to its management information system. The module captures data on overhead costs and time scheduling, and links the data to the accounting system. The Bureau has begun using this module for all its new projects.

12.384 Improvements in workflow and project control. In 2000, the Bureau undertook a broader systematic review of its project delivery processes and procedures. The review began the process toward obtaining ISO 9001 certification in 2002. The Bureau plans to establish and maintain a formal quality control system to ensure that delivered projects meet objectives, satisfy client expectations, and comply with standards. The Bureau is also investigating the possibility of having Public Works and Government Services Canada or other federal government partners take on some project management responsibilities.