Current Reports

2004-710 Audit of PWGSC’s National Accounts Verification Framework,
Final Report
2005-09-27

• Executive Summary
  
  
• 1 Introduction
  • 1.1 Authority for the Project
  • 1.2 Objective
  • 1.3 Scope
  • 1.4 Background
  
  
• 2 Findings
  • 2.1 Governance
  • 2.2 Procedures, Delivery Processes and Performance Standards
  • 2.3 Risk Management
  • 2.4 Reporting Mechanisms
  
  
• 3 Conclusion
• 4 Recommendations
• 5 Appendix
  • 5.1 Details on Audit Sample
  • 5.2 Risk Categorization Matrix

Executive Summary

Introduction

The objective of the audit was to assess the adequacy of PWGSC’s National Accounts Verification Framework (NAVF) for ensuring compliance with the requirements of Treasury Board Policy on Account Verification. The audit examined the NAVF’s governance, delivery processes and procedures, risk management practices and reporting mechanisms.

Primary responsibility for verifying individual accounts rests with officers who have the authority to confirm and certify entitlement pursuant to Financial Administration Act (FAA) section 34. Persons with this authority are responsible for the correctness of the payment requested and the account verification procedures performed. The NAVF was designed to enable financial service organizations throughout the department to provide assurance that FAA section 34 practices are adequate and are being properly and conscientiously followed.

The scope of the audit included all aspects of the NAVF which should be in place and functioning effectively to ensure that PWGSC complies with the policy, procedural and monitoring requirements of the Treasury Board Policy on Account Verification. The effectiveness and consistency in application of the NAVF at Financial Operations in the National Capital and other Regions and Special Operating Agencies (SOAs) were examined for the period April 1, 2003 to September 30, 2004. The audit did not include an independent assessment of PWGSC’slevel of compliance to FAA section 34.

Conclusion

The NAVF’s governance, delivery processes and procedures, risk management practices, and reporting mechanisms were assessed for adequacy in ensuring compliance with the requirements of Treasury Board policy on Account Verification.

The NAVF provides an objective, risk-based, examination of the level compliance with account verification for FAA section 34 across the department. NAVF reporting of errors at rates greater than acceptable levels has led senior management to institute corrective actions, such as training managers with section 34 delegated authority on their responsibilities, and updating the guidelines for delegation of authorities. The trend in account verification error rates being reported has shown improvement since the NAVF’s implementation in April 2003.

Notwithstanding the success of the NAVF in identifying areas requiring improvement, a significant number of errors have gone undetected in transactions reviewed through its quality assurance processes. The department has accepted some risk by conducting NAVF quality assurance reviews on a post-payment basis for all but the most sensitive transactions, which are reviewed prior to payment, and by reviewing statistical samples rather than entire populations of medium and low risk transactions. Due to the number of unidentified critical errors however, the risk level remaining after NAVF processes have been conducted is greater than has been known and accepted.

The wide scope of transactions reviewed, and the large number of potential causes of account verification error necessitates that individuals conducting reviews must have sufficient knowledge of various departmental operations, application of FAA section 34 and other policies such as Hospitality, Travel or Memberships. Detailed, documented NAVF procedures for conducting quality assurance reviews have been effective in providing a common format and method for evaluating account verification across the department. However, more effort, based on the four recommendations provided in this report, is required to ensure the consistent accuracy of reported results.

Recommendations to the Chief Financial Officer (CFO), Finance Branch are contained in the main body of the report.

1 Introduction

1.1 Authority for the Project

This audit was approved by the Public Works and Government Services Canada (PWGSC) Audit, Assurance and Ethics Committee as part of the 2004-05 Audit and Evaluation Plan and was confirmed as a priority in the Audit and Ethics Branch mid-year review.

1.2 Objective

The objective of the audit was to assess the adequacy of PWGSC’s National Accounts Verification Framework (NAVF) for ensuring compliance with the requirements of the Treasury Board Policy on Account Verification (1994/10/01). Specifically, the audit assessed whether the NAVF has appropriate:

• Governance structures, accountability and reporting relationships which ensure the activities are appropriately monitored and adequately supported by senior management.
• Procedures, delivery processes and performance standards to effectively and consistently fulfill the Treasury Board policy requirements.
• Risk management practices which identify and distinguish the risk levels associated with transactions displaying various risk characteristics.
• Reporting mechanisms providing timely and complete information to monitor PWGSC’s compliance with Treasury Board Policy on Account Verification.

1.3 Scope

The scope included all aspects of the NAVF which should be in place and functioning effectively to ensure that PWGSC complies with the policy, procedural and monitoring requirements of the Treasury Board Policy on Account Verification. The effectiveness and consistency in application of the NAVF at Financial Operations in the National Capital and other Regions and Special Operating Agencies (SOA) were examined for the period April 1, 2003 to September 30, 2004. The audit did not include an independent assessment of PWGSC’s level of compliance to FAA section 34.

1.4 Background

It is government policy to pay on time, amounts that represent a legitimate obligation and that are correct. Departmental account verification processes are therefore to be designed and operated in a way that will maintain probity while taking into consideration the varying degrees of risk associated with each payment.

Treasury Board policy requires that departmental processes for quality assurance of the adequacy of section 34 account verification be tailored to reflect the risk level of the transactions under review and must include the review of all high risk transactions. Review of medium and low risk transactions on a sample basis is permitted. Since April 2003, PWGSC has relied on the NAVF, the pre-payment review of highly sensitive transactions and the post-payment quality assurance review of other transactions to ensure that PWGSC is in compliance with the Treasury Board Policy on Account Verification.

Primary responsibility for verifying individual accounts rests with officers who have the authority to confirm and certify entitlement pursuant to FAA section 34. Persons with this authority are responsible for the correctness of the payment requested and the account verification procedures performed. PWGSC’s Financial Services organizations have the responsibility to provide assurance of the adequacy of section 34 account verification pursuant to FAA section 33. The NAVF was developed by Financial Operations Directorate, now Accounting and Controls (ACD), Finance Branch, to provide this assurance across PWGSC.

2 Findings

2.1 Governance

Primary control over the National Accounts Verification Framework is within ACD, Finance Branch, providing appropriate separation between the quality assurance review function and Section 34 line managers. The NAVF document adequately describes the responsibilities of ACD and of Regional and SOA financial services organizations for NAVF functions. The document also details the sampling model used for selecting transactions for NAVF quality assurance reviews. A supplementary detailed operating procedures manual has also been developed and communicated with individuals responsible for conducting NAVF quality assurance reviews.

The functional reporting relationships established between ACD and Regional and SOA financial services are effective for conducting NAVF quality assurance reviews. The NAVF clearly describes the functional reporting relationship between ACD, Regional offices and SOAs’ financial services regarding the conduct of these reviews.

The NAVF is not viewed by Section 33 officers to significantly contribute to their ability to provide assurance of the adequacy of Section 34 account verification. Section 33 officers interviewed were aware that the NAVF existed, but there was little or no knowledge of the procedures for selecting transactions, conducting reviews, or results that had been observed. Regional or SOA specific quality assurance processes, not the NAVF, are being relied upon by Section 33 officers to fulfill their responsibilities. Treasury Board Account Verification Policy identifies that responsibility for the system of account verification and related financial controls rests with those officers who are delegated payment authority pursuant to FAA section 33. The Policy also states that financial officers with payment authority (FAA section 33) must provide assurance of the adequacy of the FAA section 34 account verification and be in a position to state that the process is in place and is being properly and conscientiously followed prior to exercising their payment authority.

Senior management is adequately monitoring and supporting the NAVF. The Executive Committee receives quarterly reports from NAVF quality assurance audits and recommendations for improvements from ACD. There was evidence of action taken in response to NAVF findings targeted at improvement of account verification in the department.

2.2 Procedures, Delivery Processes and Performance Standards

NAVF quality assurance review procedures have been adequately designed to address key Treasury Board Policy elements for account verification for FAA section 34. The procedures established require that transactions selected for quality assurance reviews are examined and evidence obtained to demonstrate that:

• the work had been performed, the goods supplied or the services rendered or in the case of other payments, the payee was entitled to or eligible for the payment;
• relevant contract or agreement terms and conditions had been met including price, quantity and quality;
• where a payment is made before the completion of work, delivery of goods or rendering of services, as the case may be, that such advance payment is required by the contractual terms of the contract;
• the transaction is accurate and the financial coding has been provided; and
• all relevant statutes, regulations, orders in council and Treasury Board policies have been complied with (e.g. Travel, Hospitality, Memberships, Training, etc.).

NAVF quality assurance procedures requiring transactions to be reviewed for errors that are not reported represents an inefficient use of resources. Transactions are reviewed for potential errors categorized as critical or non-critical. Critical errors are defined as those serious enough to require that a correction be made while non-critical errors in comparison are not considered material and pose minimal risk to the department. Though all transactions subjected to NAVF quality assurance review are examined for both critical and non-critical errors, only the results for critical type errors are utilized.

Findings from NAVF quality assurance reviews are not sufficiently accurate. The audit found a significant number of unidentified errors in transactions that had previously been selected for quality assurance review as described in the NAVF. 24%¹ of transactions re-reviewed contained unidentified critical errors. Overall, 51% of the transactions re-reviewed contained critical and/or non-critical errors that had not been identified through the NAVF quality assurance process. Refer to appendix 5.1 for a description of the sample.

In the 89 transactions re-reviewed by the audit, 17 critical errors had been identified through the NAVF quality assurance process. The audit found an additional 24 critical errors in these transactions. Based on the limits of the audit sampling methodology it would be inappropriate to provide a point estimate as to the true critical error rate for all departmental transactions, however in our opinion it can be fairly concluded that the error rates were significantly higher than had been reported.

In addition to critical errors, the audit found 52 undetected non-critcal errors. In the sampled transactions, NAVF quality assurance reviews had identified 55 non-critical errors.

Delivery processes by which NAVF quality assurance reviews are conducted have been appropriately designed. On a quarterly basis, ACD selects individual transactions for quality assurance review, notifies financial services units which transactions to review, consolidates findings and reports results. The reviews are conducted by ACD’s Quality Assurance Unit for the National Capital Region and by Regional and SOA Financial Services according to procedures developed by ACD and documented in a detailed procedures manual.

Adequate performance standards have been established in the NAVF. Department wide results from NAVF quality assurance reviews, which are conducted on a post-payment basis, are measured against maximum tolerable error rates of 4% for high risk and 6% for medium and low risk transactions. These performance standards were deemed acceptable when the NAVF was developed, however there is a need for their periodic review to ensure senior management continues to accept the risk exposure they represent.

¹ Statistically valid result with a confidence level of 90% and a confidence interval (precision) +/- 3%

2.3 Risk Management

Appropriate risk management practices which identify and distinguish the risk levels associated with transactions displaying various risk characteristics have been established and implemented. A transaction’s risk level is determined, for NAVF quality assurance purposes, by weighing the type and dollar value against criteria including; complexity of policy interpretation, recoverability, negative perception and personal benefit. The result of the risk determination process is that all departmental transactions are categorized into risk strata as highly sensitive, high, medium, or low risk, or excluded from review. All highly sensitive transactions are pre-payment reviewed, 100% of transactions categorized as high risk are post-payment quality assurance reviewed, and a sample of medium and low risk transactions are quality assurance reviewed, also on a post-payment basis. Very low dollar value transactions are excluded, but they are reviewed through periodic special post-payment reviews.

The population of transactions is appropriately defined and includes the entire scope of departmental transactions excluding payments related to personnel and contributions to employee benefits plans (see Appendix 5.2). The risk stratification system is being applied as intended and the sampling techniques employed in the selection process are consistent with sound statistical methodology. Treasury Board Policy requires departments to ensure that sampling methodology used is consistent with sound sampling theory with the preferred approach being statistical sampling.

2.4 Reporting Mechanisms

NAVF reporting mechanisms are appropriate and provide timely information to the various stakeholders. The NAVF utilizes a three phased approach to report on the quality of account verification in the department:

• Upon identification of a critical error in a transaction that has been reviewed, a memorandum is sent directly to the responsible section 34 officer providing them with an opportunity to correct the error and/or ensure it is not repeated. A courtesy copy (CC) is sent to the section 34 officer’s supervisor and financial management advisor. The memorandum is intended to bring attention to the error quickly and includes a description of the error found.
• Critical error results from all Regions and SOAs are rolled up into a quarterly report and presented to the Executive Committee. These reports have a department wide focus, and include critical errors identified by type and cause as well as recommendations for improvement. Reports are issued to senior management approximately three months following the end of the quarter being examined.
• Following presentation of results to the Executive Committee, memoranda are sent from the CFO, Finance Branch (formerly Assistant Deputy Minister (ADM)/CFO; Finance, Accounting, Banking and Compensation Branch) to Assistant Deputy Ministers, Chief Executive Officers (of SOAs) and Regional Directors General listing critical errors found in transactions reviewed from their organization. The memos request a response indicating what follow-up action will be taken to correct the individual problems identified.

3 Conclusion

The NAVF’s governance, delivery processes and procedures, risk management practices, and reporting mechanisms were assessed for adequacy in ensuring compliance with the requirements of Treasury Board policy on Account Verification.

The NAVF provides an objective, risk-based, examination of the level compliance with account verification for FAA Section 34 across the department. NAVF reporting of errors at rates greater than acceptable levels has led senior management to institute corrective actions, such as training managers with section 34 delegated authority on their responsibilities, and updating the guidelines for delegation of authorities. The trend in account verification error rates being reported has shown improvement since the NAVF’s implementation in April 2003.

Notwithstanding the success of the NAVF in identifying areas requiring improvement, a significant number of errors have gone undetected in transactions reviewed through its quality assurance processes. The department has accepted some risk by conducting NAVF quality assurance reviews on a post-payment basis for all but the most sensitive transactions, which are reviewed prior to payment, and by reviewing statistical samples rather than entire populations of medium and low risk transactions. Due to the number of unidentified critical errors however, the risk level remaining after NAVF processes have been conducted is greater than has been known and accepted.

The wide scope of transactions reviewed, and the large number of potential causes of account verification error necessitates that individuals conducting reviews must have sufficient knowledge of various departmental operations, application of FAA section 34 and other policies such as Hospitality, Travel or Memberships. Detailed, documented NAVF procedures for conducting quality assurance reviews have been effective in providing a common format and method for evaluating account verification across the department. However, more effort, based on the four recommendations provided in this report, is required to ensure the consistent accuracy of reported results.

4 Recommendations

It is recommended that the CFO, Finance Branch:

1. Improve quality control procedures used to assess the accuracy of National Account Verification quality assurance review findings and consistency of application of the guidelines.
2. Identify training requirements and implement a training program for individuals responsible for conducting National Account Verification quality assurance reviews.
3. a. Examine the rationale to identify non-critical as well as critical errors in National Account Verification quality assurance reviews. Based on results of the examination amend procedures as necessary.
   b. Include results for all errors identified through National Account Verification quality assurance reviews in the reporting framework.
4. Communicate National Account Verification quality assurance results with departmental officers having FAA section 33 delegated authority.

5 Appendix

5.1 Details on Audit Sample

To test the consistency of application of the NAVF quality assurance review procedures and reliability of reported results the audit re-reviewed a random sample of transactions that had previously been selected for quality assurance review as described in the NAVF. The critical test in this exercise was to determine if the audit, following the same guidelines and similar procedures, would replicate the results obtained by the NAVF quality assurance reviews.

Population:
4179 transactions dated from April 1, 2003 through September 30, 2004 that were post-payment quality assurance reviewed following procedures detailed in the PWGSC Post-payment Audit Guide as per the NAVF.

Confidence Level: 90%

Confidence Interval: +/- 3%

Expected error rate: 3%

An "error" was defined as a transaction with a critical error identified by Audit and Ethics Branch auditors that was not identified and recorded through the NAVF quality assurance review process. The expected error rate was based on results from a previous study where by transactions from the fourth quarter 2003-2004 that had been reviewed by financial services in the Regions and SOAs were re-reviewed by ACD. That study examined 379 transactions and found 9 additional critical errors that had not been identified through the initial NAVF quality assurance review process. (9/379 = 2.37% rounded up to 3%)

Sample size: 89 transactions

Sample characteristics:
By Fiscal Year:
Fiscal year 2003-04 - 61 transactions in the sample, 2,861 in the population;
Fiscal year 2004-05 - 28 transactions in the sample, 1,318 in the population.

By Region or SOA:
National Capital Region (headquarters) - 40 transactions in the sample, 1994 in the population (40, 1994);
Atlantic Region - (10, 415);
Ontario Region - (10, 374);
Pacific Region - (4, 272);
Quebec Region - (9, 385);
Western Region - (9, 366);
Consulting and Audit Canada - (2, 192);
Translation Bureau - (5, 181).

By Risk Category:
High Risk - (17, 816);
Medium Risk - (32, 1478);
Low risk - (40, 1885).

Interpretation of results:
The sample result is statistically valid with a confidence level of 90% and a precision of + /- 3%. This means that there are 90 chances out of 100 that the number NAVF quality assurance reviewed transactions containing unidentified critical errors was between 21% and 27%.

5.2 Risk Categorization Matrix