External Audit of the Quality Management System for Annual Audits

I — Executive Summary

The Office of the Auditor General (OAG) carries out about 90 audits of the annual financial statements of the Government of Canada and various government entities. It has developed extensive quality control procedures to ensure that a high standard is maintained in performing this core function for Parliament.

The OAG has engaged an external audit firm, PricewaterhouseCoopers LLP, to conduct, for the first time, an audit of its Quality Management System for Annual Audits.

We conducted a wide ranging search of related best practices for quality management systems and peer reviews concerning recommendations of professional institutes and actual private sector experiences of the UK, USA and Canada. Based on this information and OAG practices, appropriate audit criteria were developed for the audit of each of the following eleven management processes:

1. Authority;
2. Independence, objectivity and integrity;
3. Conduct of the audit;
4. Consultation;
5. Security, access to information and file retention;
6. Resourcing;
7. Leadership and supervision;
8. Performance management;
9. Professional development;
10. Respectful workplace; and
11. Practice review.

The audit criteria related to these elements are outlined in Appendix A and were accepted in writing by the Auditor General of Canada. The design of the Quality Management System for Annual Audits was then assessed for completeness using the criteria developed. Next, by means of interviews, examination of documents and files, limited reviews of selected audit files, as well as some re-performance of quality standard reviews done by OAG staff, we assessed the operating effectiveness of the Quality Management System for Annual Audits.

We are pleased to report that the Quality Management System for Annual Audits was suitably designed and operated effectively during the 1999 calendar year to provide reasonable assurance that the OAG conducted Annual Audits in an appropriate manner.

II — Background and context

The Auditor General of Canada is an officer of Parliament. He carries out his duties through his Office, the Office of the Auditor General of Canada (OAG). The OAG provides independent audit and information services to support the legislative and oversight needs of Parliament. Its work emphasizes making a difference for Canadians by promoting answerable, honest and productive government that reflects a commitment to sustainable development. It carries out independent audits and examinations that provide objective information, advice and assurance to Parliament. This is done to promote: a fair and frank accounting of government's stewardship; efficiency and productivity; cost-effectiveness; the collection of revenues; and compliance with authorities.

The OAG has been in place since Confederation. Its activities cover issuing opinions on the fairness of annual financial statements of governmental entities (Annual Audit) and conducting value for money audits of government activities. The AG legislation has undergone several revisions over the years, most recently to add the responsibilities of the Commissioner of the Environment and Sustainable Development. This report refers only to the Annual Audit activities.

The OAG has seven headquarters groups and five regional offices involved in Annual Audit work. During fiscal year 1998-1999, the OAG carried out about 90 annual audits of the financial statements of the Government of Canada, the governments of the Territories, federal Crown corporations and other entities. The annual audit of the summary financial statements of Canada is by far the largest of the audits, accounting for about 30% of total Annual Audit hours. These audits require about 150,000 hours to complete, at a fully loaded cost of $15 million. The portfolio of entities is very diverse in terms of size, type and nature of audit risk.

The OAG has a Quality Management System (QMS) for Annual Audits that encompasses policies, practices and guidance that continuously evolve in response to emerging standards and best practices. It addresses three areas:

• Audit management expectations for: authority; independence, objectivity, and integrity; conduct of the audit; consultation; and security, access to information and file retention.
• People management expectations related to: resourcing; leadership and supervision; performance management; professional development; and respectful workplace.
• Continuous improvement expectations for internal practice review of audits and practice development initiatives.

The OAG determined that it was advisable to undertake an external audit of its QMS for Annual Audits. Based on a competitive proposal process, PricewaterhouseCoopers LLP was engaged to undertake this first-ever assessment of the OAG's Quality Management System for Annual Audits.

III — Audit approach

The OAG's Quality Management System for Annual Audits encompasses policies, practices, and guidance that continuously evolve in response to emerging standards, best practices and audit risks. The QMS addresses audit management expectations, people management expectations and continuous improvement expectations.

At the outset of this assignment we obtained all relevant information about the objectives and design of the QMS, to ensure that we had a thorough understanding of the Annual Audit services provided, the risks that must be managed and the QMS elements in place to manage these risks.

We conducted the engagement in two phases, namely:

• assessing the suitability of the audit criteria; and
• obtaining an appropriate level of assurance.

Each of these phases is described below.

1 — Assessing the suitability of the audit criteria

Essential to the conduct of the audit was the need to have appropriate audit criteria against which to measure achievement. The OAG had developed criteria for each of its eleven Quality Management Elements and our first task was to assess the suitability of these established criteria for purposes of examining the QMS. Final audit criteria selected had to be accepted by the OAG. We set out in advance that the criteria would be:

• relevant;
• reliable;
• essential;
• understandable; and
• complete.

In assessing the suitability of the proposed criteria, we considered recommendations of the public accountancy institutes in Canada, United Kingdom and United States of America and referred to relevant best practices, benchmarks and authoritative literature covering auditing, process controls, human resource management and information technology issues. The eleven Quality Management Elements are:

1. Authority;
2. Independence, objectivity and integrity;
3. Conduct of the audit;
4. Consultation;
5. Security, access to information and file retention;
6. Resourcing;
7. Leadership and supervision;
8. Performance management;
9. Professional development;
10. Respectful workplace; and
11. Practice review.

The first five elements relate to Audit Management and were addressed by the professional auditors on our team. A very significant body of knowledge exists in Canada and internationally to assess the suitability of the criteria. The fundamental objective of the QMS with respect to audit management is to provide reasonable assurance that Annual Audits are conducted in accordance with legislative requirements, professional standards and OAG policies.

Elements six to ten relate to People Management namely: resourcing; leadership and supervision; performance management; professional development; and respectful workplace. In other words, one of the objectives of the QMS is to ensure good people management practices in the Annual Audits conducted by the OAG. We were very aware that an organization may have a plethora of well-documented human resource policies and programs, but if managers do not put these into practice through their actions and attitudes, the policies and programs may not be effective.

The final element deals with Continuous Improvement to integrate evolving best practices into the operations on an ongoing basis.

Audit criteria covered "suitable design" and "effective operation" of the QMS with respect to both audit and people management. We note that this assignment was not intended as an examination of the OAG's actual operations in these audit and people management areas, but rather to evaluate the way in which the QMS ensures appropriate audit and people management.

To do this, we discussed with the OAG the expectations of the QMS in each of these areas. We conducted interviews with:

• Deputy and Assistant Auditors General;
• line managers and employees;
• representatives from the Human Resources Group; and
• representatives in Professional Practices and Review Group who work with QMS.

To determine the intent of the QMS vis-à-vis both audit and people management, we also enquired about the specific OAG policies, practices, and standards that the QMS uses. As well we considered the audit criteria that had been drafted by the Office.

We determined that the audit criteria that had been developed by the OAG were appropriate for our examination. Appendix A of this report details the criteria, which were agreed upon and accepted in writing by the Auditor General of Canada before we began our examination of the QMS design and operation.

2 — Obtaining an appropriate level of assurance that the QMS was suitably designed and operated effectively

Once we had the criteria against which to measure the design and operation of the QMS, we moved to the next phase. We assessed the extent to which the QMS was indeed suitably designed and operated effectively to provide reasonable assurance that the OAG's Annual Audits were appropriately conducted and that people management practices were in accordance with professional standards, legislative requirements and OAG policies.

First, to assess the design of the QMS, we referred to the eleven elements of the QMS for audit and people management, and determined how they were developed. How was adherence to criteria measured? Did adherence to these criteria mean that the intent of the QMS was being carried out?

Next, we assessed the extent to which the QMS operated effectively with respect to audit and people management. We verified the existence and effective functioning of controls embodied in:

• OAG policies;
• Audit practices;
• Human resource activities;
• Internal practice reviews and reports; and
• Information technology risk reviews.

Our examination also encompassed discussions with relevant staff and management to assess their understanding of the QMS. All phases and elements of the QMS were evaluated and selected re-performance of the QMS procedures was undertaken at various times to determine effectiveness of procedures during the 1999 calendar year. We reviewed several audit files for which audit opinions were issued during the 1999 calendar year. In addition, we reviewed the results of the quality reviews conducted during the year by the OAG's Professional Practices and Review Group.

Based on the results of our work we issued our report, which is formalized in the next section.

IV — Auditors' report

To the Auditor General of Canada

We have examined the Quality Management System (QMS) for Annual Audits of the Office of the Auditor General of Canada. The Office's annual audits cover the financial statements of the Government of Canada, federal Crown corporations and other entities, and territorial governments and corporations. The QMS is intended to provide reasonable assurance that the Office conducts annual audits in accordance with applicable legislative requirements, professional standards and Office policies.

The Office of the Auditor General is responsible for the design and effective operation of the QMS, as well as for the quality of the annual audits. Our responsibility is to express an opinion on whether the Office of the Auditor General's QMS provides reasonable assurance that the Office conducted annual audits in accordance with applicable legislative requirements, professional standards and Office policies for the year ended December 31, 1999.

The criteria used to assess the effectiveness of the QMS were selected specifically for this audit based on generally accepted auditing standards, legislative requirements including the Auditor General Act and the federal and territorial Financial Administration acts, and quality control standards established by professional bodies. These criteria are listed in Appendix A.

Our examination was made in accordance with the standards for assurance engagements recommended by the Canadian Institute of Chartered Accountants. Accordingly, it included such tests and other procedures as we considered necessary in the circumstances.

In our opinion, the Office of the Auditor General's QMS for Annual Audits was suitably designed and operated effectively during the year ended December 31, 1999 to provide reasonable assurance that the Office conducted annual audits in accordance with applicable legislative requirements, professional standards and Office policies based on the criteria described in Appendix A.

Chartered Accountants

January 31, 2000
Ottawa, Canada

Appendix A

Office of the Auditor General

Audit Criteria for the Quality Management System for Annual Audits