Français | Contact Us | Help | Search | Canada Site | |||||
About Us | Publications | Media Room | Site Map | OAG Home |
O A G |
|
International Peer Review of the Value for Money Audit Practice of the Office of the Auditor General of Canada
Summary ReportTo: The Auditor General of Canada 1 An international peer review team with representatives from the Supreme Audit Institutions of the United Kingdom, France, Norway, and the Netherlands examined the value for money audit practice of the Office of the Auditor General of Canada (the Office). Two members of the United States General Accounting Office participated as observers. 2 The objective of the Office's value for money audit practice is to provide Members of Parliament with independent, objective and supportable information that they can rely on to examine the government's performance and hold it to account. To achieve that objective, the Office has established a quality management framework to ensure that value for money audits are carried out in accordance with the Office's legislative authorities, recognized standards of professional practice and with due regard for economy, efficiency and effectiveness. 3 The Office is responsible for the design and effective operation of its value for money audit practice including the quality of the audit products. Our responsibility is to determine whether the Office's value for money audit practice is suitably designed and operating effectively to achieve its objectives. 4 Our examination was carried out in accordance with commonly accepted auditing principles consistent with the Code of Ethics and auditing standards issued by the International Organization of Supreme Audit Institutions. 5 The criteria used to assess the Office's value for money audit practice are based on the audit policies stipulated in the Office's Value-for-Money Audit Manual. These policies derive from the legislative authorities specified in the Auditor General Act (1995) and relevant auditing standards promulgated by the Canadian Institute of Chartered Accountants. The Office's value for money audit policies are attached at Appendix A. 6 We assessed whether the Office's quality management framework is suitably designed to incorporate legislative authorities, recognized standards of professional practice and appropriate measures to ensure that value for money audits are carried out with due regard to economy, efficiency and effectiveness. 7 We also assessed the extent to which the value for money audit practice is operating effectively - to produce independent, objective and supportable information that Members of Parliament can rely on to examine the government's performance and hold it to account. 8 We found that as at 31 December 2003, the Office's value for money audit practice was suitably designed and operating effectively to achieve its objectives. 9 The remainder of this report highlights potential opportunities that management may wish to consider to enhance the quality of the Office's value for money audit policies and practice. Caroline Mawhood - Review Leader 4 February 2004 Opportunities for ImprovementIntroduction1 The international peer review team conducted an independent and objective review of the Office's value for money (VFM) audit practice in accordance with commonly accepted auditing principles and standards. The team examined the Office's documented policies and procedures, a number of the Office's 2003 audits and interviewed staff involved in the audits and with responsibility for functional areas. The review team members are listed at Appendix B which also outlines the review team's methodology in more detail. 2 We are pleased to report that as at 31 December 2003 the Office's VFM audit practice was suitably designed to incorporate legislative authorities and recognised standards of professional practice, and to ensure due regard for economy, efficiency and effectiveness; and that it was operating effectively to provide Parliament with independent, objective and supportable information which can be relied on to examine the government's performance and hold it to account. 3 In performing the peer review we drew on the broad and diverse experience of the international team of peers to make a number of observations on the Office's approaches and practices and to identify a number of areas which the Office should consider for improving its VFM audit practice and quality management framework (QMF). Overall VFM audit approach and quality management framework4 Audit policies The Office has appropriate audit policies in place to ensure that the resultant audit reports achieve their objectives. The Office has an extensive quality review process, covering all aspects of the VFM audit procedure from the selection of audits through to post-tabling activities, and including environmental and sustainable development audits and follow-up work. This strong quality assurance process helps to ensure that appropriate audit policies are followed. Our review of audit chapters found that the QMF is operating effectively. The roles of the quality reviewer, practice reviews, and the VFM Management Committee are developing and providing quality assurance throughout the VFM audit cycle. These are sound mechanisms for checking if controls are operating effectively and ensuring that the audit approach is challenged.
5 VFM auditing policies are written as "should" statements in the Office VFM audit manual and must be complied with. Other "should" statements are also included in the text of the VFM audit manual and are expected practices. Where the "should" statements relate directly to a VFM audit policy or Canadian Institute of Chartered Accountants standard they are consistent with those policies and standards. There are a number of other controls relating to the process of producing a VFM audit, some of which are essential to produce a good quality VFM product. But each control appears to have an equal risk weighting and some of the controls may have been developed for financial audits and do not always readily read across to VFM work. There is a risk that the quality assurance process focuses on ensuring a standard level of quality without promoting a general improvement in the quality of audit products. To address this risk the review team believe that in developing the QMF the Office should consider:
Good practice6 The review team observed a number of good practices in the Office's VFM process and QMF (Figure 1). These add weight to the robustness of the Office's VFM products and process controls. There are some good examples of timely coverage of subjects of concern to Canadians that can lay the foundation for further work (for example in the audit chapters on the Road Transportation in Urban Areas and Canada's Strategy to Combat Money Laundering). There is good work on guidelines for accountability arrangements, on a rating tool for measuring departmental performance and an overview report on sustainable development issues.
7 Audit planning Strategic planning, audit selection and coverage of the audit field are sound with good concentration on risk assessment in audit plans (one pass planning) and on the contribution of audits to the five focus areas of the Office. This gives the Office's work coherence and direction. The risk assessment process is conducted as part of wider discussions between the relevant departments and the Office. The audit selection process is developing and the Office recognises that there will need to be more emphasis placed on how VFM resources are allocated to audits in subsequent rounds.
8 Audit implementation The main audit findings and conclusions are supported by the evidence and there is good consultation with departments and other stakeholders. Review of audit files and interviews with staff suggested that detailed file review and interview were the main audit techniques used and in the audits reviewed the evidence fully supported the report findings and conclusions. There is often wide consultation with interested groups and organisations, for example on sustainable development audits, and all audits have an advisory committee of external experts. There are functional experts and subject matter experts, to for example, advise audit teams on quantitative analyses or accountability arrangements, as well as quality reviewers for each audit. 9 Performance measures The Office measures its impact in improving the government's accountability, operations and services by determining the percentage of recommendations acted upon within four years of being made. Each year the Office publishes a report with a number of audit chapters following-up earlier work. The Office's annual report also has some performance measures, for example, the number of recommendations endorsed by the Public Accounts Committee. 10 Feedback The Office has introduced a number of mechanisms to obtain feedback on VFM audit work from its main stakeholders. In 2002 the Office commissioned a detailed survey of Members of Parliament. Overall it found a high degree of support for the role of the Office. In response to comments from Parliamentarians the Office has prepared a strategy to enhance communications, addressing one of the main findings of the survey. Implementation of this strategy should help to ensure that key messages are communicated more pro-actively. 11 The Office has also recently begun conducting surveys of officials whose departments have been the subject of a VFM audit in the past year. These surveys are intended to provide feedback on the Office's VFM audit approach and process and on the recommendations that officials are expected to act on. The surveys will provide the Office with opportunities to continue to improve its work and to address any concerns arising. Areas for improvement12 The review team identified a number of areas which the Office should consider for improving its VFM products. Audit planning13 Looking at the scope of some audits The Office could consider examining in more depth the scope of some audits early in the design and planning phase to ensure the scope is appropriate to meet the audit objectives; for example by giving more emphasis to clarifying the issues, criteria, methodology, sources of evidence and likely areas for recommendation. This could also help to identify at an early stage the need for external expertise and internal technical and subject matter expert input. 14 Audits could benefit from greater consideration at the planning stage of the issues and likely recommendations that would address economy and efficiency matters. Consideration could also be given to how to design recommendations that are clear about the action departments need to take, and are focused on causes, results, and departments' performance. Audit implementation15 Reviewing the audit methodologies routinely used on VFM audits to collect and analyse audit evidence The methodologies used in undertaking VFM audit are sound but there may be opportunities to make more use of techniques such as focus groups and quantitative analysis. This could provide opportunities to present audit findings in a more compelling way. There has been some use of surveys, benchmarking and statistical analysis (see for example the audit Correctional Services Canada - Reintegration of Male Offenders, which used a survey of parole officers). A richer set of methods could be more cost effective for the auditor and auditee and lead to more recommendations that result in beneficial change and improved efficiency and economy. Figure 2 outlines some VFM techniques and how they could be used.
16 The use of a wider set of methods could be usefully supported by:
17 Organisational learning There are also opportunities for more organisational learning. Currently, for example, there is no comprehensive system to collect and analyse all lessons from audits and feedback from stakeholders and to bring this together with the quality review material to inform guidance and training. Audit reporting and follow up18 Recommendations should be specific and action-orientated Earlier consideration of the areas for recommendation in the audit cycle could enable audit teams to focus on beneficial change and discuss with departments at an earlier stage. 19 When the Office look at effectiveness, they focus on whether the entity has established procedures to measure its results and program effectiveness. They have found that better results can be achieved through better management and therefore audit recommendations often tend to focus on accountability and procedural issues - requiring audited bodies to set clearer performance measures, improve management information, and clarify responsibilities. Audits could however benefit from stronger, more specific, action-orientated recommendations focusing on economy and efficiency and performance issues - focusing on the results achieved and how they can be improved. It is also not always clear from audited bodies' responses to the recommendations what action they intend to take. 20 Report presentation The audits we have reviewed have been generally sound, well-structured, succinct and well written. The policies set out in the VFM audit manual ensure that findings are accurately reported and addressed to meet the concerns of stakeholders. 21 Some audits make good use of graphics - for example the audits Rating Departmental Performance Reports and Managing the Safety and Accessibility of Pesticides include a number of graphs that communicate findings well. Other reports could benefit from a wider use of graphics and illustrations to present complex numerical data and other information. The Office could also give more prominence in reports to where external experts and consultants have been used and thereby add weight to the audit findings. 22 Doing more to show that the VFM audits make a difference Although the Office has a good set of mechanisms to measure their impact, they could benefit from showing that VFM audits do make a difference by assessing whether a problem has been resolved by implementing the recommendations and whether performance has improved. Use of VFM resources23 Looking at the cost of individual audits with more analysis of comparative costs. Any good VFM practice should strive to improve its efficiency both for its internal operations and for its clients. The Office has a diversity of audit products: full VFM audits, audit notes, follow up reports and coverage of environment and sustainable development matters. The average time to be spent on an audit is around 5,000 hours for a follow-up audit, 9,000 hours on a regular VFM audit, and 10,000 hours for a government wide audit, and the average cost of regular VFM audits in 2003 was over $1,100,000. 24 More analysis of the cost elements of individual audits is needed to determine whether the resources allocated to them reflect the needs of the audit. Costs are closely linked to the methods used and the size of audit teams. Audit teams can at some stages in the audit be somewhat large. Although the whole team would not be working on the audit full-time there is some inflexibility in the way that staff are allocated to audit. Human resources are generally allocated to audits based on past experience and opportunities may exist to review and reallocate resources at various stages of audits 25 The Office could consider more use of audit notes and small scale VFM audits to test new methodologies, train staff, make better use of VFM resources and enable positive issues to be highlighted. Small scale VFM audits are a good vehicle for testing new methodologies and approaches and in doing so enable innovation to flourish where this could be stifled by time constraints in a "normal" VFM audit. Summary of suggestions26 The Office should consider:
Appendix AOffice of the Auditor General of Canada
|
Last Updated: 2004-03-11 |