|
|
 |
Peer Review Report's Suggestions for Improvement and Our Action Plan
|
1. Looking at the scope of some audits for example by giving more emphasis
to setting out the issues, criteria, methodology, sources of evidence and likely
areas for recommendations at the design and planning stage. This could also help
identify the need for external expertise, and internal technical and subject matter
expert input.
|
Action
At the design and planning stage of each value-for-money audit, we produce
a survey report that sets out significant audit issues, audit objectives and criteria,
audit approaches and methodologies, sources of evidence, and potential areas for
recommendations. After consultation with internal and external experts, we develop
an audit proposal.
By December 2004 we will
- update our guidance and methodology on survey reports and audit proposals
to specify coverage of the issues suggested by the peer review;
- communicate our expectations concerning these planning reports to our value-for-money
practitioners;
- systematically monitor our practices against our expectations; and
- include the names of potential external advisors and their areas of expertise
in the audit proposal.
(Our action on suggestion 4 deals with audit recommendations.)
|
2. Reviewing the breadth of the audit methodologies routinely used on VFM
audits to identify the opportunities for expanding the range of techniques
used including use of comparative analysis and more quantitative and qualitative
techniques such as focus groups. This could add depth to the audits and may lead
to some savings in audit resources which could be used to tackle more complex
areas of the audits.
|
Action
By December 2004 we will
- identify and assess methodologies used by other national audit offices;
- revise and promote our Auditor's Tool Kit, which contains guidance on alternative
methodologies; and
- devote a significant portion of the 2004 annual symposium for value-for-money
practitioners to how alternative methodologies can be applied.
(Also see our action on suggestion 3.)
|
3. Expanding the training and guidance available to VFM staff on methodologies
and presenting report data. This could include stronger recommendations and improving
the presentation of audit reports by greater use of figures, graphics and case
examples to support the main findings. There is also scope for learning lessons
from audits and ensuring the Office has the appropriate mix of staff competences
to deliver quality value-for-money products.
|
Action
Methodologies and presenting report data
By June 2005 we will
- develop and deliver training courses for value-for-money practitioners on
evidence-gathering methodologies and techniques for presenting data in reports.
Stronger recommendations
By December 2004 we will
- develop a course on how to make stronger recommendations and include it in
our value-for-money professional development program.
Improving presentation of reports
By December 2004 we will
- review the Canadian Institute of Chartered Accountants' study on using graphics
and tables to present complex numerical and financial data, tailor it to our needs,
and reference it in our Value-for-Money Audit Manual.
Scope for learning lessons
By December 2004 we will
- develop a database containing lessons learned from value-for-money audits
and suggestions for improvement to our quality management framework; and
- establish a systematic approach to ensuring that lessons learned and suggestions
received from practitioners are considered, and, where appropriate, incorporated
into our quality management framework and value-for-money audit procedures.
Ensuring appropriate mix of competencies
By December 2004 we will
- complete a review of all value-for-money audit teams to ensure that the appropriate
mix of human resources is available to manage audit workload over the long term;
- redesign our professional development curriculum to focus on the competencies
necessary to deliver value-for-money products; and
- determine whether core elements of the curriculum should be mandatory for
all value-for-money practitioners.
|
4. Recommendations should be specific and action-orientated. Earlier
consideration of the areas for recommendation in the audit cycle could enable
audit teams to focus on beneficial change and discuss with departments at an earlier
stage.
|
Action
By December 2004 we will
- consult our stakeholders—including our value-for-money practitioners,
federal departments, and the Treasury Board Secretariat—to define the key
issues associated with our report recommendations;
- finalize guidance on how to write specific and action-oriented recommendations
and get approval from our Practice Development Committee;
- reference the guidance from our Value-for-Money Audit Manual and incorporate
it in standard practice;
- adjust our practice to ensure that potential recommendations for each line
of enquiry are considered in the planning stage of the audit; and
- develop a course on how to make stronger recommendations and include it in
our value-for-money professional development program.
In 2004, we issued with the Treasury Board Secretariat a protocol for deputies
and senior managers in departments on working with the Office on VFM audits. The
protocol encourages a transparent process and open dialogue with departments at
each stage of an audit.
|
5. Improving the presentation of reports both in terms of use of graphics
and tables to present complex numerical and financial data to enliven the chapters.
Footnotes might be used to show sources of evidence. Skeleton or outline reports
could also be used to develop key graphics at an earlier stage. Finally, more
prominence could be given in audits (perhaps in the About the Audit section of
each chapter) to where experts and consultants have been used and thereby add
weight to the audit findings.
|
Action
By December 2004 we will
- review the Canadian Institute of Chartered Accountants' study on using graphics
and tables to present complex data, tailor it to our needs, and reference guidance
in our Value-for-Money Audit Manual (same as action for suggestion
3).
By June 2005 we will
- develop courses for value-for-money practitioners on techniques for presenting
complex data in reports (see action for suggestions 3 and
4).
By September 2004 we will
- determine whether or not reference notes would add value to our reports and
if so, how they might best be incorporated.
By June 2004 we will
- identify options for highlighting, in our reports and communiqués,
the methods we use to ensure quality in our value-for-money audits, including
the use of experts; and prepare a proposal for approval by the Executive Committee.
|
6. Doing more to show that the VFM audits make a difference, for example
in following-up on not only where recommendations have been implemented but what
they have achieved in terms of improved performance.
|
Action
By February 2005 we will
coordinate the processes we use to generate information for our annual status
report, which addresses progress on audit recommendations, and for our performance
report, which highlights action and changes from our audit work, to ensure that
they adequately describe instances where our audit work can be linked to improved
government performance.
We have recently revised our methodology for following up on audit recommendations.
Our revised methodology will address suggestion 6.
By June 2005 we will
- survey a random sample of parliamentarians and members of Parliament from
key committees, such as the standing committees on Public Accounts and Environment
and Sustainable Development. The survey will help us understand how familiar parliamentarians
are with the Office and our products, and how we can improve our value-for-money
products to better serve their needs.
By November 2005 we will
- survey a sample of parliamentary and legislative assistants who read our reports
and brief parliamentarians, to see how we can improve our value-for-money products
to better serve parliamentarians.
By June 2005 we will
- survey parliamentary committee clerks and research staff of the parliamentary
library who read our reports, to see how we can improve our value-for-money products
to better serve parliamentarians.
We try to survey parliamentarians within one year after an election.
We will
- continue to survey senior managers responsible for areas we have audited,
after each report is tabled, to obtain their feedback on the value of our process
and products.
|
7. Looking at the cost of individual audits with more analysis of the comparative
costs of audits and possibly the cost elements of individual audits to determine
whether the resources allocated to them reflect the needs of the audit—this
is closely linked to the approval of audits and the methods used and the perceived
'cost' of the QMF (quality management framework).
|
The work done in response to this suggestion is fully integrated with the
Office's Modern Comptrollership Capacity Assessment and Action Plan.
Action
By March 2005 we will
- refine our existing guidelines used to plan and monitor value-for-money audits
to address the level of effort normally associated with each stage of the audits;
- communicate these refinements to our value-for-money practitioners and reference
them in our Value-for-Money Audit Manual; and
- refine our approval and monitoring processes to reflect these guidelines,
including the introduction of our new executive reporting package.
|
Year