Audit of the National Grants and Contributions Performance Tracking Directorate Activities

Project No.:  6575/00

 

 

 

Internal Audit and Risk Management Services

 

Director General: J.K. Martin

Audit Director:     S. Saint

Team Leader:      G. Tousignant

Auditors:              M. Gagnon

                            J. Ouellet

                            B. Sanderson

 

 

 

APPROVAL:                     Original copy signed by:

                                       

DIRECTOR:                          Stuart Saint                           November 7, 2001                                   

DIRECTOR GENERAL:      James K. Martin                November 7, 2001           

                                                         

October 2001

 

 

 

 

TABLE OF CONTENTS

 

1.0    EXECUTIVE SUMMARY

2.0    INTRODUCTION

3.0    AUDIT OBJECTIVES AND APPROACH

4.0    FINDINGS

5.0    CONCLUSION

 

 

Appendices

appendix a  –  level of assurance

appendix B  –  Summary of audit objectives and their criteria

appendix c  –  Action Plan

 

1.0     EXECUTIVE SUMMARY

 

The “National Grants and Contributions Performance Tracking Directorate” (PTD) was created in the fall of 1999 and is designed to act as a quality control body by measuring the quality of the administration of Grants and Contributions (G&C) agreements within HRDC.

 

This new Directorate is conducting ongoing reviews of G&C program administration through the life cycle of a typical G&C project.

 

Tracking and examination is done by reviewing files and performing financial reviews of sponsor documentation. Using random sampling the Directorate collects and analyzes data and reports on the various stages of G&C administration. PTD reviews provide the Department with ongoing and detailed information on its level of compliance with required and expected administrative standards for G&C.

 

This audit was proposed by the PTD Advisory Committee and specific Terms of References were approved by the Audit and Evaluation Committee on February 8, 2001. This audit is intended to provide senior management with an opinion on the adequacy of the internal control systems being put in place by the PTD and how well they are performing in applying approved sampling, review and quality assurance methodologies to provide accurate information on the administration of departmental Gs&Cs.

 

The overall objectives of this audit are to assess:

 

• the effectiveness of the application of the selection and review methodologies developed for the Directorate;
• the adequacy of the management control framework being put in place for the unit;
• how effectively the National Grants and Contributions Performance Tracking Directorate has measured and reported the results achieved;
• the extent to which reliance can be placed on information extracted from the performance tracking system; and
• the extent of coordination with other monitoring bodies in HRDC.

 

 

Overall conclusion

 

Based on our audit work and the successful implementation of management’s action plan to address the issues identified in this report, we conclude for the following audit objectives that: 1)      the sample selection and file review methodologies are effective. 2)      the review methodology is applied fairly and consistently. 3)      the management control framework is effective. 4)      within the limits detailed by PTD in their reports, a high level of reliance may be placed on the information produced by PTD. 5)      while, in our opinion, PTD has substantially fulfilled its obligations under this part of its mandate, we are not in position, at this time, to issue an opinion on the adequacy of coordination of PTD activities with other bodies within HRDC. The audit was designed to provide conclusions with a high level of assurance on the effectiveness of the application of the review methodology (objective 2) and reliance to be placed on the information produced by PTD (objective 4) and with a moderate level of confidence on the other components of the audit (objectives 1, 3 & 5). Ongoing performance measurement of the management of G&C agreements is an essential tool for monitoring and assessing the management control framework relating to G&C. Regular reports should continue to be made to senior management and related HRDC accountability and reporting committees. HRDC should continue to closely monitor and report on the management of G&C. We believe that if the four recommendations included in Section 4 of this report are implemented as discussed with management, the overall process and control framework already in place will be reinforced and improved.

 

2.0          INTRODUCTION

 

This report presents the results of the audit of the National Grants and Contributions Performance Tracking Directorate (PTD) of Human Resources Development Canada (HRDC) covering the activities performed by PTD from February 2000 to March 2001.

 

The National Grants and Contributions Performance Tracking Directorate (PTD) was created in the fall of 1999. With a staff of 23 members and a budget of $2.7 million, the Directorate is designed to act as a quality control body by measuring the quality of the administration of Grants and Contributions (G&C) agreements.

 

PTD has the following mandate:

 

• provide assurance that Grant & Contribution funds are being managed and spent in accordance with program terms and conditions;
• highlight key areas of risk and provide guidance on approaches to mitigate those risks;
• contribute to knowledge transfer throughout the department; and
• co-ordinate activities with other monitoring and post-audit functions carried out within HRDC and by external agencies to ensure adequate coverage, without duplication of effort.

 

This new Directorate is conducting ongoing reviews of G&C program administration through the life cycle of G&C projects.

 

The Directorate uses the following process in the performance of its mandate: (available in the PDF Version only).

 

 

PTD gathers its information by reviewing program area files and performing financial reviews of sponsor documentation. The Directorate collects and analyses the data collected and then reports on the various sub-activities of Grants and Contribution (G&C) administration. In order to fulfil its mandate, the Directorate is using a rigorous sampling methodology to draw its sample of files for review. This sampling methodology was developed in consultation with Internal Audit and Risk Management Services of HRDC (IARMS) and was subsequently reviewed by the Office of the Auditor General (OAG). As well, the file review, financial review of sponsor documentation and quality assurance methodologies were developed after an extensive consultation with program management and were reviewed by IARMS and the OAG. The Directorate also submits regular, periodic progress reports.

 

PTD reviews provide the Department with ongoing and detailed information on its level of compliance with required and expected administrative standards for Grants & Contributions agreements. The Directorate released three Quarterly Progress Reports in the period covered by our audit. The last one reported on the 7,600 files that were active as at March 31, 2001with a start date between February 1, 2000and June 30, 2000. A total of 419 files were reviewed by PTD in their first year of operation.

 

PTD Audit Universe Related to Grants and Contributions

 

The Performance Tracking Directorate’s audit universe for their Third Progress Report consisted of all Grant & Contribution files with a start date between February 1, 2000and June 30, 2000, with some exceptions. There are three broad areas of Grants and Contributions that are excluded from PTD review. Multi-year Aboriginal files were also excluded, as they did not meet the start date criteria for PTD review. In summary these are:

 

• allotments and sub-activities that relate to individual income support only. As these involved money paid directly to individuals as the result of a counselling intervention and because of the low risk and materiality level of these payments. These agreements are reviewed by local Program Officers as part of the claim review process. PTD has not yet developed plans to review these exclusions.
• transfer payments to provinces, as these expenditures will be audited by provincial Auditors General.
• Summer Career Placement, as this program does not have the same administrative requirements as other G&C programs. Also this program has a very high number of files, with individual low dollar values. Using random selection methods, the data of this program would have an impact on national results that would not be in proportion to the value of the program. This program is of short duration and funded only once a year. These agreements will be the subject of a special PTD review that is in the planning stages.
• as pointed out in the second PTD progress report, Aboriginal Relations Office (ARO) files were selected and subsequently excluded from the sample. This decision was correctly taken based on the fact that most existing ARO agreements cover five-year Aboriginal Human Resource Development Agreements that began before the start-date parameters of the Performance Tracking Population. A specific review of ARO agreements is in process and a report should be released shortly. IARMS will review the report.

 

The list of all files in the PTD audit universe is generated through the finance system based on every financial commitment made through Grants and Contributions by HRDC.

 

The following charts (Available in the PDF Version only) provide an overview of the distribution of G&C files and expenditure population and the exclusions for the year 2000.

 

 

PTD General Conclusions from its March 2001 Report:

 

• PTD reported that there "continues to be significant, measurable improvement in the overall program management and administration of G&C”;
• It should be noted that HRDC management has taken a number of specific measures and initiatives to enhance the overall administration of HRDC G&C since June 30, 2000. The full impact of these additional activities is not reflected in the March 2001, PTD Quarterly Progress Report, but is expected to become apparent in the findings of future PTD assessments.

 

 

3.0          AUDIT OBJECTIVES AND APPROACH

 

This audit has two intended results. The first is to provide senior management with an opinion on the adequacy of the internal control systems in place in the Performance Tracking Directorate. (PTD). The second is to assess how well the PTD is performing in respect of its mandate.

 

The specific objectives of this audit are to assess:

 

1.      the effectiveness of the application of the sample selection and file reviewmethodologies developed by PTD;

 

2.      the effectiveness of the application of the file review (file review & financial review) of sponsor documentation;

 

3.      the adequacy of the management control framework put in place for the PTD;

 

4.      the extent to which reliance can be placed on information from the performance system for various PTD purposes; and,

 

5.      the adequacy of the extent of coordination of PTD activities with other monitoringin HRDC.

 

The audit was conducted at NHQ from May 16 to June 29, 2001, following the production of a planning report which detailed the following: the audit methodology to be applied in the course of the field work; a set of criteria to be used as a guide in reaching conclusions regarding the adequacy of PTD practices; and a general approach for gathering appropriate evidence to support audit conclusions and related recommendations. The audit was designed to provide conclusions with a high level of assurance on the effectiveness of the file & financial review and reliability of the data collected and with a moderate level of confidence on the other components of the audit project.

 

The fieldwork included interviews, analysis of documentation and working tools developed by PTD, and a review of a valid statistical sampling of project file (file and financial review) which provide a high level of confidence. The fieldwork focussed on PTD activities performed in relation to the 3 Quarterly Progress Reports submitted in the course of their first year of activities.

 

 

4.0          FINDINGS

 

OBJECTIVE 1

 

Assess the effectiveness of the application of the (1) sample selection, and (2) file review methodologies developed by the Performance Tracking Directorate (PTD).

 

This objective is divided in 2 sub-objectives, the first one addressed the issues related to the effectiveness of the application of the sample selection methodology and the second sub-objective addressed the issues related to the effectiveness of the file review methodologies developed by PTD.

 

(1) Effectiveness of the application of the sample selection methodology:

 

The sampling methodology used by PTD was developed in consultation with the Internal Audit Risk Management Services of HRDC and PTD with the assistance of an external consultant who specializes in statistical sampling. The methodology was reviewed and endorsed by the Office of the Auditor General of Canada. Our audit focussed on the following:

 

the adequacy of the sample selection criteria;

• the proper use of the sample selection criteria; and
• the accuracy of the download of data from the financial system.

 

We reviewed the approved sampling methodology and the complete sampling process applied by PTD for the production of each of their 3 quarterly progress reports. For each report the sampling process was applied as prescribed.

 

As discussed previously in this report, the PTD audit universe has several exclusions. In our opinion, these exclusions are appropriate and like exclusions should continue to be made as long as PTD separately report on the compliance and efficiency of these excluded programs.

 

Conclusion

 

We conclude that, in all material respects, the sampling methodology was applied consistently and in conformity with the approved instructions.

(2)   Effectiveness of the file review methodologies developed by PTD:

 

The file review methodology developed by PTD covers the following aspects:

 

A detailed file review assessment questionnaire, the Tool Use Guide (TUG), was developed by PTD and reviewed by the Office of the Auditor General for comments and improvements. This questionnaire is used by the file reviewers to evaluate compliance with current policy and procedures regarding G&C agreements and is updated on an ongoing basis. The captured data is fed into the PTD database for data analysis and reporting purposes. The content of the questionnaire covers the following components of the life cycle of a typical G&C project:

 

-       Application

-       Environmental Assessment

-       Recommendation and Approval

-       Contracting

-       Amendments

-       Payments

-       Advances

-       Monitoring

-       Follow-up and Evaluation

-       Compliance with Terms and Conditions

-       General Administration

 

and for financial review:

 

-       Financial Documentation Review

-       On-site visit where appropriate

 

For every instance of non-compliance, PTD makes an observation and a recommendation that will require follow-up with the related program authority.

 

For quality control, an independent group completes a second complete review of PTD results. This group included professional resources provided under agreement with Consulting and Audit Canada. Following the quality control assessment, file observations and recommendations are sent to the appropriate program authority for validation and preparation of a management response. In addition all completed reviews require a management certification completed by the PTD Director. Upon completion of the quality control process and once management approval is obtained, management responses and follow-up action plans are incorporated into the final summary of results. The final summary is then forwarded to the relevant program authorities for follow-up and implementation of corrective actions.

The Office of the Auditor General in their October 2000 report Chapter 11, page 11-32 reported: "The performance tracking system the Department has developed to monitor progress is an innovative approach to ongoing performance measurement. If applied systematically, it should give management a good measure of progress being made".

 

We wish to commend PTD for the priority that is placed on upgrading and enhancing the effectiveness of Tool Use Guide questionnaire (TUG). The TUG is regularly reviewed and up-dated, in order to ensure that the file review and quality control officers have the best possible tool to consistently evaluate the files they review, and is in compliance with any revised financial policy, such as the revised Treasury Board Policy on Transfer Payments.

 

PTD’s current policy requires that files selected for review be copied in full and the copies forwarded to PTD for review within 10 working days. The file return timeframe allows for the following activities to be undertaken:

 

• email sent to the regional audit coordinator requesting the list of files to be copied;
• regional audit coordinator sorts the files by location and contacts local HRCC program officer requesting the files documentation to be copied;
• the copied files are returned to the regional audit coordinator; and
• the copied files are packaged together and subsequently returned to NHQ PTD.

 

As some of the files contain a lot of information, it requires effort and time to copy the contents of the file. In our opinion, the allocated 10 working days time frame seems appropriate for files requested from an HRCC office. However, for files held at Regional Office and for NHQ programs, this allocated time frame may be excessive.

 

In order to improve the control over the receipt of files, PTD has recently implemented a control log, which records the request date and the PTD receipt with elapsed working days. The control log used for the July 2001 Progress Report demonstrated that 69% of files were not submitted to PTD within the allocated time frame of 10 working days. Some of them were submitted to PTD in a time frame exceeding 30 working days.

 

Conclusion

 

We conclude that PTD has developed appropriate file review methodologies that, when applied consistently, adequately measure HRDC performance in the administration of grants and contributions.

 

Recommendation 1:

 

We recommend that PTD re-examine the time allocated to local HRCC, RHQ and NHQ program officers to forward the files selected for PTD review.

 

Recommendation 2:

 

We recommend that PTD review all excluded programs to find cost-effective methods of measuring HRDC performance.

 

 

OBJECTIVE 2

 

Assess the effectiveness of the application of the file review (file review & financial review of sponsor documentation)

 

In order to respond to this audit objective, the following methodology was applied.

 

File Review Component:

 

A total of 419 files were reviewed by PTD for inclusion in the 3 National Progress Reports released during our audit period. From these 419 files reviewed, we randomly selected a statistically valid sample of 42 files. This sample provides a high level of confidence on the review results. From the PTD Tool Use Guide questionnaire, we selected 22 of the most significant/key risk questions. We then conducted a complete review of the PTD assessment, to ensure that the conclusions reached by PTD in their review were appropriate, that all related audit evidence was on file and reported adequately in the PTD database (SNIFURS).

 

For each of the key questions, we applied the same review process performed by PTD in the course of their file review assessment, as described in the PTD Tool Use Guide for the files reviewed. Our specific objective was to ensure that the PTD process and instructions were applied correctly and consistently. We also asked two general questions regarding Terms and Conditions and standard contract forms.

 

In less than 1% of the responses reviewed our determination of the outcome disagreed with that of PTD. The challenge role performed by the PTD quality control function is highly effective. In addition to the quality control function, management review performed by the PTD Director reinforces the overall quality process and the reliability of the PTD assessment. The File Review Results and the Final Summary of Performance Tracking Results provide PTD and the program authority with a clear summary of the PTD assessment. This is complemented by appropriate observations and recommendations which also reflect the input from the program authority and conclude with the corrective action to be implemented (or already implemented) with targeted completion date.

Conclusion

 

We conclude that, in all material respects, the PTD file review assessment methodology was applied correctly and consistently.

 

 

Financial Review Component:

 

The Performance Tracking Directorate also conducts complete, independent, financial reviews of many of the files selected for review. The objective of the PTD financial verification process is to ensure that expenses claimed by recipients are allowable under the agreement made with the recipient. Eligible expenses include actual and incurred expenditures that are supported by adequate documentation.

 

Two types of review approaches are used by PTD. Wherever low expenditures are reported on claims, financial reviews are conducted using supporting financial documentation provided by facsimile. Visits to recipients’ locations are conducted on the remainder. The same quality control and management review process as in the file review process is also applied to financial reviews.

 

In our audit, we selected 10 files where a financial review was conducted by PTD. We conducted our own financial review of these files, in order to satisfy ourselves that the financial review conducted by PTD met the expected objective, that relevant financial documentation is on file and that appropriate conclusions were reported adequately in PTD data-base, including relevant follow-up required.

 

In one of the ten files selected for review, we found that the financial review had not been performed as thoroughly as we expected. However, this file was reviewed at an early stage in the development of the PTD methodology and user guides and is not considered indicative of the overall quality of the work performed by PTD. Files selected from subsequent batches demonstrated a consistently high degree of diligence on the part of the reviewers.

 

In the course of this audit, we identified a potential area for improvement relating to follow up of management action plans. For files where the corrective action can be effected immediately, PTD has ample controls to ensure adequate correction is duly implemented. Where the corrective action is to be implemented at a future date (i.e., in future files of this type) or broader in aspect (i.e., a procedural change for the treatment of all files) PTD will solicit, through a formal action plan, a management response which is intended to correct the problem. However, PTD does not have in place a formal process for following up on the implementation of the required corrections reflected in the action plan, and may not be the appropriate agency within HRDC to perform such follow-up.

 

We wish to commend the PTD reviewers and quality assurance personnel for the excellent organization of the files we reviewed. The required information was indexed and easy to find. The evidence in the file backed up conclusions and recommendations made by PTD.

 

Conclusion

 

We conclude that the PTD financial review process and conclusions are, in all material respects, appropriate and accurate.

 

Recommendation 3:

 

We recommend that PTD initiate discussions for the development and implementation of a formal follow-up and tracking system for situations that cannot be immediately corrected.

 

 

OBJECTIVE 3

 

Assess the adequacy of the management control framework put in place for the PTD.

 

During the audit, we examined the various controls exercised by PTD over the sampling and file review processes to ensure that the controls are adequate, effective and designed to address any significant factors that could affect the achievement of PTD objectives. PTD developed an Operational Guide which outlines the PTD process and the quality control framework. The process is summarized below:

 

File Review Process:

 

• The Tool Use Guide (TUG) is up-dated on an on-going basis.
• All files are reviewed in accordance with the TUG.
• An enhanced Quality Assurance function is in place to ensure that PTD review findings are accurate and validated by sufficient supporting evidence.
• The results of file reviews are communicated to the program officers and local management.
• Management responses are duly reflected in the final review results.
• When incidences of non-compliance are found, there is a management review prior to responses being obtained.

Reporting Process:

 

• A reporting database was created to capture the answers to the questions in the TUG.
• Regular reports are made per the schedule outlined in the Six Point Action Plan and PTD Annual Plan.
• Draft reports are submitted to various senior officials within the Department for review and comment.

 

Based on our audit work, it is our opinion that there is a degree of over control related to the conduct of the reviews by PTD. We believe that there are efficiencies to be gained through a re-examination of the control framework relating to the number of staff involved in the file review process. In most of the cases there are two officers performing initial review. Their findings are substantiated by a supervisor’s review. This is followed by a Quality Assurance review, often by two officers, prior to soliciting management responses through a formal action plan, for observations noted which required corrective actions. Quality Assurance monitors the quality of management responses received to ensure that the issues raised have been properly addressed. The file is then sent to PTD Director for final approval and signoff.

 

We note that PTD has begun discussion of the frequency of formal reporting in order to decrease the number of formal Quarterly Progress Reports as ad hoc reporting and trend analysis on results is increased. Their recommendations will be reviewed by the PTD Advisory Committee and presented to senior management for discussion and decision.

 

Conclusion

 

We conclude that the PTD management control framework in place is effective, but that improvements in efficiency may be realized.

 

Recommendation 4:

 

PTD should continue to examine its quality control framework in order to identify areas where efficiencies may be found.

 

OBJECTIVE 4

 

Assess the extent to which reliance can be placed on information from the performance tracking system for various PTD purposes.

 

As part of HRDC commitment under the Six-Point Action Plan for Strengthening Administration of Grants and Contributions and PTD Annual Plan 2000/2001, PTD has successfully and in a timely manner met the HRDC commitment to report its results quarterly. As of March 31, 2001, three quarterly progress reports have been submitted to the Minister and made public. These progress reports convey information on the degree to which HRDC is compliant with Treasury Board and other relevant policies and guidelines.

 

In addition to the quarterly progress reports, PTD has submitted relevant internal reports that are used to communicate:

 

• the results of specific file reviews and on-site visits; and
• areas for improvement (at a regional, program or departmental level) in order to continue strengthening the administration of grants and contributions.

 

As of the March 2001 Progress Report, PTD has reported valid statistical results for the National Cumulative Analysis by Business Process. For the Regional Cumulative Analysis by Region and the National Quarterly File Review Analysis by Business Process, partial valid statistical results were reported, with a note outlining that some data were provided for information only due to the low number of some responses applicable at that time. All of these limitations are clearly noted by PTD in their reports.

 

We conducted a test of the PTD database in order to ensure that the results reported in the PTD progress report matched with the results recorded in the PTD database. Our audit work demonstrated that there was no variance between the database and the information reported in the quarterly reports.

 

Based on their own assessment, PTD has also clearly identified in their Quarterly Progress Reports specific areas that require improvement. Some examples of recommended improvement that may be found in the Quarterly Progress Reports included: finalization of agreements; disposal of assets; GST reimbursement; regional results for the five smaller regions; and program results for programs with small number of agreements.

 

PTD management also indicated that they have not yet examined sufficient files to report statistically valid results on the closeout stage. To address this situation, a special selection of files will be drawn to report on finalization issues.

 

PTD has had preliminary discussions on risk ranking the file review criteria into high, medium and low categories. This will assist the reader to focus on the most significant and key issues and will therefore improve the information provided.

 

In the financial monitoring area we observed that PTD reports on the fact of performance monitoring as planned. During the course of our audit work we observed changes made to the documentation standards for a financial monitor. They are now more rigorous than they were. In our opinion, this limited PTD’s ability to accurately assess the adequacy and reliability of the financial monitoring performed by program officers. On-site financial monitoring performed by program officers is one of the most important activities in file administration. Subsequent to our audit period, both the program areas and PTD have strengthened their processes and tools to perform and evaluate the financial side of G&C agreements.

 

Conclusion

 

Within the limitations detailed by PTD in their reports, a high level of reliance can be placed on the information produced and reported by PTD.

 

 

OBJECTIVE 5

 

Assess the adequacy of the extent of coordination of PTD activities with other monitoring bodies in HRDC.

 

We examined two areas of the PTD mandate under this objective. The first area was the extent of coordination of file review with other audit and review functions within HRDC to ensure adequate coverage without duplication of effort. These file reviews may include local and regional Program Operations Consultant (POC) reviews, Post Audit and Program Compliance reviews applied by program staff, PTD file reviews and Internal Audit and Risk Management Services audits. The second area is the extent to which PTD is contributing to the transfer of knowledge about the administration of G&C within HRDC.

 

It is clear that a key part of PTD’s mandate is coordination of PTD activities with other monitoring and post-audit functions in HRDC. PTD has done a good job of communicating its plans, methods and results to the other monitoring bodies within HRDC.

 

Subsequent to our audit, HRDC has developed and implemented a Quality Assurance Framework strengthening the management of G&C. This framework is based on modern comptrollership principles and aims at striking a balance between working with clients to achieve results and the proper administration of public funds. The framework clarifies the roles, mandate and relationships between the various groups involved in the Quality Assurance framework for G&C.

The key elements of this framework are:

 

• project monitoring;
• quality control;
• post-audit;
• program compliance (POC) functions; and
• the Performance Tracking Directorate.

 

The framework also addresses the issue of duplication and overlapping in setting the mandate and intent of each of the elements in the framework. As the Post Audit and POC functions are finalized and their activities commence, we expect that the level of coordination among the three functions/roles will reflect the quality of the groundwork that has been laid by PTD which should lead to an improved transfer of knowledge about G&C within HRDC.

 

Conclusion

 

Our audit covered PTD activities ending March 2001. At that time the other components of the Quality Assurance Framework were under development. While, in our opinion, PTD has substantially fulfilled its obligations under this part of its mandate, we are not in position to provide an opinion on the adequacy of coordination of PTD activities with other bodies within HRDC.

5.0          CONCLUSION

 

Based on our audit work and the successful implementation of management’s action plan (Appendix C) to address the issues identified in this report, we conclude that:

 

1)          the sample selection of files and file review methodologies are effective;

 

2)          the review methodology is applied fairly and consistently;

 

3)          the management control framework is effective;

 

4)          within the limits detailed by PTD in their reports, a high level of reliance may be placed on the information produced by PTD;

 

5)          while, in our opinion, PTD has substantially fulfilled its obligations under this part of its mandate, we are not in position, at this time, to issue an opinion on the adequacy of coordination of PTD activities with other bodies within HRDC.

 

The audit was designed to provide conclusions with a high level of assurance on the effectiveness of the review methodology (objective 2) and reliance to be placed on the information produced by PTD (objective 4) and with a moderate level of confidence on the other components of the audit project (objectives 1, 3 & 5).

 

HRDC senior management should continue to keep its focus on the proper management of G&C. We believe that if the four recommendations included in Section 4 of this report are implemented as discussed with management, the overall process and control framework already in place will be reinforced and improved.

 

APPENDIX A

 

 

Level of Assurance

 

 

Assurance is provided at one of two levels of assurance, a higher level of assurance and a more moderate level.

 

A higher level of assurance is provided by designing procedures so that, in the internal auditor’s professional judgment, the risk of an inappropriate conclusion is low. A higher level of assurance is only attainable through utilizing procedures such as inspection, observation, enquiry, confirmation, computation, analysis and discussion.

 

A more moderate level of assurance is provided by designing procedures so that, in the internal auditor’s professional judgement, the risk of an inappropriate conclusion is reduced to a more moderate level through procedures, which are normally limited to enquiry, analysis and discussion.

 

 

APPENDIX B

 

SUMMARY OF AUDIT OBJECTIVES AND THEIR CRITERIA

 

OBJECTIVE NO. 1 «Assess the effectiveness of the application of the (1) sample selection, and (2) file review methodologies developed by the Performance Tracking Directorate (PTD).»
SUB-OBJECTIVE NO. 1	To demonstrate that: the application of the PTD «Sample Selection» methodology is effective.
AUDIT CRITERIA	1.     The «Sampling» methodology selected by PTD has the explicit support of qualified authoritative sources including the Office of the Auditor General;
	2.     The «Universe» of all HRDC Grants & Contributions files, that should be subject to PTD performance monitoring, is reliable (i.e. is both complete and accurate) and relevant;
	3.     When PTD obtains data from FAS Systems on the «monitoring universe», it utilizes a (standard) request that does not limit or bias the data to be supplied to it;
	4.     PTD has requested a description of the controls used by FAS Systems to ensure that the transfer of «Universe» data to PTD is reliable; and that FAS has demonstrated to PTD that these controls ensure reliable transfer of data;
	5.     On each occurrence of transfer of data to PTD, FAS has provided explicit assurance that all transferred data is both relevant and reliable;
	6.     For each Quarterly File Review exercise undertaken by PTD, an authoritative departmental source, external to FAS, has provided specific and clear direction on reporting objectives to PTD in a timely manner;
	7.     For all Quarterly Review exercises undertaken by PTD, PTD should have applied the appropriate formulas and methodology to match the specific reporting objectives and select the appropriate sample of files for review;
	8.     If, in any of the above exercises, PTD has altered the number of files or the specific files selected that would make the sample of files used by PTD differ from the sample that would have been identified by the appropriate formula, such action should have the specific support of authoritative sources that an appropriate sample has not been contaminated.
SUB-OBJECTIVE NO. 2	To demonstrate that the application of the PTD «General File Review» methodology is effective
AUDIT CRITERIA	1.     PTD has requested on a timely basis that copies of all files identified in the file sample selection exercise are submitted from the appropriate HRCC Offices;
	2.     PTD takes steps to ensure that all files received from HRCC Offices match the files requested; and they also take steps to ensure that the files and their contents are complete with all relevant documentation;
	3.     PTD periodically submits and obtains timely approval (from departmental management) for a plan which identifies sufficient file review resources needed to properly complete any and all Quarterly file review exercises on a timely basis;
	4.     PTD has taken steps to ensure that controls are in place in its «File Review Data Base System» (SNIFURS) that appropriately controls the ability of PTD and other individuals to access and change the content of that Data Base;
	5.     PTD has reviewed, and received the approval of senior management in HRDC for, its approach for ensuring that the list of questions included in any file review questionnaire adequately reflects all significant management and third party concerns re the proper administration of Grant and Contributions programs;
	6.     The rationale for all changes to questions for any Quarterly File review exercise by PTD has been reviewed with and approved by HRDC management in a timely manner;
	7.     All PTD decisions to allocate specific files to specific «File Review Officers» for review have been based on criteria which ensures that these FROs do not have a conflict-of-interest in conducting the file reviews;
	8.     Responses to all key questions for all files reviewed are supported in the audit review working papers by references to appropriate evidence in the file documentation received from the HRCC Offices with a supporting rationale referring to specific PTD directives or guidelines;
	9.     All questions for all files included in any Quarterly review exercise are responded to and all responses by all file reviewers reflect a consistent interpretation of PTD directives or guidelines;
	10.  The audit working paper files, for all files reviewed, clearly indicates that the input and advice of the local HRCC Project Officer was sought by PTD review staff;
	11.  There is evidence on the working paper files of «Supervisory» Review challenge to file review officer responses to questions. There is also evidence that the challenges are consistent with direction provided to file review officers by the PTD Tool Use Guide and other desktop guides and controls;
	12.  There is similar evidence on the working paper files with respect to challenges to responses to questions from the file review officers by the «Quality Control Officers» (QCOs);
	13.  All QCO comments are consistent for the same Question for all QCOs;
	14.  All final comments on file questions, as negotiated between the FROs and the QCOs, are consistent with the guidance provided by all the various PTD directives and guidelines and the evidence on the file documents;
OBJECTIVE NO. 2 «Assess the effectiveness of the application of the file review Report methodology developed by PTD»
SUB-OBJECTIVE NO. 1	To demonstrate that the application of the PTD file review Report methodology on the results of the review of the originating HRCC Office file documentation (the «Internal Report» process) is effective.
AUDIT CRITERIA	1.     All files being reported on are included in the «sample» of files selected by PTD for review and reporting and all files in the sample are reported on;
	2.     The local HRCC Project Officer and management are given an opportunity to review and comment in a timely manner on the draft «summary of results» (SOR)report;
	3.     The comments and/or additional evidence provided by the HRCC Project Officer or management are duly considered by PTD and the SOR Report has been modified or amended as appropriate;
	4.     The final SOR Report reflects an explicit and clear assurance or commitment from HRCC management to take action (or reflects that action has been taken or is being taken) to address any significant deficiencies noted in the SOR Report. The report may also reflect comments from HRCC management which puts significant review issues or findings into an appropriate perspective;
	5.     Action plans to address deficiencies include a specific deliverable or measurable change to be effected by a specific date at a specific cost as applicable and appropriate;
	6.     PTD has followed up, or has a plan to follow-up which will assess and report, on the status of action plans until all actions have been confirmed to have been concluded satisfactorily for all «Internal Reports»;
SUB-OBJECTIVE NO. 2	To demonstrate that the application of the PTD file review Report methodology on the Summary Report on the results of all file reviews (the «Quarterly Report» process) is effective.
AUDIT CRITERIA	1.     All «Internal Report» results from the sample of all files selected for review in the PTD Quarterly Review process are included and reflected in the analysis of all questions reported on in the «Quarterly Report»;
	2.     Appropriate HRDC PTD and G&C Committee members are given an opportunity to review and comment in a timely manner on the draft «summary of results» (SOR) «Quarterly» report;
	3.     The sample of files reported on satisfies the minimum number needed to validly represent findings of a «National», «Regional», «Program» or «Other» basis as appropriate for all questions;
	4.     Where data from results of «Internal Reports» is used in «Quarterly Reports», that the criteria #2 through #5 for «Internal Reports» are satisfied;
	5.     The reporting objectives, and the sample size selection appropriate to those objectives, are the same for each report in the series of «Quarterly Reports» to ensure the validity and comparability of conclusions per issue or question;
OBJECTIVE NO. 3 «Assess the adequacy of the management control framework put in place for the PTD».
SUB-OBJECTIVE NO. 1	To demonstrate that the Controls exercised by PTD (the «Management Control Framework») over the «Sample Selection», «File Review» (including «Audit Review», «Supervisory Review», Quality Control Review», and «Management Review»), and «Review Reporting» (including «Internal Reports», and «Quarterly Reports») processes are adequate and effective.
AUDIT CRITERIA	1.     The control mechanisms or tools employed by the PTD are designed to address all of the significant risks that could affect the effective and efficient achievement of PTD objectives;
	2.     All of the control mechanisms or tools employed by the PTD are demonstrated to effectively achieve their objectives in addressing specific risks as individual tools or as a set of tools;

 

 

SUB-OBJECTIVE NO. 2	To demonstrate that the PTD «Quality Assurance» function is effective.
AUDIT CRITERIA	1.     All PTD «working tools» are appropriately updated at the beginning of each Quarter;
	2.     Local (HRCC) Program Officers (POs) are routinely contacted for confirmation / feedback regarding facts / information and review results on every file selected for the file review sample;
	3.     All feedback of PO s with respect to file review facts and results are appropriately considered by PTD and reflected in the final review results for all files;
	4.     All «Supervisory Reviews» of File Review Working Papers routinely and explicitly address the four primary objectives of the SR reviews for all files reviewed;
	5.     All independent «Quality Control» (QC) reviews ensure that the file review Working Papers properly support the audit report («Statement of Results» - SOR) and that all necessary auditing procedures have been performed (in accord with Chapter 420 of the QA Review Manual for Internal Auditing of the IIA. All QC reviews determine that review findings: -       are accurate, and -       are validated by sufficient supporting evidence
	6.     All «Management Reviews» (MR s) address the three primary objectives of the MR review;
	7.     All «Internal Reports» are sent to local / regional offices for comments and properly and appropriately reflect and address responses received;
	8.     Every «Quarterly Report» incorporates the findings of all «Internal Reports» as appropriate;
	9.     The draft «Quarterly Report» is submitted on a timely basis to: -       the PTD Advisory Committee; and -       the G&C Steering Committee for review and advice and the final «Quarterly Report» appropriately reflects that advice.

 

 

OBJECTIVE NO. 4 «Assess the extent to which reliance can be placed on information from the performance tracking system for various PTD purposes».
SUB-OBJECTIVE NO. 1	To demonstrate that the PTD File Review results are reliable enough to support PTD management decisions for various purposes including: -       reporting to 3rd parties; -       general management; -       audit planning; and -       (PTD) staff training
AUDIT CRITERIA	That audit objectives one (1) to three (3) are demonstrated to have been satisfied; (If all of the important processes and controls by which PTD staff generate information regarding the performance and administration of all HRDC Grant & Contribution programs are demonstrated to be effective and reliable does this not then imply that the information generated from the use of these processes and controls could in turn be considered to be reliable - thus complete and accurate)   If this is so, then the only other limitation on the use of PTD review results would be to ensure that the information to be used for any particular management decision purpose is relevant or appropriate to the purpose of that management decision.
OBJECTIVE NO. 5 «Assess the adequacy of the extent of coordination of PTD activities with other monitoring bodies in HRDC».
SUB-OBJECTIVE NO. 1	To demonstrate that the of PTD «coordination» activities with other «monitoring» bodies in HRDC is adequate and appropriate.
AUDIT CRITERIA	1.     The plans outlining the type and extent of PTD «coordination» activities with respect to other «monitoring» bodies in HRDC and the actual activities themselves, do not duplicate the planned «coordination» activities of any of the other bodies;
	2.     PTD «coordination» plans have been approved by HRDC management responsible for overall «monitoring» activities in HRDC regarding the administration of Grants & Contributions programs in that they are deemed to satisfy PTD objectives with respect to the «coordination» of «monitoring» activities;
	3.     Actual PTD «coordination» activities are carried out as per their approved plans.