Audit of International Academic Mobility Initiative

The 2000-2001 HRDC Internal Audit Bureau Resource Utilization Plan included cyclical audits of Grants and Contributions programs. The audit of the International Academic Mobility (IAM) Initiative was selected on the basis of the July 2000 renewal of the Learning Initiatives Terms and Conditions with the Treasury Board Secretariat.

HRDC’s Learning and Literacy Directorate manages the IAM Initiative centrally at NHQ. Through various programs, the Initiative supports the acquisition of international skills and competencies by Canadian post-secondary students, thus preparing students for work in an increasingly global economic environment. The IAM Initiative consists of three programs: the Canada- European Community Program (ECP), the North American Program (NAP) and Special Initiatives. The IAM Initiative was awarded $3.4 million in program funding for 2000-2001.

The objectives of the audit were to assess the adequacy of the management control framework and to determine whether major risks are effectively managed. The audit was conducted at NHQ from November 01, 2000 to February 28, 2001 and included interviews, analysis of program documentation and project file reviews. During this period, the IAM Initiative was undergoing or had recently undergone major transition changes to comply with a number of HRDC’s revised administrative and managerial practices for Grants and Contributions programs. At the time of the audit, IAM’s management committed to completing the following processes, which were in progress:

The following recommendations are addressing issues identified during the course of our work:

1.      the implementation of a process to report periodically on project outcomes and results;

2.      the update and use of the IAM Initiative’s database (once fully implemented) and the update of the CMS program data to provide accurate financial information;

3.      the implementation of a quality assurance process for the review of the IAM Initiative project files at the program or directorate level;

4.      the training of project officers on performing the monitoring function, with emphasis on financial monitoring; and

5.      the development of an explicit monitoring methodology: the use of standardized forms such as a monitoring checklist and activity and financial monitoring reports is strongly encouraged.

The management responses to these recommendations are reported under Section 3.0 of this document. A Corrective Action Plan is also included under Appendix C.

It should be noted that corrective action was started as soon as management was made aware of the issues raised during the course of this audit.

The 2000-2001 HRDC Internal Audit Bureau Resource Utilization Plan called for audits of Grants and Contributions (Gs&Cs) programs, including the International Academic Mobility (IAM) Initiative.

The International Academic Mobility Initiative

HRDC’s Learning and Literacy Directorate (LLD) manages the IAM Initiative centrally at NHQ. Through various programs, the Initiative supports the acquisition of international skills and competencies by Canadian post-secondary students, thus preparing students for work in an increasingly global economic environment.

The main programs of the IAM Initiative provide contributions for periods of 3 or 4 years to co-operative projects between a consortium of universities and colleges in Canadaand abroad. Project activities include international student placements, development of joint curricula, and the use of new learning technologies to provide an international learning dimension for non-mobile students. Participating institutions are required to make a financial commitment and to also consider a means of sustaining their co-operative activities beyond the funding period. The IAM Initiative was awarded $3.4 million in program funding for 2000-2001. The IAM Initiative comprises three programs: the Canada- European Community Program (ECP), the North American Program (NAP) and Special Initiatives.

Audit Overview

The audit objectives were to assess the adequacy of the management control framework and to determine whether major risks were effectively managed. Focus was on accountability for results and quality of monitoring. The audit was designed to provide conclusions with a moderate level of assurance (Appendix A). Elements of the management framework that were examined can be found throughout the report and in Appendix B.

The audit was conducted at NHQ from November 2000 to February 2001 and focussed primarily on program activity beginning on or after February 1, 2000. Audit findings are based on enquiry, discussion, project file reviews and program documentation analysis.

The project file reviews were performed using a detailed assessment questionnaire which covered six aspects of the business process related to project life cycle: (1) application, (2) assessment, recommendation and approval, (3) agreement, (4) payments and advances, (5) monitoring, and (6) close-out. The file sample consisted of all files, which started on or after February 2000. IAM management provided 20 files for examination (7 European Community, 10 North American and 3 Special Initiatives) and all six aspects of the business process were examined. An additional file selection consisted of files where on-site monitoring visits had been conducted at the time of the audit for the purpose of reviewing monitoring activities only. IAM management identified 28 files, “the old files”, with most of these files having a starting date prior to February 1, 2000.

The findings are presented in accordance with the audit criteria, which are also described in Appendix B.

3.1     Program Mandate / Strategy

3.1.1  Program objectives are clearly stated, measurable, and expected outcomes and results are clear.

Overall, the IAM Initiative’s terms and conditions as reflected in the individual contribution agreements include a clear statement of its objectives and expected outputs. The Accountability for Results subsection (3.2.2.) of this report presents more detail on the subject of measurability of outcomes.

3.1.2  The program’s policies, procedures and operational guidelines are clearly defined and consistently applied.

At the time of the audit fieldwork, the LLD/IAM Initiative Unit was undergoing major transition to comply with HRDC’s revised administrative and managerial practices for Gs&Cs programs. Within the IAM Initiative, there were a number of policies, procedures and guidelines still under development. Management expected the North American Partners’ Guide to be completed by May 15, 2001 and the Annual Report Template to be completed by September 30, 2001. The Special Initiatives Guidelines and Final Report template, under development, were to be completed before the next call for project proposals. The completion of the Standard Contribution Agreement was awaiting Financial and Administrative Services and Legal Services input or approval for specific agreement clauses. The completion of this agreement template was also expected prior to the next call for project proposals.

Program officials said the Special Initiatives projects were still in a pilot stage. IAM’s management will ensure that eligibility criteria are defined and applied in the next call for project proposals. For both the North American and the European Community Programs, IAM project officers were documenting on file the rationale for approving individual projects. We encourage project officers to continue this practice with the Special Initiatives program.

3.1.3  There is an understanding of the risks facing the program and a risk mitigation strategy is in place.

Risks were discussed within the Program Management Study (currently known as the Program Management Initiative) and the development of the Results-Based Accountability Framework for IAM. The Program Management Study identified specific strategies to address the issues identified. Because of the constant evolution of the programs’ internal and external environments, we encourage the LLD to perform periodic assessments of the IAM Initiative’s risks. It is our opinion that periodic risk assessments and the development of mitigation strategies would help to ensure the Initiative’s programs continue meeting its objectives.

3.1.4  The program’s promotional effort is sufficient to ensure that eligible institutions are made aware of the IAM program.

The IAM Initiative maintains a database containing contact information for universities and colleges across Canada. Annually program officials initiate a ‘call-out’ to these institutions inviting them to submit proposals. In our view, this call-out system is a good process for reaching eligible institutions.

3.2     Accountability for Results

3.2.1  The roles and responsibilities are clearly defined and transparent.

The roles and responsibilities of both HRDC and the institutions were clearly defined in contribution agreements. In the context of multiple partner agreements such as the North American program, we consider it a good management practice to define in writing the definition of the roles and responsibilities of the partners in a Memorandum of Understanding (MOU). We encourage IAM’s management to consider incorporating this practice for the EC program as well.

3.2.2  The project outcomes are measured against planned results and are reported in a timely manner.

The LLD has met the Treasury Board requirement to have an accountability framework completed by October 31, 2000. The strategy for measuring results against the framework is clearly stated. The LLD management said that its implementation would commence in the fall 2001.

At the time of the audit, the LLD was not reporting on project outcomes and results. We believe the absence of a common reporting practice prevents the LLD from compiling results in a comparable manner within individual IAM programs and between IAM programs. LLD management expected the implementation of the Annual Report templates to commence in the fall of 2001, which will help to rectify this situation.

Recommendation # 1

The LLD should institute a process to report periodically on project outcomes and results.

Department’s response:LLD agrees with this recommendation and has taken action.

The IAM Initiative has met the Treasury Board requirement to have an accountability framework completed. This framework was approved by the LLD and Treasury Board in November 2000, and includes indicators for measuring program success. These indicators will be gathered through a series of information sources, such as annual project reports and regular student surveys.

A formative evaluation of the program has recently been initiated to examine the program relevance, short-term impacts, design, delivery and monitoring mechanisms. This will allow IAM to collect information on the contribution that the IAM initiative has made to students’ academic and professional development. The evaluation report is expected in the fall of 2001.

The achievement of results for the IAM initiative will be reported to Parliament and Central Agencies through HRDC reporting mechanisms such as the Report on Plans and Priorities and the Departmental Performance Report. Results will be posted on the IAM web site.

3.2.3  Appropriate administrative and financial controls have been designed and implemented.

We believe that the LLD has established a good approach to review, evaluate and select IAM projects, considering the international complexities inherent to this Initiative. The three project evaluation stages for IAM’s NA and EC components of: 1) meeting IAM’s eligibility criteria, 2) ranking the projects by members of Canada’s academic community, and 3) the final selection of projects in collaboration with IAM’s international partners, are effective in ensuring the best projects are selected.

In most EC projects reviewed (start date on or after February 1, 2000), the new agreement template was used and milestones and business cases were included. Payment and advance processes were for the most part appropriate with the exception of the items listed below.

The LLD has corrected or was in the process of addressing most IAM administrative and financial control issues observed during the audit. The following is a summary of the administration and control issues observed:

• Administrative issues were found with respect to project start and end dates. Some agreements for the year 2000 allowed for an effective project start date of more than 30 days prior to the date the agreement was actually signed. Furthermore, agreement amendments were initiated to extend the life of projects after the project end date. Even though an overpayment could not be established on account of HRDC’s obligation to pay, technically, payments were made before or after agreement dates.
• Verification of recipients’ outstanding debt prior to the ratification of the agreement was lacking in some cases. This situation was identified in the 2000 European Community files and was subsequently corrected.
• It is our opinion that financial monitoring of projects required improvement in the majority of files that we reviewed. The audit also found insufficient documented evidence of off-site monitoring. The Monitoring Section (3.4) of the report provides more information on these issues.
• Accurate financial information on IAM projects was not readily available. We found that the Corporate Management System (CMS) financial data did not reflect IAM’s actual financial situation. For example, it was not possible to match the commitment value in the project file to the commitment data entered in CMS for the same project. Furthermore, the forecast information was not entered for the project life cycle. At the time of the audit, the LLD did not have direct data entry access to CMS. The financial data was sent to Accounting Operations, Financial and Administrative Services (FAS), for input into CMS.

In our opinion, it is critical that the IAM Initiative’s budget, commitment and expenditure information be kept up to date. While we were informed that action has been taken to correct this situation, the recommendation that follows aims at ensuring that the required attention is given to financial management.

Recommendation # 2:

The LLD should ensure that the IAM Initiative’s database (once fully implemented) contains up-to-date information and work closely with Accounting Operations, FAS, to ensure that the data is inputted into CMS

Departmental Response: LLD agrees with this recommendation and has taken action.

Until recently, most of the tracking of financial information on the IAM Initiative was done manually. Significant progress has been made over the last six months to automate the tracking and reporting of all financial information. Data has now been entered into the IAM database and work is underway to reconcile this information with the information included in the departments financial system, the CMS.

The CMS system is managed by Finance and Administrative Services, and they will work closely with IAM to ensure that all commitment information is entered into the system on a timely basis, that all forecast information for the project life cycle is kept up to date, and that commitment and expenditure information is current.

3.2.4  There are processes in place to clarify policies, resolve issues, and ensure good communications across all program stakeholders.

Aside from normal day-to-day communications, IAM officials stated that good communication is ensured amongst all program stakeholders through the annual Project Director Meeting. This meeting provides an opportunity for both government and institutions to discuss and resolve program-related issues and to share best practices among program participants.

The LLD is represented at the Operational Policy Committee. This Committee serves as a forum for discussion of crosscutting operational policy issues and leadership regarding operational and policy issues related to Grants and Contributions. This committee enables the communication between programs, including the sharing of best practices and lessons learned.

3.3     Supporting Program Capability

3.3.1  The program has access to needed resources, information, skills, tools and training to ensure successful delivery.

Resource planning is done through the yearly business planning process. Currently, the program is being supported with supplementary resources to strengthen program management.

The IAM Initiative database was developed to capture information on the measures identified in IAM’s accountability framework. As such, information on the administrative and financial status and performance of each project will be included. At the time of the audit, this database was not yet operational. We were informed that the required data from active files would be entered by June 30, 2001. Thereafter, the plans call for IAM project officers to maintain the database.

As part of the Six-Point Action Plan implemented in response to the 1999 Program Integrity / Grants and Contributions Audit, a revised training policy was issued, identifying mandatory courses for program delivery staff. At the time of the IAM audit, most of the program staff had received the two-week Gs&Cs departmental training course.

Coaching and feedback provided through quality assurance functions are also important components of training. In response to the Six-Point Action Plan, the Gs&Cs Project Team developed a Quality Assurance Framework to strengthen the management of Gs&Cs programs. LLD was in the process of determining how best to implement a quality assurance framework for its programs at the time of the audit. We believe that the implementation of the Quality Assurance Framework that provides for the quality control of 15% of projects will reinforce the training of IAM’s staff.

On a more specific note, the review of ‘older files’ indicated a strong need for additional training in project monitoring. The monitoring section (3.4) of this report presents more detail on this subject.

Recommendation #3:

The LLD should implement a quality assurance process for the review of the IAM Initiative files at the program or directorate level.

Department’s response: LLD agrees with this recommendation and has taken action.

The Department has acknowledged the need for ongoing quality assurance to ensure appropriate file and project management. In order to ensure that this is done on a broad basis within LLD, a new senior operations consultant has been hired on a full time basis to conduct a quality review of 15% of randomly chosen files. To complement this, the Performance Tracking Directorate has also been over sampling LLD active contribution files in order to ensure an adequate review of small programs, identify concerns, and make recommendations on areas for improvement.

These are permanent processes and structures to detect and correct problem files, and to identify and correct deficiencies in operational policy, procedures, and employee knowledge and skills.

As a result of the recommendations made in the IARMS Audit Report, a post-audit quality assurance process was undertaken to address specific issues that were identified in the IAM files.

3.3.2  There is a communications plan that ensures consistency and congruence with other programs.

The consistency and congruence with other programs was reviewed through the Program Management Study and continues to be accomplished through LLD participation on the Operational Policy Committee.

3.4     Monitoring

3.4.1  Policies and systems are in place to measure and report on project and program performance.

Issues pertaining to the availability of policies and systems to measure and report on project and program performance have already been mentioned in this report (subsections 3.2.2, 3.2.3 and 3.3.1). LLD management stated that the following actions were underway:

• A results-based accountability framework has been developed. The IAM Performance Reporting Strategy is in its final stage of preparation and implementation will commence in the fall of 2001.
• The refining of database reporting capabilities is underway in order to ensure that program information is readily available at all times by June 30, 2001.
• The reconciliation between the information in the IAM database and the data in CMS will be completed by June 30, 2001.

3.4.2  The contribution arrangements are being effectively monitored to ensure that funds are spent according to the terms and conditions of the agreement.

The audit included a review of the monitoring practices, which was performed on the “old files”. We reviewed the monitoring practices from the date of the active file review completed in February / March 2000. We expected to find completed monitoring risk assessment forms, explicit monitoring methodology, documented evidence of activity and financial monitoring, and copies of annual reports from the recipients.

In lieu of individual project monitoring plans, the LLD had developed an overall monitoring plan for projects under the IAM Initiative. The plan included a strategy for multiyear projects with higher education institution and for projects of shorter duration. The plan called for one visit to every active IAM project before the end of June 2001. Several on-site visits had been conducted at the time of the audit.

The project monitoring reports evidenced that key activities were consistently addressed during the visits. Several reports however did not include the names of participants. In our opinion, participants’ information is required to track program expenditures.

In the majority of files reviewed, it is our opinion that financial monitoring activities required improvement. Attention should be paid to the monitoring methodology and the documentation of financial records and expenditures verified. Further, in-kind expenses budgeted by the recipient were not reviewed as part of the monitoring processes to ensure that they were actually incurred. It is our opinion that program officers would benefit from having a clear and comprehensive financial monitoring methodology that includes the use of a financial monitoring checklist and a monitoring report to guide them through this process.

There was little documented evidence of off-site day-to-day monitoring being done. Further, the lead Institution’s Annual Reports were not consistently on file and were not submitted consistently for every year of a given project. Agreements stated that the “recipient(s) shall provide an activity report at the end of each year of the project”.

Recommendation# 4:

The LLD should ensure that IAM project officers are fully trained in project monitoring, with emphasis on financial monitoring.

Department’s response: LLD agrees with the recommendation and has taken action.

IAM acknowledges the need for additional training to be provided to all IAM project officers, particularly in the area of financial monitoring. While most IAM officers participated in the two-week Gs and Cs departmental project officer training course, many were new to program administration and required additional information sessions. As employee turnover necessitates an ongoing commitment to training, training will be an important part of the new Program Operations Consultant’s role.

Recommendation # 5:

The LLD should develop an explicit monitoring methodology: the use of standardized forms such as a monitoring checklist and an activity and financial monitoring report is strongly encouraged.

Department’s response: LLD agrees with this recommendation and has taken action.

IAM acknowledges the need for an explicit monitoring methodology, the use of standardized forms, such as a monitoring checklist and an activity and financial monitoring report. IAM officers had developed and used a customized monitoring methodology that put the emphasis on monitoring project activities. However, this addressed predominately on-site visits and lacked certain details from a financial perspective. While off-site monitoring activities were ongoing, at times, insufficient documentation was placed on the file.

Standardized forms were developed during the fall of 2000 and were put in place in January of 2001. File management activities after that date have benefited from the use of these tools.

A clear monitoring methodology for the IAM program has now been developed and implemented. This methodology applies the risk management principles and builds on the monitoring methodology developed and implemented by the department as a whole. Included in this are standardized forms for the IAM program including a monitoring checklist, and activity and financial monitoring reports.

During the course of our work, we recognized that progress was being made on a number of actions to improve the management of the program and we also identified a number of areas that required attention.

Subject to the completion of outstanding commitments and the action plan to address issues identified in this report, we conclude that the control management framework is appropriate to support the achievement of the IAM Initiative objectives and that major risks are effectively managed.

This conclusion is based on the audit work performed, which was designed to provide conclusions with a moderate level of assurance (see Appendix A).

LLD’s overall comment: LLD takes seriously all the issues raised in the Internal Audit and Risk Management Services' report.

LLD and the IAM Initiative are determined to build on the improvements that have already been made in the administration of these contribution agreements. IAM will also ensure that all files are in compliance with policies and procedures relating to the good management of contribution agreements. IAM will continue to work with all interested parties to ensure that the right balance is achieved between responsive client service and prudent management of taxpayers' money.

IAM has made good progress with respect to ensuring that staff members have a good understanding of file management processes and policies related to contribution agreements. These improvements will be sustained and will continue to be strengthened to address existing and emerging issues.

IAM is pleased that the Internal Audit and Risk Management Services recognizes that:

• the overall IAM initiatives’ terms and conditions as reflected in the individual contribution agreements include a clear statement of its objectives and expected outputs;
• the annual call for proposal system is a good process for reaching eligible higher education institutions;
• IAM has established a good approach to review, evaluate and select IAM projects;
• IAM has corrected or was in the process of addressing most IAM administrative and financial control issues by the end of the audit;
• IAM officials ensure good communication amongst all program stakeholders through the annual Project Directors’ Meetings; and
• IAM is represented at the Operational Policy Committee to ensure sharing of best practices and lessons learned.

IAM has undergone major changes to comply with a number of HRDC’s revised administrative and managerial practices for Grants and Contributions programs. The IARMS report acknowledges that a broad range of initiatives were in progress at the time the audit was conducted. IAM’s management is committed to completing the following processes:

finalizing the North American Partner’s Guide and the Annual Report Template;

completing the Standard Contribution Agreement to include the required clauses related to the new TB and FAS policies;

implementing the IAM Results-Based Accountability Framework ;

defining the eligibility criteria for Special Initiatives projects before the next call for proposals;

ensuring that Special Initiatives Guidelines and Final Report template are finalized before the next call for proposals; and

finalizing and maintaining IAM’s internal database and making appropriate links to HRDC’s CMS database.

In summary, IAM confirms its commitment in making fundamental and permanent improvements to the management and administration of this contribution program

APPENDIX C

RESPONSE TO IARMS’ RECOMMENDATIONS - INTERNATIONAL ACADEMIC MOBILITY