Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

May 1, 2002

Introduction

This formative evaluation was undertaken to assess the relevance, design and delivery, and short term impacts of the Program for North American Mobility in Higher Education, and the Canada-European Community Program for Co-operation in Higher Education and Training, which are administered under the International Academic Mobility Initiative. The evaluation process was initiated in 2000, five years after the programs started operation, and examined projects that were approved between 1995-1998.

The Learning and Literacy Directorate (LLD) has reviewed the formative evaluation report that was recently completed. While sample sizes were not sufficient to be statistically reliable, nonetheless, we are still pleased with the overall findings and the insight that was provided on the IAM Initiative.

Positive Findings

Many of the findings throughout the report are positive. For example, participating faculty and students confirm that there is a continuing need to encourage international linkages and mobility activities within Canada's post-secondary institutions, and that IAM continues to be relevant in providing funding and a framework for supporting these activities. Faculty surveyed for the evaluation noted that IAM provides unique linkage and mobility opportunities that go beyond traditional bilateral exchange arrangements, and encourage innovation in international learning programs.

The evaluation report indicates that projects are involving not only students who study abroad at partner institutions, but also a high number of students who do not study abroad. These non-mobile students benefit from the international dimension that the project brings to their education through, for example, taking courses for which the curriculum has been jointly developed with international partners.

For the students who do study abroad, the financial stipend that they receive under IAM program is intended to assist with the incremental costs of studying abroad, above and beyond what the student would normally pay to pursue their education at their home institution. LLD was pleased to note that the average stipend that surveyed students reported receiving covers 77% of the incremental costs that they identified. This is important, since 63% of surveyed students reported being recipients of some form of student loans.

Faculty reported considerable success in reaching agreements for credit transfer and recognition, and half of these faculties indicated that these agreements would benefit future students. Surveyed students reported little difficulty in receiving credit for their study or internships abroad. As well, most faculties reported that collaboration with their partners has continued since their IAM funding ended.

Surveyed students reported improved language skills, knowledge of other cultures, and improved knowledge of international issues as a result of participating in IAM projects. Students also reported personal benefits including better ability to adapt to new situations.

Areas for Improvement

The IAM formative evaluation report also identified several areas where improvements could be made, particularly in the clarity of IAM's application guidelines and selection processes. IAM guidelines, application procedures and selection processes are developed in co-operation with our international partner governments, and are reviewed annually to ensure clarity. While many improvements have been made since the early days of the initiative, we continue to refine the guidelines and selection processes based on comments received from our sponsors at Project Directors' Meetings and from members of our External Review Committee.

The report also identifies measures that would assist IAM in developing strong methodologies for a future summative evaluation of the program. These measures included continuous monitoring of students and faculty during and after project implementation, the use of a standard annual report template for project directors, and obtaining informed consent to contact students in the future for program evaluation purposes.

IAM is already in the process of implementing many of these measures in response to internal program management review processes that occurred before or concurrently with the formative evaluation. These review processes included the Program Management Initiative (2000), the renewal of the Learning Initiatives Terms and Conditions under which the program operates (2000), and an audit that was conducted by HRDC's Internal Audit Bureau (2001).

A Results-Based Accountability Framework, outlining measurable performance measures for IAM's programs, was developed as part of the renewal of the Terms and Conditions. As part of this process, IAM has developed student surveys and annual report templates that will permit continuous collection of comparable data on student experiences and project activities. Responses will be entered into a database that can be used in future evaluations. These tools are now in the process of being implemented and we will be in constant contact with our partners to monitor the extent to which they are user friendly and can be easily implemented.

IAM has drafted a participant information form that students will voluntarily complete to provide consent for HRDC to contact them in the future for the purpose of evaluating IAM's programs. This form will be reviewed by HRDC's privacy office, as well as our partnering institutions, to ensure that it complies with privacy laws both of the federal government and the post-secondary institutions that will be involved. Once finalised, this will assist in developing future survey frames.

Conclusion

At the first meeting of the IAM Steering Committee on March 23, 2001, a decision was made to revise the methodology that had been included in the Terms of Reference. Case studies and key informant interviews were replaced with a greater emphasis on the student survey, in an attempt to gather preliminary information on short-term benefits. While this change in methodology provided a series of "lessons learned" that will guide the development of future summative evaluation survey methodologies, LLD notes that it will also be important to include methodologies such as case studies and key informant interviews in future evaluations of IAM. This will be particularly important if establishing a statistically reliable sample of faculty and students continues to prove to be difficult.