Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The summative evaluation of the OLT focuses on assessing whether the program objectives have been met through projects funded by the LTI, CLN, and LTW initiatives. Specifically the evaluation² was designed to determine the following:

• If the objectives of the OLT are still valid and relevant;
• How important OLT funding was to the establishment of the projects; and
• The extent to which the OLT contributed to the development and use of learning technologies by reaching target groups, creating partnerships and developing sustainable projects.

Evaluating partnership styled programs designed to achieve their objectives through groups and organizations beyond the direct influence of the program presents a number of practical challenges. In the case of the OLT, the program's objectives are achieved indirectly through funding projects undertaken by project sponsors and partners. The projects and project outputs are highly diverse, and their final outcomes/impacts depend on additional persons, groups and organizations that are also beyond the direct influence of the program. This means, in specific instances, the impacts (e.g. on target groups, partnerships and communities) are not easy to track or measure.

The evaluation methodology developed for the OLT evaluation recognized there challenges and attempted to address them, for example, by:

• Emphasizing the use of multiple lines for all evaluation questions;
• Employing both qualitative and quantitative data collection methods; and
• Making full use of case studies and site visits to provide evidence on project targeting, outputs and the usefulness of project activities and outputs.

The main components of the evaluation approach are presented below.

3.1 Document Review

A document review was conducted to provide a context for the summative evaluation. The document review examined the current state of learning technology theory and practice within and outside Canada. Additionally, the document review addressed key issues and lessons learned from domestic and international learning technology program development, impacts and outcomes of learning technology implementations, and trends in learning technology.

3.2 Administrative Data Review and Analysis

Data review activities were undertaken with three data files, one for each of the three funding initiatives (LTI, LTW and CLN). The data review of these files included projects that were funded and projects not recommended for funding (i.e., non-funded projects). For the purposes of the evaluation, withdrawn projects were not included in the analysis.

There was no missing information at the project level. Although there were only a limited number of variables available in each database when the evaluation started, more were added as the evaluation progressed. The lack of a full range of variables for each project (e.g. including users/participants) is not unique to the OLT and has been encountered in the evaluation of many other HRDC programs.

The specific information available for the projects funded by each of the three funding initiatives included project title, location, project sponsor name and address, project start and completion date, project type (pilot or developmental) and status (active completed, withdrawn, non-funded). The amount of funding received by the project from the OLT was also available.

Prior to analysis, data coding was completed for province, organization type (e.g., non-profit organization, university, college, etc.), initiative (CLN, LTI, LTW), and project status (active, completed, closed, and non-funded). All organizations were placed into only one category, regardless of whether there was reason to code them into two or more categories. These additional data elements provided a basis for examining the distribution of funding applications and funding levels.

3.3 Case Studies

Case studies were conducted on ten OLT funded projects. Three case studies were conducted with LTI funded projects, three with LTW funded projects, and four with CLN funded projects. The LTW is the most recently implemented initiative and, therefore, most of the LTW case-study projects were in the early stages of completion. This means that the case study evidence for these projects should be considered preliminary in nature.

The case studies consisted of a one or two day site visit with each of the ten projects. Locations for the site visits included Vancouver, Victoria and Lumby (B.C.), Edmonton (Alta.), Winnipeg (Man.), London, Ottawa and Toronto (Ont.), Montreal (Que.), and Halifax (N.S.).

Site visits assessed a range of different aspects of the OLT program, such as partnerships developed in the course of OLT projects, deliverables/outputs of learning technology projects, sustainability, and lessons learned from the OLT project. During the site visit, the facility was toured and supplementary materials provided by the sponsors were reviewed. At that time, any hardware obtained for the project and any software developed or used for the project, such as CD-ROMs and websites, were observed.

All of the case studies collected information on project targeting and the types of outputs/activities of the project. The case studies also collected information on project impacts and often included structured interviews with project partners.

Each case study also attempted to obtain information directly from end-users through interviews/focus groups. It was difficult to complete this part of the case study because of a lack of tracking end-users at the project level. In the end, however, focus groups (each with nine participants) were arranged with end-users at three of the four CLN case study sites.

• Vancouver Community Network: Project to provide Internet tools/access and training at community sites, and targeted on individuals with low income and literacy skills. The end-user focus group was conducted with seniors;
• Monashee Learning and Training Centre: Project to establish a hub/centre of service in the remote community of Lumby to facilitate increased access to lifelong learning and to increase the presence of on-line businesses. The end-user focus group was conducted with persons who had taken courses at the Centre; and
• Creative Retirement Manitoba: Project aims to provide seniors with necessary government and community information by on-line access, combined with courses and mentoring programs to improve seniors' computer skills. The end-user focus group was conducted with seniors.

An interview guide was developed to use with project sponsors and partners. A site-specific moderator's guide was developed for each case study site in which a focus group was conducted.

Appendix B provides a summary of each of the case studies including the project target population/client group, client needs/project objectives, an assessment of the extent to which client needs/project objectives were met, barriers/factors affecting the achievement of the objectives, and other issues.

3.4 Surveys of Project Sponsors, Project Partners and Non-funded Applicants

Three mail surveys (completed by mail and telephone follow-up) were conducted:

• A project sponsor survey;
• A project partner survey; and
• A non-funded applicant (non-funded projects) survey.

The surveys were designed to collect information on project activities and project partnerships, to determine satisfaction with the application process and OLT information/services, and to assess impacts of each project and the OLT program overall. All instruments were pre-tested and available in both English and French.

All LTI (124), CLN (124) and LTW (25) funded project sponsors were mailed a questionnaire. However, only a sample of the 782 non-funded LTI (717), CLN (59), and LTW (6) applicants were sent a survey for completion. The selected sample of non-funded applicants was comprised of a random sample of the LTI non-funded projects (245), and all non-funded CLN (59) and LTW (6) applicants.

The sample for the project partner survey was obtained by asking funded project sponsors for the names and contact information of up to three of their project partners. Questionnaires completed by funded project sponsors provided the names of 173 project partners, all of which were mailed a project partner survey questionnaire.

Table 3-2 provides a detailed breakdown of the sample of funded project sponsors, partners and non-funded applicants who completed the mailed questionnaire. The final sample of project sponsors had lower proportions of CLN and LTW developmental projects. The final sample of non-funded projects had higher proportions of non-funded CLN and LTI applicants. This was due, in part, to the length of time between the survey and applicant submission of funding applications to the LTI initiative, which operated between 1996 and 1999. The final response rate based on the sample is also included in Table 3-2. The sample was obtained by excluding cases where the organization that applied for or received funding no longer exists, or the contact person and all individuals involved in the application process were no longer at the organization and could not be tracked.

3.5 Key Informant Interviews

To complement the above lines of evidence, structured interviews were conducted with a broad range of key informants, resulting in a total of 17 key informant interviews (Table 3-3).

The interview guide was developed to address the informant's involvement in and perspectives on the OLT initiatives and issues related to learning technologies. The guide was pre-tested with a community organization representative to ensure that the questions were clear, meaningful, and presented in a logical sequence. Introduction letters were sent to informants prior to the interviews. Interviews were conducted in-person and by telephone in the official language preferred by the interviewee. Thank-you letters were sent to all interviewees after the interview was complete.

3.6 Strengths and Limitations of the Evaluation Approach

The evaluation approach developed for the OLT recognized the range of practical difficulties in evaluating this type of partnership program:

• Program outcomes, benefits and impacts depend on organizations, groups and individuals that are beyond the direct influence of the program. As a result, in individual instances, final outcomes of program funding are not easy to track or measure.
• Program and project outputs/outcomes are highly diverse, end-use populations are also diverse, and there are no major common metrics (such as the number of job placements) on which performance may be quantified.
• Obtaining data on project outputs, outcomes and end-users can be difficult, the available data can be uneven across projects, and key data for evaluation purposes may simply not be collected by project sponsors.
• Program expectations and objectives can be broad and over-reaching.

A number of steps were taken to address these practical difficulties in evaluating the OLT:

• The multiple lines of evidence approach was emphasized to allow for findings from one approach to be substantiated/corroborated by findings from other lines of evidence. In the OLT evaluation, this allows for greater reliance and confidence to be placed in findings, especially those from non-objective sources (e.g. project sponsor survey respondents);
• Case studies that illustrate and/or demonstrate program outputs and impacts are an important line of evidence, especially in the case of partnership styled programs involving varied outputs and a diverse end-user population. The case studies also add concreteness and provide an in-depth understanding/insights of how the impacts are actually achieved. In the OLT evaluation, ten case studies were conducted, with three of them involving focus groups of end-users; and
• The OLT evaluation included a survey of non-funded projects to add an important perspective against which to compare the funded projects.

Although the evaluation approach developed for the OLT recognized and attempted to address the challenges in evaluating this type program, the following limitations should be noted:

• Evidence from the survey of project sponsors can be viewed as non-objective opinions from a group with a vested interest in program continuation. As sponsors are recipients of program funds, the findings from the survey alone must be interpreted with caution. Greater reliance can be placed upon the findings, however, in instances where they are corroborated from other lines of evidence.
• Data on the end-users of project outputs were unavailable as the program never required the collection of such data from the funded projects. End-user data will not be consistently collected until a new RMAF is in place that sets out the expectations and means for collecting the information. In order to partially compensate for the unavailability of end-user data, focus groups were conducted with the end-users for three of the four CLN case studies.

Footnotes

2	The Evaluation Matrix is contained in Appendix A.