Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The Office of Learning Technologies (OLT) was launched by Human Resources Canada (HRDC) in 1996 as part of an education and training strategy for adult learners. It was established as a contributions program. The original objectives of the OLT have been refocused to address economic changes and at the time of this evaluation the revised objectives of the OLT were:

• To help increase the capacity of Canadians and their community to use learning technologies in order to build a knowledge-based economy and society; and
• To help close the economic and social divide between those who have computer skills and those who do not.

Three funding initiatives have been established under the OLT with specific funding parameters, objectives and target audiences:

• The Learning Technologies Initiative (LTI) operated from 1996 to 1999. During those years, LTI objectives evolved but with a focus on testing the effectiveness of learning technologies in a variety of settings and with diverse groups of adult learners.
• The Community Learning Networks Initiative (CLN) was established in 1998 to support time-limited (3-year) pilot projects in partnership with community organizations. These projects offered access to a variety of learning resources and assisted communities in establishing their own computer networks.
• The Learning Technologies for the Workplace Program (LTW) was established in 2000 and partners with non-profit organizations, industry associations, and educational institutions to provide funds (on a cost-shared basis) to projects that demonstrate the direct application of a learning technology in the workplace. This funding initiative is specifically aimed at small and medium sized enterprises and at disadvantaged workers.

Approach to the Evaluation

Evaluating partnership style programs presents a number of practical challenges. In the case of the OLT, the program's objectives are achieved indirectly through funding projects undertaken by project sponsors and partners. This means that program outcomes, benefits and impacts depend on organizations, groups and individuals that are beyond the direct influence of the program. As a result, in specific instances, final outcomes are not easy to track or measure. Also, program and project outputs/outcomes are highly diverse, end-user populations are highly diverse, and there are no common metrics (such as the number of job placements) on which performance may be quantified.

The evaluation methodology developed for the OLT evaluation recognized these challenges and attempted to address them in a number of ways. For example, the use of multiple lines of evidence was emphasized to allow for findings from one approach to be substantiated/corroborated by findings from other lines of evidence. Also, case studies were used to help illustrate and/or demonstrate program outputs and impacts. The case studies also added concreteness and provided an in-depth understanding of how impacts are actually achieved. The evaluation also included a survey of non-funded projects to provide a reference for the findings on funded projects.

The main components of the evaluation approach included a review of the literature and pertinent documents, a review and analysis of administrative data, case studies, surveys of project sponsors, project partners and non-funded projects, and key informant interviews.

Evaluation Findings

Are the objectives of the OLT still valid and relevant? Information collected from the document review, case studies, key informant interviews and surveys of project sponsors and partners supports the need for a national program that encourages the use of learning technologies to help develop new technical skills and upgrade old skills, particularly in small, isolated communities.

Both the document review and interviews with stakeholders indicated that learning technology and community learning networks provide an important vehicle for implementing widespread learning opportunities. At the same time, however, the general view was that a number of barriers must be overcome (e.g. a lack of awareness/acceptance of learning technologies; a lack of access to infrastructure, technology and funding). Stakeholders indicated that they see the OLT and the federal government as having an important role in facilitating infrastructure, developing and facilitating partnerships, and providing financial assistance and funding.

How important is OLT project funding? Two-thirds (66%) of the projects that applied for but did not receive OLT funding did not proceed. According to non-funded applicants, the major reason that their project did not proceed was the lack of alternative sources of funding.

Only a minority (6%) of the unaccepted projects proceeded in full without OLT funding. Another 28% proceeded in a reduced or substantially reduced form. Most of the projects that proceeded without OLT funding had a narrower scope of reach, mostly targeting the local community, and were less likely to have used/tested or developed a website, the Internet or specialized software. The majority of survey project sponsors (85%) indicated that they would have been unable to deliver the same project scope and activities without OLT funding.

Did OLT contribute to the development and use of learning technologies in Canada in innovative ways? OTL funded projects have used, tested and developed a wide variety of learning technologies, including computer-based training, the Internet, websites and CD-ROMs, to respond to the needs of end-users. Projects adopted innovative strategies built upon the input/expertise of partners to help tailor the learning technologies to their end-user groups.

Evidence from the case studies, surveys and key informants suggests that the program has also provided access to workplace training through LTW funding and has supported projects aimed at skills upgrading and lifelong learning.

Has OLT developed sustainable partnerships? The OLT has led to the development of partnerships with private sector, community and non-governmental organizations. At least one new partnership was developed by 77% of the surveyed project sponsors, with each sponsor developing (on average) 3 new partnerships. The evidence also suggests that these partnerships will continue in some form after OLT funding support ends. Looking specifically at LTI funded projects, which generally ended by 1999, 73% of the partnerships were reported to have extended beyond the life of the project.

Who was reached by the OLT? OLT was designed to reach adult learners, especially those who were members of designated equity groups (persons in rural/remote areas, women, youth, seniors, visible minorities, persons with disabilities and Aboriginal people).

The evaluation encountered a number of difficulties in trying to identify end-users because the program never required the collection of such data from funded projects. The available evidence indicates that OLT projects had considerable success in reaching most of the designated equity groups. Three exceptions were young mothers, Aboriginal clients and ESP populations.

Are projects created through OLT sustainable? The available evidence from the case studies, surveys and key informants suggests that many OLT projects would either continue in some form after OLT support ends, or produce outputs/products that will be used after funding ends. More than three-quarters (78%) of the surveyed project sponsors felt that their project would continue in some form after the expiration of OLT funding.

Did OLT increase the sharing of knowledge about learning technologies? Project sponsors generally agreed that their interaction with the OLT has increased their knowledge of new approaches in learning technologies (88%), learning technologies information (80%), and trends in the use of learning technologies (59%). For project sponsors and partners, the primary advantage of partnerships was said to be the sharing of knowledge and skills to provide different perspectives and ideas on learning technologies use and development.

Did OLT contribute to improved accessibility to learning technologies? Project sponsors and partners agreed that funded projects have broadened access to learning opportunities (95%) by helping develop the communities' technological infrastructure (66%) and facilitating the development of community expertise in learning technologies (74%).

All initiatives have improved accessibility to learning technologies by partnering with organizations with access to targeted end-users. The CLN initiative has had the strongest relationship to the community as evidenced through significant linkages to Community Access Program (CAP) sites. Almost half (44%) of all CLN projects were directly linked, through the project partner or sponsor, to a CAP site. Additionally, by funding CLN projects facilitated by community colleges or institutes (22%) and non-profit organizations (38%) that serve the community, OLT has further increased accessibility to learning technologies. Although the CLN funded projects have a range of specific objectives, most were aimed at increasing learning skills (23%), reducing barriers (23%), or increasing accessibility to technologies (27%).

The available evidence also indicates that OLT projects benefited end-users. End-user focus groups for three of the four CLN project case studies identified a number of impacts, including reduced social isolation, increased confidence in learning and a reduced aversion to using computers. Also, the survey of project sponsors indicated that OLT projects have helped workers take advantage of technology (82%), facilitated lifelong learning in the community (81%), and helped develop a more skilled workforce (77%).

Is the OLT cost effective? The evaluation was unable to use standard methods in considering cost-effectiveness due to the lack of data. Therefore, the evaluation was limited to reporting on the perceptions of key informants in this area and incrementality in the leveraging of additional funds.

The key informants generally saw the OLT management model as cost effective. In part, it was suggested that this cost-effectiveness comes from the ability to involve other organizations in OLT funded projects and to leverage a significant amount of resources by facilitating cooperation among organizations across the country.

Areas for Improvement

The broad over-reaching objective of the OLT (i.e. to reduce the economic and social divide between those who have computer skills and those who do not) needs to be recast in more realistic terms. The current objective of the OLT needs to be recast to be something that a program with a limited budget can realistically achieve. An example might be to make the objective to improve computer skill sets of end-users and reduce their reluctance/inability to use newer technologies.

Program controls need to be put in place to minimize the risk that funds leveraged by OLT projects are not incremental, and to safeguard against overlap and duplication. Although there is evidence that OLT projects are leveraging additional funds, the current program design does not include safeguards to reduce the risk that leveraged funds are not incremental (i.e., would have been used for similar activities in the absence of the program). Similarly, given that some of the unaccepted projects gained funding from other sources, there is a need for the program to be able to demonstrate that it has safeguards in place to prevent, or at least limit, any overlap or duplication of activities funded by other sources (e.g. by coordinating activities with Non-governmental organizations (NGOs), other levels of government, etc.).

While OLT has been able to reach designated equity groups, certain groups may require additional program effort to be adequately reached. While OLT funded programs have been relatively successful in reaching persons in remote/rural areas, women and seniors, there has been less success in reaching the Aboriginal community, young mothers and ESL populations. Aboriginal groups, especially, may require a targeted initiative to address the multiple issues identified in serving this population.

There is a need for better outcome tracking at the end-user level. There is a need to provide for the tracking of end-users, for example by requiring project participants to register with project staff.

Project sponsors and partners lack experience partnering. Project sponsors often lack required knowledge to develop and mobilize formal partnerships. Lack of knowledge of partnering impacts both the development and maintenance of partnerships and can lead to conflicts during project development. This is particularly problematic for non-profit organizations and community agencies. In general, organizations would like more direction and assistance from the OLT in terms of the development and maintenance of partnerships.

OLT staffing levels and staff turnover have impacted the program. There is a perception that there is insufficient OLT staff to manage current projects. Further, high staff turnover within the OLT has created an impression that staff are not sufficiently familiar with the program or with learning technologies. Generally, project sponsors would like more interaction and direction from OLT staff.

There is a strong perception that OLT should allocate more resources to understanding and disseminating best practices and lessons learned. It is felt that OLT can act as a data warehouse for learning technology information, research and findings from projects. Stakeholders also requested additional opportunities for project sponsor interaction and information sharing between and within project initiatives.

Delays in receiving OLT funding (i.e., from the developmental phase to pilot phase as a result of the application process) resulted in waning project momentum and interest among partners, sometimes resulting in partner drop out. Informants and sponsors suggested that there be a "fast track" procedure for obtaining funding between the developmental and pilot phases, for projects that are already running. The issue of waning partner interest was more salient when community organizations were involved, given the unstable nature of their internal funding and staff resources.