Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

This section examines the administration and delivery of the OLT by examining non-funded projects, the ratio of funded to non-funded projects for the three funding initiatives, awareness of the OLT, and the level of satisfaction with the application process and project administration.

5.1 Non-funded Projects

OLT funding appears to be an important factor in the development of projects aimed at using technology to promote learning, with only 6% of the non-funded projects proceeding in full without OLT funding. The survey of non-funded project applicants indicated that only 6% of the non-funded applicants pursued the full project as outlined in the funding application (as shown in Table 5-1). Another 28% pursued less than the full project, with most of these projects going ahead in a substantially reduced form (i.e. less than 25% of the project that had been outlined on the OLT application). The other 66% of non-funded projects did not proceed at all after being unaccepted for funding by the OLT program.

Applicants are applying for OLT funding when other sources of funding are not readily available. The major reason given by non-funded applicants for projects not proceeding, entirely or in part, was lack of alternative funding sources. As further indication that other sources of funding were not readily available, most (73%) non-funded applicants would apply again for OLT funding. Only 5% of those who would not apply again to OLT indicated it was because they felt they could obtain funding elsewhere.

Projects that proceeded in the absence of OLT funding obtained alternate funding through donations (22%), internal sources (22%), government (22%), and other sources (34%).

Almost two-thirds of the unsuccessful applicants either did not receive or could not recall the reason for the rejection of their application. The file review conducted for the formative evaluation of the OLT indicated that the main reasons a project would not be recommended were:

• Results are unlikely to have a broad impact;
• Proposal requires significant reworking;
• Sufficient similar activities being supported by OLT;
• Results would add little to the existing knowledge base on LT;
• Project does not represent an innovative use of LT;
• Major portion of project is in purchase of capital expenditures;
• Activities focus primarily on connectivity and communications;
• Activities focus primarily on service delivery or provision of training; or
• Activities focus primarily on transfer of media (e.g. paper to CD).

The survey of the non-funded project applicants indicated that it was common for unsuccessful applicants to indicate they did not receive or could not recall the reason for the rejection of their application (64%). Among the thirty-six percent (36%) that could recall the reason for application rejection, it was common for the project to have not met OLT funding criteria (17%), or to have insufficient partnerships developed (11%).

Looking specifically at the sub-group of non-funded applicants that proceeded with the project in the absence of OLT funding (which accounted for 34% of the non-funded projects), most (72%) could not recall or did not provide a reason for why their application was rejected. Only 6% of the non-funded applicants indicated their project did not fall within the scope of OLT objectives.

For evaluation purposes, the non-funded "active" projects can serve in some respects as an adequate reference against funded projects. The suitability of using the non-funded "active" projects as a reference group was assessed by examining the main objectives of the proposed project as stipulated in the proposal application. The objectives of these projects varied greatly; however, most aimed at providing training or addressing the learning needs of equity groups. Therefore, non-OLT funded projects can serve in some respects as an adequate reference against the funded projects. Although these 18 projects do not provide an ideal comparison or control group, particularly given the relatively few projects that provided data, comparisons are made between the OLT funded projects and the unaccepted projects that proceeded in the absence of OLT funding where appropriate in the remaining sections of the report.

5.2 Projects Funded by Type of Initiative

Among the three funding initiatives, LTW has the highest ratio of accepted applicants. Table 5-2 shows the number of applications submitted to each of the three OLT funding initiatives and the percentage of applications that were funded or failed to receive funding from the initiative. Information from the file review indicates that the LTW funded the highest proportion of applicants. However, it should be noted that this funding initiative had also run for the shortest duration at the time of this evaluation.

5.3 Awareness of the OLT Initiative

Project sponsors most often became aware of the OLT program through the OLT website (40%), and project partners usually became aware of the program through the sponsor's funding application (61%). Project sponsors, commonly became aware of the OLT program through the OLT website (40%), national or provincial associations (21%), and OLT staff (15%). One-third (31%) of LTI funded project sponsors were notified by OLT, through email, of the call for proposals. Project partners primarily became aware of the program through the sponsor's funding application (61%).

5.4 Satisfaction with the Application Process and Project Administration

Project sponsors, and those partners involved in the application process, were generally satisfied with the application process, although there was some confusion around the "matching funds" component of the application. Project sponsors and partners generally agreed on the following:

• Program-initiative objectives were clear (82%);
• Eligibility criteria set forth in the application guidelines were clear (77%);
• Proposal assessment process was adequately described (62%); and
• Selection criteria were clear (62%).

At the same time, the case study interviews and key informant interviews suggested that there is an opportunity to improve the application process. For example, the application process was viewed as cumbersome or unclear in some respects, and in need of streamlining. In particular, applicants noted that they were unsure what constituted a matching fund and felt that contributions in-kind should be permissible as matching funds by the OLT. For some, the inability to utilize contributions in-kind as matching funds had resulted in difficulties obtaining project partners.

In the area of project administration, project sponsors and other stakeholders were most concerned with the time and effort required from project staff to meet the program reporting requirements. Resource allocation requirements for management of the OLT contribution arose as a significant project administration issue among project sponsors and partners. Organizations found that they generally utilized more staff resources to run the project than they had originally anticipated. In some cases, projects required hiring additional staff to meet mandatory OLT reporting requirements. Both key informants and project sponsors commented that the pressure placed on staff resources by the project would dissuade the organization from applying again for OLT funding.

Reporting documents received from project sponsors were not optimally utilized by the OLT. There were significant limitations to the administrative data provided by the OLT: very few variables were captured in the database and no project outcome data was available. While case study interviews indicated that project sponsors had provided significant amounts of information through on-going reporting, little of this information was captured by the administrative data. Further, project sponsors expressed concern that, although OLT was provided with on-going progress reports, they never received feedback on project performance or feedback on project issues raised in the mandatory reports. Project sponsors and partners generally agreed (72%), however, that they were satisfied that OLT staff were prompt in addressing direct questions put forth by means other than mandatory reporting.

Project sponsors and partners indicated that they would have liked more contact with the OLT during project development and roll out. The general view was that project sponsors and partners would have preferred to have more contact with program staff during the early stages of the project. Further, it was noted that there had been a lack of continuity due to staff turnover within OLT.

Some projects experienced difficulties in developing and maintaining partnerships. Project sponsors and key informants noted that some difficulties were experienced in developing and maintaining partnerships. Partnership activities required considerable resources and were particularly challenging for those organizations inexperienced in partnering, (i.e. non-profit and community organizations). Larger organizations or educational institutions noted that community sponsors had little experience in partnering and therefore lacked understanding of what roles and responsibilities each partner should fulfill.

The case studies indicated that organizations with little experience in partnership development and maintenance would have liked additional support from the OLT in partnering. In some cases partners indicated that the guidelines provided by OLT were difficult to follow and required interpretation and clarification by OLT staff. They noted that this created an additional barrier for smaller agencies and less experienced applicants. Similarly, the Vancouver Community Network (VCN), which partnered exclusively with non-profit and community agencies, found capacity building was a significant issue in project development. VCN's partners were often unclear on the amount of internal resources available to commit to the project, and were unable to focus sufficient resources on the project because of other organizational priorities. Key informants echoed the problems identified by project sponsors and the case studies. In addition, key informants indicated that a lack of understanding of OLT goals and objectives was also a barrier to partnership.

5.4.1 Use of OLT Services

Project sponsors found the OLT website and staff to be useful sources of support. Project sponsors commonly accessed the OLT website (84%) and OLT staff (87%) were considered to be useful sources of support. Additional services were accessed to a lesser degree, in part because they were newly offered or offered on a time-limited basis by the OLT. These services included the following:

• OLT published reports/studies (52%);
• The annual meeting of the Advisory Network of Experts and other forums (36%); and
• OLT workshops (34%).

At the same time, the case study interviews with project sponsors and partners indicated that some organizations had not accessed or were unfamiliar with many of the resources available through OLT. This was evidenced by requests for support documents currently in place at OLT such as partnering guidelines.

5.4.2 Views on Cost Effectiveness

The evaluation was unable to use standard methods in considering the cost-effectiveness of the OLT, because the data needed to undertake a standard analysis were not available. Therefore the evaluation was limited to reporting on the perceptions of key informants in this area and the related issue of incrementality in the leveraging of additional funds.

The key informants generally saw the OLT management model as cost effective. When key informants were ask whether the OLT is cost efficient and effective in promoting learning technologies, they indicated that they saw the OLT management model as cost effective, particularly when compared to other programs. In part, it was suggested that this cost-effectiveness comes from the ability to involve other organizations in OLT funded projects and to leverage a significant amount of resources by facilitating cooperation among organizations across the country.

Although there is some evidence that OLT projects leveraged additional funds, it is difficult to determine the extent to which the leveraged funds are incremental. The OLT program requires that OLT funding is matched one-for-one by contributions from project partners. An examination of the case study projects shows that significant resources, either in-kind or financial, were leveraged through the OLT funded projects. Information from 6 case studies (Table 5-3) showed that the actual leveraging was in the order of $1.50 for every $1 of OLT funding.

There was considerable variation across the projects, however, as the leveraged amounts varied from $0.66 to $3.67 for every dollar spent by the OLT, as compared to the expectation that OLT dollars will be matched one-for-one. Also, it is not clear to what extent the leveraged funds are incremental.

Program controls need to be put in place to minimize the risk that funds leveraged by OLT projects are not incremental (i.e. would have been used for similar activities in the absence of the program), and to safeguard against overlap and duplication.

Although the program monitors projects on an ongoing basis throughout the project and requires an evaluation to be conducted at the end of the funding period, the project sponsor is not required to attest to the incrementality of the project.

Given that 34% of the non-funded projects proceeded in some form without OLT funding, and that these projects obtained alternative funding through other sources (e.g. donations, governments), it is important for the OLT to have program safeguards in place to prevent, or at least limit, the possibility that the OLT projects would have proceeded in the absence of the program. One method of accomplishing this could be to require the project sponsors to attest that the project is incremental (i.e., would not have gone ahead without OLT funding).

Similarly, given that some of the unaccepted projects gained funding from other sources, it is important for the program to demonstrate that it has safeguards in place to prevent, or at least limit, any overlap or duplication of activities funded by other sources (e.g. by coordinating activities with NGOs, other levels of government, etc.).