Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Introduction

The Opportunities Fund for Persons with Disabilities (OF) was established in 1997 to "assist persons with disabilities in preparing for, obtaining and keeping employment or becoming self-employed, thereby increasing their economic participation and independence." OF is the responsibility of Human Resources Development Canada (HRDC), with the guidance of a National Reference Group composed of representatives from national disability organizations. During the period covered by the evaluation, the National Reference Group and Opportunities Fund management met regularly to discuss and review program management issues. In the February 2000 budget, OF was renewed on a permanent basis with an annual budget of $30 million.

Delivery of OF is through HRDC Regional offices, which distribute funds to local Human Resources Centres of Canada (HRCCs) which in turn may provide funds to third-party deliverers. Ten percent of OF dollars are reserved for national projects, and since 1999-2000 another ten percent of the program funding has been transferred to Aboriginal Human Resource Development Agreements (AHRDA) for persons with disabilities projects. Potential OF clients must meet certain criteria to qualify for assistance (e.g., be ineligible for Employment Insurance (EI) benefits).

The purpose of this summative evaluation was to measure the effectiveness, impacts and effects of OF, particularly for participants, but also for employers and organizations. Evaluation issues identified for the evaluation also covered questions pertaining to program rationale, implementation, and cost-effectiveness. Evaluation activities have been conducted under the guidance of an Evaluation Working Group drawn from the National Reference Group and with representation from HRDC at the regional level. The Working Group reviewed and approved the Terms of Reference and methodology for this project before the evaluation.

The methodological approach for the evaluation was based on multiple lines of evidence, including: documentation review; key informant interviews with representatives from HRDC, the National Reference Group and project sponsors; survey of employers involved in targeted wage subsidy contracts under OF; analysis of administrative data; survey of OF participants and a comparison group; survey of project sponsors; focus groups with OF participants and stakeholders; and case studies illustrative of OF program delivery.

Because a national comprehensive listing of OF program participants was not available, a custom sampling frame was compiled specifically for the evaluation. Three sources were used to compile the sampling frame: lists of participants supplied by some regions; information forwarded by third party deliverers of OF; and HRDC's National Employment Services System (NESS). Unanticipated data integrity problems with the NESS resulted in exclusion of all completed interviews that were based on sample drawn from NESS, including all those from BC, from the final analysis. The participant survey was conducted by telephone, as this method provided an effective means of gathering information from a large number of participants in a structured way within a short period of time. To enhance opportunities to provide feedback, a TTY line was made available, as well as options for self-administered and proxy completion of the survey. It should be noted that it is still possible that given potential communication barriers, the data may under represent persons with some types of disabilities such as intellectual disabilities and hearing or speaking disabilities. Analyses of the composition of the sample provided a positive assessment of representativeness, however. Despite the methodological challenges, then, the evaluation provides useful feedback and insight into OF.

Participant Profile

Compared to participants in regular EI Part II programs, OF participants had lower levels of education, a somewhat older age profile and were less likely to be married and to have dependents. OF participants had only a moderate attachment to the labour market prior to program participation, with six in 10 not having been employed in the two years prior to their intervention. Weekly wage rates for those who worked and total annual income of OF participants prior to program participation were low by national standards.

The disability profile of OF participants is broadly similar to the overall profile of Canadians with disabilities. The distribution of OF participants' self-assessed severity of their disability in terms of the extent of limitations at work was roughly equal across mild, moderate and severe ratings. This suggests a balanced portfolio of clients in the program.

Program Activity Profile

Reflecting the differing needs of OF participants from "regular" EI clients, the program activity profile suggested interventions under OF were more intensive and that a wider-range and different mix of activities has been occurring under OF.

The types of program interventions received were broadly similar to those offered under EI Part II. The most commonly cited interventions by participants included: job placement or wage subsidy, education or training course and job search program. However, volunteering was also used as an intervention under OF (unlike EI) and use of work placements and funding for specialized equipment and job accommodations were prevalent. The majority of participants completed the full period of their OF program.

Findings

Rationale

The continuing need for a program such as OF was strongly supported by the evaluation. Though the empirical research available in Canada is limited, persons with disabilities continue to experience disadvantages in the labour market and barriers to employment opportunities remain significant. The federal government, therefore, was viewed as having a valid and necessary role in assisting persons with disabilities to employment. Key informants and the government's own policy documentation, however, suggested a need to look at a more comprehensive and cohesive labour market strategy for persons with disabilities that would encompass existing components.

The program principles and guidelines of OF were considered to be relevant, though some adjustments were suggested by key informants (e.g., incorporating notions of employability and more flexible yardsticks).

Implementation

An important perceived strength of OF program delivery was the use of individualized and flexible approaches for delivery of services to clients. The inclusion of the disability community was viewed as another positive feature of the program, if somewhat uneven in its implementation at the local level.

The program is strongly committed to partnerships, and this was considered to be innovative and a key strength of OF. The vast majority of project sponsors were involved in partnered delivery arrangements. Partnerships with other community organizations were most common and employer involvement has increased significantly since the formative evaluation. The need for fuller participation of employers — and to ensure a parallel focus on the attitudinal and systemic barriers that remain in the workplace were noted during the course of the evaluation study.

Overall, OF participants, employers and project sponsors expressed a high degree of satisfaction with OF. The incrementality of the program — that is, the extent to which activities would not have taken placed in the absence of the program — was confirmed in the surveys. The vast majority of participants, employers and project sponsors reported that OF was very important for them in terms of ability to participate in, or offer, interventions.

Areas where implementation did not reflect the program principles included: weaknesses in participant targeting with respect to the EI-eligibility criterion; and poor program monitoring. In terms of targeting, there was a significant minority of OF participants — between eight and 13 percent — identified as falling outside the program eligibility criteria.

While the OF program guidelines emphasize employment results, monitoring and reporting of results were found to be weak. The availability of information on program participation, activities and results at the NHQ level was poor. The infrastructure for data collection and roll-up was not in place due to the limited use of Contact IV, HRDC's software application created for third parties to record client data. Utilization of Contact VI had not improved significantly since the formative evaluation. The adequacy of current program performance measures was also questioned in the evaluation.

Some of the other challenges of the program had to do with gaps for particular client groups, including: those in remote communities; Aboriginal persons with disabilities; youth; those with invisible disabilities; and persons with multiple/severe disabilities. Some key informants favoured extending the eligibility criteria to include EI-eligible clients and those currently in the labour force. There were also identified gaps in the availability of interventions including: entrepreneurship, pre-employment skills, disability supports and employment in higher skill areas.

Impacts and Effects

The majority of OF participants rated their OF program as useful in providing them with direct employability-related benefits such as skill development and relevant work experience, access to proper assistive devices and job search contacts and skills. Less tangible benefits of the program were also evident, with many citing positive impacts on their self-confidence and self-esteem.

One-half of OF participants were employed (full-time, part-time or self-employed) at the time of the survey interview, exceeding the 40 percent target that was set for the program. The evidence indicated that the jobs acquired were sustainable, based on the observation that there was no deterioration of employment rates of early participants compared to later entrants within a three-year timeframe.

In terms of participants' employment outcomes compared to non-participants, OF participants showed a positive advantage over the reference group. When controlling for pre-existing differences between program participants and the comparison group in the multivariate analysis, the results indicated that OF participants were 15 percent more likely to be employed at the time of the interview, worked 14.6 percent more weeks during the post-program period and were 10 percent more likely to have ever worked in the post-program period. There were no statistically significant impacts on weekly earnings, personal income, receipt of public income support or attitudes (e.g., confidence, optimism).

The most important benefit of participating in the OF program for employers was access to a skilled employee. Only two percent of employers cited no benefits of participation. About one-third of employers indicated that participation in OF had changed their organization's perceptions about hiring persons with disabilities. Almost two-thirds of employers hired at least one OF participant after the program — the majority on a full-time and permanent basis.

While largely anecdotal, impacts of OF were noted on disability organizations themselves. OF was reported as being important in further integrating the disability community through opportunities for networking and partnerships at the local level. Participation in OF has also led to additional and better design and delivery of employment initiatives at the local level.

Cost-Effectiveness

A comparison of the income profile of participants prior to participation in OF and at the time of the interview indicated that participation in OF increased the likelihood of having employment earnings. Despite the shift in income profile, however, multivariate analyses did not detect there to be a significant impact in terms of reducing the likelihood of being on social assistance or other forms of public income support compared to non-participants. This may be at least partially due to the limitations in the data that did not support a meaningful measure of public income support over time (i.e., since the end of the intervention).

With respect to earnings gains, while there were no significant advantages for the participant group in terms of weekly wage rate, the fact that OF participants were employed for proportionately more weeks than the comparison group (seven more weeks on an annual basis) translated into a total earnings advantage. Based on an estimated number of participants of 9,800 whose average weekly earnings were $303, the total earnings benefit is approximately $20.8 million on an annual basis owing to OF participation. The budget of the OF program is about $30 million per year, or between $5,000 and $6,000 per participant.

The majority of OF projects received financial or in-kind contributions and these contributions had increased since the formative evaluation. The extent of use of EI Part II resources was limited due to the introduction of Labour Market Development Agreements (LMDAs) at that time. This transition in program delivery and funding arrangements of the affected provinces may have further reduced the availability of funding for persons with disabilities in the area of employability.

Summary

Overall, this summative evaluation has provided a positive assessment of OF. Given that barriers to employment persist for persons with disabilities, the rationale for the program is sound. The overall design, management and implementation of the program were perceived to be strong and participants and deliverers were largely satisfied with the program. Where there were weaknesses identified, these had to do with inadequacies in program monitoring and data collection, client targeting with regard to the EI-eligibility criterion and client support. Enhanced participation of employers and employer organizations, as well as increased attention to workplace-based issues were also raised.

The self-assessment of OF participants, as well as the outcome data indicate that OF assisted individuals to obtain employment and also improved their employability and quality of life. Increased time spent working in the post-program period translated into a positive earnings benefit for participants. Employers and organizations also reported program benefits. The program was judged to be incremental by the strong majority of participants, employers and organizations.

Issues in the broader program environment, specifically, harmonization with other initiatives to address employment for persons with disabilities, compatibility of income and disability support programs with OF and leveraging of EI Part II funds under the new LMDAs were identified as important future considerations.