Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Summative Evaluation of the Opportunities Fund for Persons with Disabilities

Evaluation and Data Development
Strategic Policy
Human Resources Development Canada

August 2001

Introduction

The Opportunities Fund for Persons with Disabilities (OF) was established in 1997 to "assist persons with disabilities in preparing for, obtaining and keeping employment or becoming self-employed, thereby increasing their economic participation and independence." OF is the responsibility of Human Resources Development Canada (HRDC), with the guidance of a National Reference Group composed of representatives from national disability organizations. During the period covered by the evaluation, the National Reference Group and Opportunities Fund management met regularly to discuss and review program management issues. In the February 2000 budget, OF was renewed on a permanent basis with an annual budget of $30 million.

Delivery of OF is through HRDC Regional offices, which distribute funds to local Human Resources Centres of Canada (HRCCs) which in turn may provide funds to third-party deliverers. Ten percent of OF dollars are reserved for national projects, and since 1999-2000 another ten percent of the program funding has been transferred to Aboriginal Human Resource Development Agreements (AHRDA) for persons with disabilities projects. Potential OF clients must meet certain criteria to qualify for assistance (e.g., be ineligible for Employment Insurance (EI) benefits).

The purpose of this summative evaluation was to measure the effectiveness, impacts and effects of OF, particularly for participants, but also for employers and organizations. Evaluation issues identified for the evaluation also covered questions pertaining to program rationale, implementation, and cost-effectiveness. Evaluation activities have been conducted under the guidance of an Evaluation Working Group drawn from the National Reference Group and with representation from HRDC at the regional level. The Working Group reviewed and approved the Terms of Reference and methodology for this project before the evaluation.

The methodological approach for the evaluation was based on multiple lines of evidence, including: documentation review; key informant interviews with representatives from HRDC, the National Reference Group and project sponsors; survey of employers involved in targeted wage subsidy contracts under OF; analysis of administrative data; survey of OF participants and a comparison group; survey of project sponsors; focus groups with OF participants and stakeholders; and case studies illustrative of OF program delivery.

Because a national comprehensive listing of OF program participants was not available, a custom sampling frame was compiled specifically for the evaluation. Three sources were used to compile the sampling frame: lists of participants supplied by some regions; information forwarded by third party deliverers of OF; and HRDC's National Employment Services System (NESS). Unanticipated data integrity problems with the NESS resulted in exclusion of all completed interviews that were based on sample drawn from NESS, including all those from BC, from the final analysis. The participant survey was conducted by telephone, as this method provided an effective means of gathering information from a large number of participants in a structured way within a short period of time. To enhance opportunities to provide feedback, a TTY line was made available, as well as options for self-administered and proxy completion of the survey. It should be noted that it is still possible that given potential communication barriers, the data may under represent persons with some types of disabilities such as intellectual disabilities and hearing or speaking disabilities. Analyses of the composition of the sample provided a positive assessment of representativeness, however. Despite the methodological challenges, then, the evaluation provides useful feedback and insight into OF.

Participant Profile

Compared to participants in regular EI Part II programs, OF participants had lower levels of education, a somewhat older age profile and were less likely to be married and to have dependents. OF participants had only a moderate attachment to the labour market prior to program participation, with six in 10 not having been employed in the two years prior to their intervention. Weekly wage rates for those who worked and total annual income of OF participants prior to program participation were low by national standards.

The disability profile of OF participants is broadly similar to the overall profile of Canadians with disabilities. The distribution of OF participants' self-assessed severity of their disability in terms of the extent of limitations at work was roughly equal across mild, moderate and severe ratings. This suggests a balanced portfolio of clients in the program.

Program Activity Profile

Reflecting the differing needs of OF participants from "regular" EI clients, the program activity profile suggested interventions under OF were more intensive and that a wider-range and different mix of activities has been occurring under OF.

The types of program interventions received were broadly similar to those offered under EI Part II. The most commonly cited interventions by participants included: job placement or wage subsidy, education or training course and job search program. However, volunteering was also used as an intervention under OF (unlike EI) and use of work placements and funding for specialized equipment and job accommodations were prevalent. The majority of participants completed the full period of their OF program.

Findings

Rationale

The continuing need for a program such as OF was strongly supported by the evaluation. Though the empirical research available in Canada is limited, persons with disabilities continue to experience disadvantages in the labour market and barriers to employment opportunities remain significant. The federal government, therefore, was viewed as having a valid and necessary role in assisting persons with disabilities to employment. Key informants and the government's own policy documentation, however, suggested a need to look at a more comprehensive and cohesive labour market strategy for persons with disabilities that would encompass existing components.

The program principles and guidelines of OF were considered to be relevant, though some adjustments were suggested by key informants (e.g., incorporating notions of employability and more flexible yardsticks).

Implementation

An important perceived strength of OF program delivery was the use of individualized and flexible approaches for delivery of services to clients. The inclusion of the disability community was viewed as another positive feature of the program, if somewhat uneven in its implementation at the local level.

The program is strongly committed to partnerships, and this was considered to be innovative and a key strength of OF. The vast majority of project sponsors were involved in partnered delivery arrangements. Partnerships with other community organizations were most common and employer involvement has increased significantly since the formative evaluation. The need for fuller participation of employers — and to ensure a parallel focus on the attitudinal and systemic barriers that remain in the workplace were noted during the course of the evaluation study.

Overall, OF participants, employers and project sponsors expressed a high degree of satisfaction with OF. The incrementality of the program — that is, the extent to which activities would not have taken placed in the absence of the program — was confirmed in the surveys. The vast majority of participants, employers and project sponsors reported that OF was very important for them in terms of ability to participate in, or offer, interventions.

Areas where implementation did not reflect the program principles included: weaknesses in participant targeting with respect to the EI-eligibility criterion; and poor program monitoring. In terms of targeting, there was a significant minority of OF participants — between eight and 13 percent — identified as falling outside the program eligibility criteria.

While the OF program guidelines emphasize employment results, monitoring and reporting of results were found to be weak. The availability of information on program participation, activities and results at the NHQ level was poor. The infrastructure for data collection and roll-up was not in place due to the limited use of Contact IV, HRDC's software application created for third parties to record client data. Utilization of Contact VI had not improved significantly since the formative evaluation. The adequacy of current program performance measures was also questioned in the evaluation.

Some of the other challenges of the program had to do with gaps for particular client groups, including: those in remote communities; Aboriginal persons with disabilities; youth; those with invisible disabilities; and persons with multiple/severe disabilities. Some key informants favoured extending the eligibility criteria to include EI-eligible clients and those currently in the labour force. There were also identified gaps in the availability of interventions including: entrepreneurship, pre-employment skills, disability supports and employment in higher skill areas.

Impacts and Effects

The majority of OF participants rated their OF program as useful in providing them with direct employability-related benefits such as skill development and relevant work experience, access to proper assistive devices and job search contacts and skills. Less tangible benefits of the program were also evident, with many citing positive impacts on their self-confidence and self-esteem.

One-half of OF participants were employed (full-time, part-time or self-employed) at the time of the survey interview, exceeding the 40 percent target that was set for the program. The evidence indicated that the jobs acquired were sustainable, based on the observation that there was no deterioration of employment rates of early participants compared to later entrants within a three-year timeframe.

In terms of participants' employment outcomes compared to non-participants, OF participants showed a positive advantage over the reference group. When controlling for pre-existing differences between program participants and the comparison group in the multivariate analysis, the results indicated that OF participants were 15 percent more likely to be employed at the time of the interview, worked 14.6 percent more weeks during the post-program period and were 10 percent more likely to have ever worked in the post-program period. There were no statistically significant impacts on weekly earnings, personal income, receipt of public income support or attitudes (e.g., confidence, optimism).

The most important benefit of participating in the OF program for employers was access to a skilled employee. Only two percent of employers cited no benefits of participation. About one-third of employers indicated that participation in OF had changed their organization's perceptions about hiring persons with disabilities. Almost two-thirds of employers hired at least one OF participant after the program — the majority on a full-time and permanent basis.

While largely anecdotal, impacts of OF were noted on disability organizations themselves. OF was reported as being important in further integrating the disability community through opportunities for networking and partnerships at the local level. Participation in OF has also led to additional and better design and delivery of employment initiatives at the local level.

Cost-Effectiveness

A comparison of the income profile of participants prior to participation in OF and at the time of the interview indicated that participation in OF increased the likelihood of having employment earnings. Despite the shift in income profile, however, multivariate analyses did not detect there to be a significant impact in terms of reducing the likelihood of being on social assistance or other forms of public income support compared to non-participants. This may be at least partially due to the limitations in the data that did not support a meaningful measure of public income support over time (i.e., since the end of the intervention).

With respect to earnings gains, while there were no significant advantages for the participant group in terms of weekly wage rate, the fact that OF participants were employed for proportionately more weeks than the comparison group (seven more weeks on an annual basis) translated into a total earnings advantage. Based on an estimated number of participants of 9,800 whose average weekly earnings were $303, the total earnings benefit is approximately $20.8 million on an annual basis owing to OF participation. The budget of the OF program is about $30 million per year, or between $5,000 and $6,000 per participant.

The majority of OF projects received financial or in-kind contributions and these contributions had increased since the formative evaluation. The extent of use of EI Part II resources was limited due to the introduction of Labour Market Development Agreements (LMDAs) at that time. This transition in program delivery and funding arrangements of the affected provinces may have further reduced the availability of funding for persons with disabilities in the area of employability.

Summary

Overall, this summative evaluation has provided a positive assessment of OF. Given that barriers to employment persist for persons with disabilities, the rationale for the program is sound. The overall design, management and implementation of the program were perceived to be strong and participants and deliverers were largely satisfied with the program. Where there were weaknesses identified, these had to do with inadequacies in program monitoring and data collection, client targeting with regard to the EI-eligibility criterion and client support. Enhanced participation of employers and employer organizations, as well as increased attention to workplace-based issues were also raised.

The self-assessment of OF participants, as well as the outcome data indicate that OF assisted individuals to obtain employment and also improved their employability and quality of life. Increased time spent working in the post-program period translated into a positive earnings benefit for participants. Employers and organizations also reported program benefits. The program was judged to be incremental by the strong majority of participants, employers and organizations.

Issues in the broader program environment, specifically, harmonization with other initiatives to address employment for persons with disabilities, compatibility of income and disability support programs with OF and leveraging of EI Part II funds under the new LMDAs were identified as important future considerations.

The Labour Market Directorate (LMD) of Human Resources Development Canada (HRDC) has reviewed the summative evaluation of the Opportunities Fund (OF) for Persons with Disabilities. We find the report to be generally favourable — with participants, employers and project sponsors continuing to emphasize the importance of OF in terms of their ability to participate in, or offer, a broad and flexible variety of program interventions.

Other positive features include the participation of the disability community, program flexibility, emphasis on partnership models for delivery, and a focus on employment outcomes. Although the impacts in terms of the reduction in draw on income support were less significant, individuals who participated in the program improved their employability, self-esteem and overall quality of life. Employers benefited by accessing skilled employees, while increasing their understanding of barriers faced by people with disabilities.

Some changes in design and delivery have been recommended and steps have been taken to address weaknesses, particularly in the areas of program implementation, accountability and communication.

OF continues to serve a broad spectrum of persons with varying disabilities. The overall percentage of employed (49 percent) has exceeded the initial target of 40 percent. Although OF represents only a small portion of the Government of Canada's Disability Strategy, it continues to address real needs in the disability community. Labour Market Directorate (LMD) fully endorses the evaluation's conclusions.

Program Rationale

Need for Comprehensive Labour Market Strategy

A key finding of both the formative and summative evaluations was the need to look at a more comprehensive and cohesive labour market strategy for persons with disabilities. In Canada, responsibility for initiatives to enhance the employability of persons with disabilities is a shared one. Supported by the findings of the Federal/Provincial/Territorial Labour Market Needs Analysis, HRDC working together with provinces and territories, has developed the framework for a labour market strategy for persons with disabilities. This is an important step in fulfilling the Government of Canada's commitment in the January 1, 2001 Speech from the Throne to work with provinces/territories toward a comprehensive labour market strategy for persons with disabilities. We believe that the end result of this joint effort will be a strategy that will have a positive impact on the employment situation of persons with disabilities.

More Emphasis Needed on Employability

Program sponsors indicated that the program's emphasis on "employment" results fails to acknowledge the importance of incremental steps people with disabilities need to accomplish in order to become more employable.

Recognizing that people with disabilities often require a greater number of interventions over a longer period of time than do people without disabilities, the Opportunities Fund Accountability Framework was revised in January 2001 to include a means of measuring "increased employability", rather than focusing exclusively on "employment" results. The revised Framework now includes such key indicators as "number of interventions completed" and "number of Action Plans completed". In addition to this, the Terms and Conditions were revised to allow for agreements to have a duration of up to 78 weeks, in order to accommodate the longer-term assistance required by persons with moderate to severe disabilities. Local offices also have the capability to write concurrent agreements based on an individual's return to work action plan.

Program Implementation

Ineligible Clients Served

The evaluation data indicated a significant proportion of participants — between 8 and 13 percent — were eligible for Employment Insurance (EI) programs and therefore ineligible for assistance under the Opportunities Fund. There are several possible explanations for this to have occurred:

• many of the organizations serving people with disabilities receive funding from both the Employment Insurance program and the Opportunities Fund, making it sometimes difficult for them to track these funds and clients separately;
• a lack of understanding of the program's eligibility criteria on the part of delivery agents;
• a preference on the part of delivery agents to use the Opportunities Fund over other programs in assisting their clients with disabilities — either because of the perceived greater flexibility of the Opportunities Fund or greater familiarity with the program compared to others;
• data entry errors — some clients with disabilities who received assistance under EI were erroneously miscoded to the Opportunities Fund.

LMD worked collaboratively with HRDC's Internal Audit and Risk Management Services (IARMS) to conduct an audit of recipients to provide a reasonable level of assurance that the overall program is appropriately managed and that eligibility criteria for participants complies with the program Terms and Conditions. The audit results will be compiled in a report and shared with Treasury Board officials in October 2001, indicating that there continued to be percentage of EI-eligible clients served.

LMD has also worked closely with regional HRDC officials to develop and implement detailed instructions and guidelines to address client eligibility issues. Active and former claimants, miscoded as OF clients, have been identified and removed, and service delivery problems in each region have been addressed. Finally, a program-specific operational guide is being developed to assist regional and local HRCC staff in administering the program.

Eligibility Criteria Too Narrow

Some key informants indicated they did not favour the EI-eligibility criteria, arguing that existing mainstream programming is not sufficiently flexible, and suggesting the OF criteria be broadened to include EI-eligible individuals and those who are marginally employed.

Because of the limited funds that are available from the Consolidated Revenue Fund (CRF), the Opportunities Fund cannot address the employability needs of all people with disabilities. However, it has been recognized that in some cases, from a client service perspective, it may be more beneficial to assist an individual through the Opportunities Fund, even if he or she is eligible for assistance under the EI Part II programs.

In December 2001, the program Terms and Conditions were amended to allow, on an exceptional basis only, persons with disabilities who are eligible for Employment Insurance program to be assisted under the Opportunities Fund. Although this change in eligibility criteria will allow for more flexibility, the original program intent remains to assist persons with disabilities who do not qualify for assistance under the Employment Insurance program. Policy and operational guidelines distributed to regional staff will reflect this intent.

Gaps in Service

The report also identified gaps in client service with respect to individuals in remote communities, Aboriginal persons with disabilities, youth and persons with invisible or multiple/severe disabilities. While the broadened eligibility criteria will help improve service for clients who find it difficult to access other programs, it should be noted that other CRF-funded programs exist for serving the particular employment needs of certain client groups, such as Aboriginal and young people with disabilities.

Weaknesses in Follow-up, Monitoring and Reporting

The evaluation cites these areas of program implementation to be weak. Guidelines have now been established on appropriate procedures for follow-up and monitoring. These have been shared with employers and coordinators, with a continuing emphasis on the importance of tracking results for purposes of public accountability. Systems enhancements to better support the reporting function are also being developed in consultation with the regions.

1.1 Context and Program Description

The Opportunities Fund for Persons with Disabilities (OF) was launched in response to the Scott Task Force Report on disability issues tabled in 1996. A major recommendation of the Task Force was the establishment of an employment fund for persons with disabilities to fill a gap in labour market programming and to increase their participation in the labour force.

The objective of the OF program is to assist persons with disabilities in preparing for, obtaining and keeping employment or becoming self-employed, thereby increasing their economic participation and independence. The program is the responsibility of Human Resources Development Canada (HRDC) with the guidance of a National Reference Group. During the period under study, the National Reference Group is composed of representatives from national disability organizations and the majority of them had participated in the Scott Task Force. The National Reference Group and Opportunities Fund management met regularly to discuss and review program management issues. In the February 2000 budget, OF was renewed on a permanent basis with an annual budget of $30 million.

Potential OF clients must meet certain criteria to qualify for assistance under the program. First, they must self-identify as having a disability (a medical diagnosis is not necessary for participation). Second, potential OF clients must be legally entitled to work in Canada. Third, they must be in need of assistance to prepare for or obtain employment or self-employment, though they do not necessarily have to be job-ready. Fourth, OF clients must not be eligible for Employment Insurance (EI) benefits. Finally, clients must commit to an action plan designed to assist them into employment or self-employment.

A number of guidelines have been established for funding OF projects. These guidelines state, for example, that OF projects must be designed to move participants into paid employment or self-employment, complement existing programming funded through other sources, involve partners to lever funds and develop employment opportunities, provide post-placement follow-up and be accountable for results.

The types of interventions funded by OF can be similar in nature to Employment Benefits and Support Measures (EBSM) funded by EI such as targeted wage subsidies, job creation partnerships, self-employment and funding for workplace accommodation and adaptive equipment. In addition, the program supports volunteering activity, but EBSMs do not.

OF delivery is decentralized via HRDC regional headquarters, with the exception of 10 percent of funds that is devoted to national projects and since 1999-2000, another 10 percent of program funding is transferred to the Aboriginal Human Resource Development Agreement (AHRDA) for persons with disabilities. Based on their regional funding strategy, regional offices distribute the funds at their discretion to parent or local Human Resources Centres of Canada (HRCCs). HRCCs may, in turn, contract with third party organizations to deliver services to clients. In Quebec, OF is managed by the umbrella organization, Comité d'adaptation de la main-d'oeuvre pour personnes handicapées (CAMO). CAMO does not offer services directly to persons with disabilities but provides support to individuals and agencies that will offer those services and approves OF project applications.

1.2 Evaluation Issues and Approach

The evaluation framework for OF identified 27 evaluation issues that can be divided into four groups: rationale; implementation; impacts and effects; and cost-effectiveness. A formative evaluation completed in 1998 primarily addressed issues related to program design and implementation and identified areas for improvement. The purpose of this summative evaluation was to measure the effectiveness of OF in achieving its objectives and, in particular, to examine more closely the impacts and effects on participants. Evaluation activities have been conducted under the guidance of an Evaluation Committee drawn from the National Reference Group and with representation from HRDC at the regional level.

The methodological approach for the evaluation was based on multiple lines of evidence, including both qualitative and quantitative approaches. Key sources of data included:

• Policy and program documentation review;
• Key informant interviews with 30 representatives from HRDC, the National Reference Group and project sponsors;
• Survey of 152 employers involved in targeted wage subsidy contracts under OF;
• Analysis of administrative data assembled from HRCCs, HRDC and third party service providers;
• Survey of 1,034¹ OF participants;²
• Survey of 490 comparison group cases.³ The comparison group cases were drawn from HRDC administrative files — the National Employment Services System (NESS), Standard Data Files and Human Resource Investment Branch files — then linked to available disability flags and finally matched to the known earnings, EI and social assistance usage of OF participants;
• Self-administered survey of project sponsors (n=99);
• Six focus groups with OF participants and stakeholders; and
• Five case studies of OF-sponsored projects illustrative of program delivery.

1.3 Scope of the Evaluation and Limitations

The participant survey for the summative evaluation of OF was conducted during the first three months of 2000, and it covers projects undertaken in the first two years of the program. Participants were roughly equally split in terms of when their intervention was completed (45 percent in 1998 and 53 percent in 1999).

Because a comprehensive national listing of OF program participants was not available, the sampling frame for the survey of participants had to be compiled specifically for the evaluation. The sampling was based on three data sources:

• Lists of participants supplied by regional offices from HRCCs in Alberta, New Brunswick and BC and from the CAMO in Quebec;
• Lists of participants forwarded by OF third party deliverers or project sponsors;
• OF participants listed on the Department's National Employment Services System (NESS) which included some HRCC case-managed clients and some participant data uploaded from Contact IV from third parties.

All survey data deriving from participants listed on NESS were later excluded (representing 766 participants). This was due to coding errors discovered on NESS that misidentified some individuals as OF participants. Because the participant list submitted by BC was also collected from NESS, these data were eliminated and therefore, no direct observations are available for this region.

While the summative evaluation survey sample was altered from its original design because of the unanticipated data integrity issue, it still provides a great deal of meaningful information about the program.

The participant survey was conducted by telephone, as this method provided an effective means of gathering information from a large number of participants in a structured way within a short period of time. To enhance opportunities to provide feedback, a TTY line was made available, as well as options for self-administered and proxy completion of the survey. Yet, it is possible that given the nature of survey data collection, the data may under-represent persons with some types of disabilities such as intellectual disabilities and hearing or speaking disabilities. Finally, the evaluation report looks at cost-effectiveness of the program, as well as leveraging of funds, and participants' use of income transfers and employment earnings gains. However, evaluation questions pertaining to the unit costs of delivering different types of interventions under OF could not be answered due to limitations in the administrative data on program cost by type of intervention.

1.4 Organization of the Report

There are four chapters following this one in the report. Chapter Two presents a profile of OF participants and their program activities based on the survey sample. Chapter Three addresses program rationale, design and implementation issues. The impacts and effects of OF are examined in Chapter Four. Conclusions are presented in Chapter Five.

2.1 Profile of Participants

The following sections provide a profile of Opportunities Fund for Persons with Disabilities (OF) participants surveyed for this evaluation. As mentioned previously (Section 1.3), it should be noted that the survey sample does not include OF participants listed on the National Employment Services System (NESS) and participants from BC.

Socio-Demographic Profile

Survey results indicate that there was roughly an equal gender distribution among OF participants (51 percent men, 49 percent women). The average age of OF clients was 38 years. One in 10 participants were youth (under 25 years old) and one-third were 45 years or older. One in five OF participants had not completed high school, though almost half (48 percent) reported having at least some post-secondary education. Less than one-third reported their current status as married (roughly consistent across regions). Two-thirds (68 percent) of OF participants did not have dependents. Almost one in five OF participants indicated an equity group status unrelated to their disability, with five percent indicating they were Aboriginal and 12 percent being members of a visible minority group.

For reference purposes, OF participants were compared to participants in regular Employment Insurance (EI) Employment Benefits and Support Measures (EBSMs).⁴ The distribution of participants by sex is comparable. OF participants, however, have lower levels of education, have a somewhat older age profile and are less likely to be married and to have dependents. Anglophones are over represented in the program and OF participants are more likely to be visible minorities.

Work Experience Profile

Overall, OF participants had only a moderate attachment to the labour market prior to program participation, with six in 10 not having been employed in the two years prior to their intervention. Those who had been employed in the two years prior reported having been with their employer for 4.5 years on average. The majority of those who worked were in full-time jobs. Weekly wage rates of OF participants were fairly low — about $278 compared to a national average of $596.⁵

The majority of participants who were working two years prior were no longer at this job one week before starting OF (70 percent). The most frequently mentioned reason for the job termination was layoff/lack of work/contract ended (43 percent). Just over one in seven (16 percent) left their job as a result of their disability (higher among those with disabilities affecting mobility or agility at 23 percent).

Many OF participants did not have any significant or recent experience in the labour market. Among those who had not worked in the two years prior to participation in the program, one-half had not worked for five years prior to entering OF and 12 percent had never worked at all (higher among those with intellectual disabilities at 28 percent). For most of these individuals, their main activities during this period of time were looking for work (55 percent) and going to school (30 percent).

Income/Earnings/Income Transfer Profile

An analysis of Human Resources Development Canada (HRDC) participant administrative data indicated that earning and income related information was not available for a significant portion of OF participants. At least 40 percent of them did not submit records for taxation purposes, during each of the three years prior to participation in the program. As a result, income and earnings data obtained from these administrative data might have overstated actual averages among OF participants assuming that non-filers are most often those with no income or earnings to report. Among those who had information available, OF participants had consistently low employment earnings in each of the three years leading up to their participation in the program (between $2,473 and $3,349 annually among those with earnings information). Total income was also low during this period. Total annual income was less than $10,000 with less than half of this amount coming from employment earnings.

Consistent with participants' limited prior work experience, use of Employment Insurance (EI) in the period prior to the program was relatively low. However, while the stated program target is non-EI eligible clients, 13 percent of OF participants had collected EI in the three years prior to program participation. The issue of EI eligibility is discussed in more detail in Section 3.1(b). The amount of social assistance received was fairly consistent in each of the three years prior to program entry (between $4,117 and $4,433 annually among those who received benefits during these years).

Profile of Disability

According to the survey results, the majority of OF participants reported having one disability, while 22 percent indicated having more than one type of disability. Just over half of OF clients indicated agility and mobility as the nature of their disability. Twenty-one and 18 percent of participants each reported a psychiatric and intellectual disability. A visual disability was reported by 12 percent of clients and speaking and hearing disabilities were each mentioned by less than one in 10. Illnesses and systemic conditions (e.g., blood disorders, kidney disease, environmental sensitivities, epilepsy) were together mentioned by one in 10 participants. One percent of OF participants reported having no disability⁶ and another six percent either did not know or refused to specify the nature of their disability.⁷

The disability profile of OF participants is somewhat similar to the overall profile of Canadians with disabilities based on the 1991 Health and Activity Limitation Survey (HALS) data. While the methods of determining type of disability are quite different, HALS also found that the majority of persons with disabilities have agility (50 percent) or mobility (53 percent) disabilities.⁸

Eight out of 10 participants indicated other than job-related causes of their disability. The distribution of OF participants' self-assessed severity of their disability in terms of the extent of limitations at work was roughly equal across mild, moderate and severe ratings. This suggests that, extensive "creaming" in the program may not be an issue and that employability outcomes reflect a balanced portfolio of clients in the program.

2.2 Profile of Program Activity

Table 2.1 presents the yearly allocation of funds for OF by region and for the program overall, based on administrative data. A total of $30 million is allocated on an annual basis for OF. The majority of funds are distributed to regions, while 10 percent of OF dollars are reserved for national projects, and in 1999-2000 another 10 percent was transferred to Aboriginal Human Resource Development Agreement (AHRDA) for persons with disabilities. The regional allocation is distributed on a per capita basis based on the estimated number of working age persons with disabilities in each region.

Because there are no complete administrative data available at the national level on OF program activities, the profile of program activity is based on survey data (Table 2.2). According to these data, the majority of OF participants (72 percent) received more than one intervention as part of their participation in the OF program. Almost half (48 percent) of interviewed participants reported meeting with an employment counsellor. Of those who met with a counsellor, the contact was typically individualized (92 percent reporting one-on-one sessions) and intensive (94 percent had more than one contact and 45 percent reported six or more meetings).

The use of action plans was limited, with just over one-third (36 percent) of participants reporting developing a return to work action plan. This was confirmed in the focus groups where few participants appeared familiar with the action planning process. However, it should also be noted that the limited apparent use of action plans may be at least partially a reflection of clients not recognizing this terminology or because the process of assessment and case planning often may be quite informal. The majority of those who undertook an action plan (86 percent) reported completing this plan at the time of the interview.

The types of program interventions in which program clients participated were broadly similar to those offered under EI Part II. The most commonly cited interventions by participants included: job placement or wage subsidy by 52 percent of OF clients; employment counselling (49 percent); education or training course (45 percent); and job search program (44 percent). Volunteering was mentioned by 25 percent and 18 percent received funding for equipment or specialized assistance related to their disability.

Reflecting the differing needs of OF participants from "regular" EI clients, the program activity profile suggests that a wider-range and different mix of activities is occurring under OF. For example, the majority of OF participants received more than one intervention under OF. Volunteering is an intervention supported under OF, unlike EI Part II employment benefits and support measures. As well, the use of work placements and funding for specialized equipment and job accommodations was much more prevalent under OF than it is for EI employment benefits and support measures. The use of individualized and sometimes intensive approaches to respond to varied barriers was viewed as a strength of OF and a key factor of success of employment interventions.

In terms of the work placements, it is noted that in some cases OF third parties were involved in both client case management, as well as providing a work placement within their organization (almost six in 10 reported participating in OF in this way).

The majority of participants completed the full period of their OF program: 78 percent of participants in training and 82 percent of job placement/work experience clients had successfully completed their intervention at the time of the interview. Reasons for attrition were most likely to be attributed to leaving to take a job (more likely for job placement participants) or due to their disability.

For most participants (79 percent), OF was their only source of assistance in finding or preparing for work during the period under study.

3.1 Program Rationale

Relevance

The formative evaluation established that the rationale for Opportunities Fund for Persons with Disabilities (OF) was based on both equity and efficiency arguments. The program was to address barriers to labour market opportunities for persons with disabilities and to provide individual and societal benefits by maximizing self-sufficiency through employment. The inclusion of persons with disabilities also expands the number of Canadians who are able to make a productive contribution to the economy and to society and maximizes the labour market pool for employers.

Key informants canvassed in the summative evaluation generally believed that barriers to employment for persons with disabilities persist and that there is a continuing need for interventions to address these issues. Though there is limited empirical data available in Canada to examine the status of persons with disabilities with respect to employment,⁹ the available evidence supports this belief.¹⁰ For example, the employment rate among persons with disabilities is much lower than the rest of the working age population in Canada. Persons with disabilities also experience higher levels of long-term unemployment. Levels of income and education are lower among persons with disabilities and in general, this group is more likely to be on social assistance. The research literature further suggests a trend toward greater incidence of disability in the future owing to an aging population¹¹ and of increased interest among persons with disabilities in entering the labour market (this observation is based on United States experience).

There was agreement among key informants that the federal government should continue to play a role in addressing employment barriers for persons with disabilities. The suggested roles for the federal government included:

• providing leadership to ensure other stakeholders such as provinces and industry give a high priority to disability issues;
• provision of programs/supports to persons with disabilities and employers to further employment goals;
• ensuring that the federal government is a model in its own hiring practices;
• facilitating research to provide the basis for policy decisions; and
• playing an educative role through dissemination of research on good or promising practices, as well as promotion of the broad goals of the government's agenda to employers and the public.

While there is strong support for a federal program such as OF, key informants also noted the limitations of the approach. Various strategies such as OF, the Employability Assistance Program for People with Disabilities (EAPD) and provincial health and social service initiatives lack coherence and their implementation is highly variable. The view of many of those in the disability community is that OF should be a component of a more comprehensive labour market strategy for persons with disabilities to address barriers. These barriers include, for example, attitudinal and systemic barriers in the workplace, lack of transportation and job accommodation, disincentives in income and disability support, lack of appropriate education and workplace skills and lack of job maintenance supports.

A more co-coordinated approach is the preferred direction stated, for example, in the published proceedings of the Winnipeg Opportunity Works! Conference (1999) that concluded that OF should be linked with other strategies "within a broad vision for labour market development". This is a position that is broadly consistent with the current agenda of federal government policy on disability issues.¹²

Program Targeting

The requirement that OF participants not be eligible for Employment Insurance (EI)-supported interventions (by virtue of their eligibility for EI or reach-back status) was the most significant program targeting issue raised in the study. The data from the summative evaluation indicated that 13 percent of program participants during the period under study were eligible for EI Part II benefits and, therefore, not eligible for OF. Recognizing the importance of this issue, a subsequent review was undertaken by program management. Based on the National Employment Services System (NESS) database, regional/local offices were asked to review and verify samples of OF clients. This review suggested that about eight percent of OF participants were in fact eligible for EI Part II benefits. Consequently, the proportion of EI eligible clients was in the range of eight to thirteen percent, with the differences in the estimates being attributed to two slightly different timeframes of client groups and clientele history covered. However, regardless of the absolute percentage, program management is now in the process of addressing this issue of program delivery and implementation.

It should be noted that some program deliverers in the key informant interviews and focus groups did not favour the EI-eligibility criteria, arguing that existing mainstream programming is not sufficient or flexible enough in some cases to meet the particular needs of persons with disabilities. Some key informants also suggested that the program should also serve those who are currently employed or marginally employed and wish to improve or expand their participation in the labour market.

Other gaps that were identified in terms of clients not well served by the program included: individuals in remote communities; Aboriginal persons with disabilities; persons with "invisible" disabilities (e.g., learning, brain injury, mental health, developmental delays); youth; and persons with multiple and/or very severe disabilities.

3.2 Program Design and Management

Principles and Guidelines

As described in the previous section, there are a number of principles and guidelines that are in place to support the implementation of OF. The formative evaluation found that projects sponsored under OF were generally consistent with the program guidelines, with particular challenges being in the area of establishing partnerships with the private sector and leveraging of funds. This evaluation found that the program EI-eligibility criterion was not consistently applied as noted above (Section 3.1b) and monitoring of OF program activity and results has been weak (discussed in more detail below).

Suggestions to improve the program's principles and guidelines involved adjusting some of the underlying notions and reinforcing existing ideas. Suggestions provided by key informants included, for example:

• Adjusting the focus on "employment" to include "employability". The employment focus was generally viewed as a strength of the program leading to greater creativity and discipline in use of funds. However, many key informants preferred the broader concept of employability that would more properly reflect the incremental steps toward employment that many OF participants must take as they move from unemployment to employment. The emphasis on employability would, in their view, avoid the potential for "creaming" participants and would permit a greater range of interventions such as pre-employment interventions that are often necessary to achieving employment.
• Including a recognition that yardsticks and timeframes for outcomes will vary depending on the client group. Clients with multiple or more severe disabilities cannot be expected to achieve results similar to those of other client groups.
• Recognizing that OF should be part of a holistic approach. Quite often, successful approaches may need to address issues beyond those directly related to employment such as transportation, housing, and drug benefits.
• Focusing not just on addressing the needs of persons with disabilities to achieve employment, but also on issues in the work environment (systemic and attitudinal) that present barriers to employment, including elaborating upon the role of employers in the program.
• Reinforcing the need for participant follow-up and support following employment.

Program Communications

The formative evaluation of OF reported that communications with potential project sponsors had been weak during the early implementation of the program. This appears to have been largely resolved as the program matured. Most key informants canvassed during the summative evaluation believed that the disability community is now well aware of the Opportunities Fund.

In terms of promotion of the program to participants, there generally appear to be few difficulties in attracting clients to OF-sponsored projects; in fact, just over half of project sponsors indicated that they have waiting lists for their project. The most common way that participants become aware of or recruited into the program is through informal or formal referral by a disability or other community-based organization or through Human Resources Development Canada (HRDC). There are fewer efforts to promote the program more broadly to potential participants who are not associated with these organizations. This equity of access issue is not easily resolved, however, given limitations in the capacity of the program to deliver services to all who may be interested.

Program Management

The consultative model and explicit inclusion of organizations serving persons with disabilities in the management of OF was viewed as an innovative, important and positive distinguishing feature of OF. The National Reference Group was viewed as providing a valuable link between government and the disability community. While the goals and intentions of establishing a National Reference Group were lauded, there were varied opinions about the future direction of the group. Some key informants simply preferred more involvement of the group in management and decision-making concerning the OF program. Others believed that the size and structure of the group must be streamlined, in consultation with the disability community, to enhance its future effectiveness.

At the local level, involvement of the disability community has also increased over time. The quality of the relationships depends on the level of experience and commitment to inclusiveness of local Human Resources Centres of Canada (HRCCs), as well as knowledge of disability issues and the local disability community. Most project sponsors were satisfied with their level of involvement in the program.

Another management issue that was raised was with respect to the effectiveness of the relationship between HRDC national headquarters and the regions. Some project sponsors, for example, felt that there were inconsistencies in the vision and interpretation of OF guidelines across HRCCs resulting from a lack of clear communication about the program. This could be a natural by product of HRDC decentralization of decision-making and also may have been exacerbated by staff turnover. Many regional HRDC representatives also indicated weaknesses in communications with respect to the activities of national projects and believed that these funds would be better spent at the regional level.

Follow-up, Monitoring and Reporting

According to project sponsors, participant follow-up is a significant aspect of service delivery and 96 percent reported following up with their clients after the end of the intervention. While ongoing support was cited as a key factor contributing to successful outcomes, in the qualitative interviews and group discussions, project sponsors voiced concerns about the adequacy of funds for follow-up. OF contracts do not provide dedicated funds for follow-up activity. This expense then becomes the responsibility of the sponsoring organization.

While virtually all project sponsors provided follow-up, this support was not confirmed by all participants (or was not recalled or recognized as such by participants). Under one-half of surveyed OF participants (46 percent) recalled follow-up after the completion of their OF intervention. OF participants in the focus groups, however, were very interested in greater follow-up to provide feedback on programs and for ongoing support.

During the formative evaluation, the issues of monitoring and accountability were found to be troublesome. The data that were being collected at that time were found to be insufficient to support a summative evaluation.¹³ This evaluation study indicated that program monitoring and reporting continues to be problematic. Data regarding the number and profile of OF participants, as well as program activity and employment outcomes were not systematically available at the national level.

There are challenges to monitoring a program delivered primarily by third parties. Just over one-quarter (28 percent) of project sponsors indicated using the Department's Contact IV reporting software system for third parties to record client data (only a marginal improvement from 21 percent in the formative evaluation). Lack of awareness was the most common reason organizations were not using the system. Sponsors' use of custom electronic or manual systems to monitor and report client activity is not conducive to program-wide reporting. According to some sponsors, however, the requirements for monitoring were not made clear at the outset of program start-up, and, as a result, they had not put in place adequate procedures for client notification and release of information for program monitoring and evaluation.

The adequacy of current program performance measures was questioned in the evaluation. Employment and savings are the two measures that were established for the program at its inception. There is an interest in ensuring that these measures be extended and refined to: better reflect a preferred broader emphasis on employability that includes not only employment but also other incremental steps toward employment and be more sensitive to the unique issues of employment initiatives targeted to persons with disabilities. Program performance measures should allow for longer timeframes for program results and variable yardsticks to take into consideration the different circumstances associated with disabilities and the process of adaptation to labour market activities.

3.3 Implementation

Partnerships

Partnered delivery models were prevalent in the vast majority of OF projects, with 90 percent of project sponsors delivering their project with other partners. This represents a substantial increase from the two-thirds who reported partnering in the formative evaluation despite the often steep time commitment required to establish and nurture partnerships. Not all partnerships are newly formed as a result of OF, though the program was seen as reinforcing partnered delivery models and providing an opportunity to reinforce existing collaborations. Examples of the types of contributions from partners included: providing employment experience, client referrals and advice/consultation.

The most frequently mentioned partner organizations were other non-profit/voluntary disability organizations (67 percent). The provincial government and other non-profit organizations such as rehabilitation organizations and health or mental health units are the next most frequently mentioned partners in the delivery of OF (similar to the formative evaluation results). The role of private sector employers in the program has increased substantially since the formative evaluation (53 percent of project sponsors reported partnering with employers compared to 25 percent in 1998). Some project sponsors noted that their contact with industry has largely been limited to individual employers providing work placements and they have been somewhat less successful in bringing employer organizations to the table as partners. Involvement of unions has remained marginal.

One of the original intents of OF was that the program would be delivered in a complementary way with funds available from Employment Assistance Services (EAS) under EI Part II. EAS provides assistance to individuals who are unemployed (regardless of their EI eligibility) to prepare, obtain and maintain employment. Services may include counselling, job search skills and labour market information. Similar to the formative evaluation, the results of this evaluation suggested that EI Part II funds were not used extensively for OF clients. According to key informants, the negotiation of Labour Market Development Agreements (LMDAs) with the provinces (with the exception of Ontario, which does not have an agreement) may have complicated this effort. During the evaluation period, LMDAs were in the early stages of implementation in many provinces and the transition to new models of delivery of EAS and other EBSMs hampered the use of EI funds for OF clients. The exception was in Ontario where, the qualitative evidence suggested that, because both OF and EI Part II were delivered federally out of HRCCs, there was more use of EAS under EI Part II.

Dissemination of Information

One of the guiding principles of OF is to provide opportunities to share practices and experiences. The vast majority of projects sponsors (93 percent) were able to share best practices and experiences with others in the field, though this was typically accomplished informally through networks and colleagues. Professional associations and umbrella organizations also provided this kind of information to their members. The Internet was mentioned as a tool for information dissemination by 28 percent. The national Opportunity Works! Conference in Winnipeg in 1999 was also mentioned as providing an occasion for sharing best practices. However, there are few of these kinds of venues at the regional or provincial level.

There is an interest in more ongoing and structured exchange among those delivering OF projects. When asked how sharing of practices could be improved, project sponsors suggested: newsletters/website (43 percent); building a network for ideas (34 percent); and conferences/presentations (29 percent). In addition, program deliverers in the key informant interviews and focus groups suggested that it might be more effective to exchange lessons learned in smaller forums with a tighter focus on specific interventions or aspects of delivery.

Satisfaction with Service Delivery

The study evaluated satisfaction with the program and service delivery from the perspective of program participants, employers and project sponsors. Satisfaction with the OF program overall was high among both participants (84 percent very satisfied) and employers (95 percent satisfied). Satisfaction with specific interventions was higher among participants involved in work placements, training or counselling and somewhat lower for those who received support to start their own business.

Employer participants expressed a high level of satisfaction with support provided by HRDC and administrative aspects of the program (e.g., application and reporting processes). In terms of the program intervention, the wage subsidy was highly valued. The vast majority of employers (90 percent) believed that OF participant(s) had gained workplace skills as a result of the program and a similar proportion indicated that the OF participant's work was of value to the organization. Virtually all employers (97 percent) said they would recommend participating in the Opportunities Fund to another employer and a similar proportion (93 percent) said they would participate in projects under the Opportunities Fund again in the future.

Project sponsors' satisfaction with OF was also examined. Just over three-quarters of project sponsors indicated being satisfied with the flexibility of the program and its application and approval process (78 percent for both aspects). Timeliness, use of flexible and individualized approaches and reasonable administrative requirements were identified as key program strengths in the qualitative interviews. Lower satisfaction levels were noted with respect to the clarity of program guidelines with just 69 percent of project sponsors indicating satisfaction in this area. Aspects that fared more poorly included the availability of information on OF and the OF program criteria (66 percent and 65 percent satisfied, respectively).

A rated weakness in the program by both OF participants and employers was with respect to the initial information that was available to them about the program (16 and 24 percent dissatisfied respectively). About one-third of OF participants and one-quarter of employers also reported experiencing barriers to participating in the program. For participants, obstacles included health difficulties (seven percent), as well as lack of prior work experience making their placement more difficult, skills barriers, conflict with personnel/co-workers, and lack of sensitivity to needs/discrimination (each at four percent).

Many of these issues were echoed in the focus groups with participants who also mentioned issues such as unrealistic expectations of employers and lack of appropriate jobs in the broader economy. Some participants also admitted to having concerns about losing income and health-related benefits in returning to work. For employers, barriers identified included the need for or cost associated with workplace adaptations and difficulty with the administration of the project.

From the perspective of project sponsors, the uncertainty (at the time of the survey) about the renewal of the program was a key challenge. Moreover, in the survey of project sponsors, the most frequently mentioned recommendation (by 41 percent of respondents) was to increase funding for the program. About one-third of project sponsors recommended longer-term programming in the survey. The qualitative interviews and discussion groups also found a great deal of frustration regarding year-to-year funding arrangements under OF. Both uncertainty about program renewal and short-term funding were reported to have repercussions for the ongoing planning of activities, use of longer-term interventions, staff continuity and proper recruitment of clients.

The recent federal withdrawal from block purchase of training drew criticism from some of those involved in program delivery. There have been some challenges in appropriately communicating the meaning of the policy shift to OF program deliverers. At the same time, however, there are implications in terms of the extent to which the specialized training often required by persons with disabilities can be accommodated within the shift to individualized purchases. Problems with the approach were cited to include: difficulties for organizations in planning and promoting offerings; specific challenges for some persons with disabilities who lack the capacity to decide on interventions on their own; and accountability issues for training deliverers.

Gaps in some types of programming available under OF were identified as another challenge by those involved in program delivery. Perceived gaps included: entrepreneurship assistance, pre-employment/literacy training, interventions focused on higher-skilled occupations and sectors, and disability supports (equipment and assistive devices).

Incrementality of Assistance

The evaluation surveys examined incrementality of support provided by HRDC under OF — that is, the extent to which the program activity or interventions would have taken place without the support of OF.¹⁴ Overall, OF assistance was perceived to be incremental by the majority of OF participants, as well as employers and project sponsors. For example, almost three-quarters of participants in job placement and training interventions (72 and 71 percent respectively) said that OF made it possible for them to participate in these programs. Only nine percent of training and eight percent of job placement participants thought they would have entered their intervention without the assistance provided by OF. Incrementality of support was somewhat lower for OF participants who received assistance to start their own business (63 percent). Survey data also confirmed that, prior to participating in OF, about one in five of those who had received self-employment assistance were already working at starting their own business at that time.

The majority of project sponsors (77 percent) also reported that the projects they were involved in would not have taken place without the program. Of those surveyed, only six percent thought services and activities would have taken place without OF funding. Finally, incrementality was supported from the employer perspective: 88 percent of employers indicated that they would not have hired the OF participant if the financial assistance from the program had not been available.

4.1 Participant Impacts

The Opportunities Fund for Persons with Disabilities (OF) evaluation examined program impacts for participants in terms of employment outcomes, as well as the broader notion of employability, including development of capacity for employment and removal of barriers to employment. The study also examined non-labour market impacts of the program (e.g., confidence, and optimism). The evaluation includes participants' own assessment of the impact of the program, as well as objective employment results compared to those of non-participants.

Participants' Assessment of Impacts

From OF participants' perspective,¹⁵ the program was successful both in improving their employability, as well as their quality of life. In terms of employability, for example, over three-quarters (77 percent) of participants rated OF very useful in providing work experience. Seven in ten rated the program useful in gaining specific job-related skills (71 percent) and a similar proportion believed the program had been useful in increasing their ability to find work in the future (69 percent). Impacts of the program on quality of life such as increasing self-esteem/well-being (75 percent) and increasing participants' satisfaction with their work life (71 percent) were also highly rated. Mid-range impacts were increasing social contacts and networks in the work world (63 to 66 percent rated OF useful in this respect). OF was rated useful to a lesser extent in terms of improving participants' physical health (51 percent) and increasing participation in volunteer work (48 percent).

Two-thirds of surveyed OF participants indicated that OF had reduced barriers to employment. In terms of the types of barriers to employment that were addressed by the program, the most frequently mentioned — by three in ten participants — was overcoming a lack of confidence and self-esteem (higher among those with speech and psychiatric disabilities). Other barriers mentioned as being addressed by OF (in order of frequency) were: lack of proper assistive devices (13 percent); poor skills/training (12 percent); and lack of work experience, lack of contacts/networks, and the need for job search skills (7 to 8 percent respectively).

Employment Outcomes

Table 4.1 presents the employment outcomes for OF participants and for the comparison group. The vast majority of OF participants indicated an interest in entering the labour market following completion of their intervention. Following completion of their OF program, 59 percent of participants actively searched for work for an average of about 24 weeks (from the end of their intervention until they found a job or until the time of the survey). Non-participants (the comparison group) were looking for work for the same amount of time. Among those who did not look for work, the most commonly mentioned reason was that the participant had found a job immediately after their intervention (51 percent). One in five indicated that they did not look for work due to their disability, and the remainder cited other reasons.

Overall, 60 percent of OF participants reported that they had had a job (full-time, part-time, self-employment) since the end of their intervention. Again, this result is little different from the outcome for the comparison group. Among this group, roughly two in five said they had worked for private sector organizations. The majority of participants (62 percent overall, but higher in Atlantic) had had only one job during the post-program period, with the average number being 1.7.

At the time of the interview, almost one-half (49 percent) of OF participants were employed (self-employed, full-time employed or part-time employed) with the rate of employment being somewhat lower in the Atlantic (37 percent) and much higher in Ontario (58 percent). Just over one-quarter of OF participants (27 percent) were unemployed and looking for work at the time of the interview (higher in the Atlantic provinces and among male participants at 32 percent). The remaining one-quarter of participants were considered to be out of the labour force (i.e., unemployed and not looking for work, student, homemaker, on disability leave). Rates of current employment were somewhat lower among participants with psychiatric disabilities, but were higher among those who had received OF assistance to start their own business or for specialized equipment or assistance. The results for OF participants are more positive than the comparison group, with the OF group being slightly more likely to be employed at the time of the interview by four percentage points.

To test the sustainability of employment over time, employment rates among early and later entrants into the OF program were compared. The rate of employment does not vary significantly among participants who started their intervention in 1997, 1998 or 1999 indicating that participants' employment appears to be sustainable over time.

Of respondents who reported obtaining employment during the post-program period, the majority were working full-time hours at their current/most recent¹⁶ job (60 percent). Of those who were working part-time hours, most participants (63 percent) indicated that they would prefer to be working full-time hours. Average gross weekly earnings for OF participants in their current/most recent job were about $303. The comparison group was more likely than OF participants to be working full-time hours and this was reflected in higher earnings (about $150 more per week).

Four in ten participants who had received a work placement as part their OF intervention reported that they were employed with the same employer at the time of the survey. This rate was higher among those who had a placement with a private sector organization (46 percent) compared to participants who worked for a community-based organization (24 percent).

Respondents were asked to rate the importance of OF in helping them to get their current/most recent job. Just over one-half of OF participants (55 percent) rated OF very important in this respect (5, 6 or 7 on a 7-point scale). These results are more positive than the ratings provided by the comparison group of employment services they had received during the same timeframe.

The proportion of time spent in employment versus active job search in the post-program period was examined. The results showed an initial advantage for OF participants: OF participants were working 54 percent of weeks in the post-program period compared to 47 percent for the comparison group. Conversely, the OF group spent less time in active search, though the difference is narrower: 32 percent of weeks looking for work compared to 34 percent for the comparison group.

OF participants' annual average personal and household incomes in 1999 were $12,192 and $23,596, respectively. This is similar to the comparison group but significantly less than national estimates for Canadians as a whole, which were $28,360 and $51,500 respectively in 1999.¹⁷

Gender analysis

The employment outcomes of program participants were further analysed by gender and the results are summarized in Table 4.2. Overall there were few differences by gender based on this survey sample. Post-program employment status and household income level were the two notable differences that were statistically significant.

The household income difference is most likely (or partly) due to family composition, with female participants being more likely to be married to spouses with employment earnings or other income. However, the difference in employment status shows that male participants are more likely to be unemployed and looking for work, while female participants are more likely to indicate being out of the labour force at the time of the survey.

Non-Labour Market Impacts

The impacts of OF on aspects other than employment were explored in the survey and the qualitative research. The evaluation survey data indicated that the majority of OF participants have a great deal of confidence in accomplishing their future goals and expressed optimism about the future. Just over one-half (53 percent) of OF participants have no trouble keeping a positive attitude. But more than one-third (36 percent) of OF participants indicated that they "have trouble keeping a positive attitude" (higher among participants with psychiatric disabilities). The results indicated that OF participants had only a slight advantage over non-participants in terms of their rated confidence (by three percentage points), but were somewhat more likely to report keeping a positive attitude (by seven percentage points).

This subject was also broached in the focus groups. When asked about the broader outcomes of their involvement with OF, participants in the focus groups mentioned, for example, improved self-confidence and self-esteem, improved knowledge and management of their disability, positive health benefits, networking and support, and access to workplace accommodations. It should be noted, however, that several focus group participants cited some unintended impacts of OF, namely, a loss of confidence and cynicism when interventions did not lead to the expected employment results.

Incremental Impacts

An essential question in the analysis of the impacts of OF is the incremental impacts of the program. Despite precautions taken to construct a comparison group of non-participants with characteristics similar to participants, simple comparisons between program participants and the comparison group on key outcome indicators (e.g., employment) may still yield a biased estimate of program impact because of remaining differences between the two groups. In order to ensure that the differences in measured outcomes were not the effect of pre-existing differences between the participant group and the comparison group, multivariate analyses were conducted to control for these differences. The impact of OF was tested in this way for key employment, earnings, income and attitudinal outcomes (optimism, confidence, and ability to keep a positive attitude).

Table 4.3 presents the coefficients, their significance level and computed percentage probability¹⁸ of the OF program participation variable for each of the dependent variables tested in the multivariate analyses, with and without current/former Employment Insurance (EI) clients included in the sample. The multivariate results indicated that participation in OF has a significant impact on employment in three key areas:

• Participation in OF increased participants' likelihood of being employed at the time of the interview by 15 percent with EI clients in the model and by 20 percent without EI clients in the model;
• Participation in OF increased participants' likelihood of ever having had a job in the post-program period by 10 percent with EI clients in the model (p<0.05) and by 16 percent without EI clients in the model; and
• Participation in OF increased the proportion of weeks worked in the post-program period by between 15 percent (with EI clients in the model) and 17 percent (without EI clients).

Participation in OF did not affect receipt of social assistance and other income support, percent change in weekly wage rate, and personal income. Participation in OF also had no significant impact on attitudinal outcomes (confidence, optimism, ability to keep a positive attitude).

Factors of Success

Based on the survey data analysis, a number of respondent characteristics were found to influence employment outcomes for both OF participants and non-participants. For example, stronger and more recent employment experience, as well as higher earnings prior to participation in the program improved the likelihood of post-program employment. Higher self-assessed interest in entering the labour market was also associated with positive outcomes. Participants from the Atlantic provinces and Quebec had a lower likelihood of positive employment outcomes compared to other regions. Respondents' ratings of the severity of the disability and having a psychiatric disability were linked to a lower likelihood of finding employment.

While the statistical analysis found few significant differences in outcomes based on the type of intervention that was received under OF, those involved in the delivery of OF identified a number of key factors of successful delivery models used under OF. Use of an individualized and holistic approach to address the needs of participants was identified as the most important success factor. This was also in line with client expectations articulated in the focus groups. These participants talked about the need for service providers to have knowledge of and sensitivity to the limitations of particular disabilities and to provide support and advocacy.

Other aspects of successful interventions that were identified included:

• providing ongoing support following the end of the intervention (e.g., in terms of job search, job coaching);
• partnership approaches with organizations with particular areas of expertise (e.g., in job accommodation, training delivery for persons with disabilities);
• involvement of employers/employer champions;
• use of work placements that feature: on-the-job coaching; committed champion and trainer within the employer organization; committed champion for the program at the senior management level; and use of contractual obligations or other means such as monitoring and follow-up to ensure employers hire following the end of a wage subsidy.

Work placements (or targeted wage subsidies/internships) were frequently used interventions in OF. The perceived effectiveness of this delivery model was mixed, however. Many program deliverers and Human Resources Development Canada (HRDC) personnel strongly favoured work placements as a way for participants to gain work experience and as providing a strong potential for ongoing employment by establishing a direct connection to an employer. Some OF participants in the focus groups were more cautious about work placement experiences, noting that the duration of the placement was too short, work was often not meaningful, and there was little appreciation for job accommodation. In addition, many participants and project sponsors considered finding means to ensure hiring beyond the term of the wage subsidy period an important issue to address.

4.2 Employer Perspective

Among employers who participated in an OF work placement or targeted wage subsidy agreement with Human Resources Development Canada (HRDC), only two percent indicated that the program had no benefits for their organization. The most frequently mentioned benefit of the program (mentioned by 45 percent of employers) was a skilled, productive employee. Other benefits included: lower labour costs (36 percent); increased awareness of disability issues (20 percent); community involvement (14 percent); and, related to this, enhanced profile among customers (13 percent); and completion of a specific work task (13 percent).

The impact of OF on organizations' hiring practices and perceptions is modest: about one-third of employers indicated that the program had had an impact in this area to some extent. The impacts on hiring mentioned by these employers included: greater awareness and knowledge of the capabilities of persons with disabilities (54 percent); greater likelihood to consider a job candidate with a disability in the future (16 percent) and understanding that workplace accommodation is manageable (4 percent). Fourteen percent (representing about 8 percent of the overall employer sample) indicated that their participation in OF had resulted in their organization being less likely to consider a prospective employee with a disability.

Almost two-thirds (63 percent) of employers reported that they had continued to employ at least one OF participant after program funding ended. The majority of these employers hired workers on a full-time, permanent basis. Among those employers who had not hired all their OF participants, the most frequently mentioned reasons for not hiring were: lack of funding/end of subsidy (30 percent); lack of work (28 percent); and participant not interested or available (26 percent).

4.3 Organization Impacts

While no direct empirical evidence is available, those involved in program delivery in key informant interviews and focus groups noted the following organizational impacts of OF:

• improved and strengthened the relationship between HRDC and the disability community;
• enhanced the expertise and experience of the disability community with respect to employment issues;
• improved networking and information sharing among organizations and strengthened partnerships within organizations with their regional affiliates; and
• expanded the capacity of communities to assist persons with disabilities in the employment area.

4.4 Cost-Effectiveness

Program Benefits in the Areas of Income Transfers and Earnings Gains

The key measure of cost-effectiveness of OF is the extent to which participation in the program leads to an increase in earnings from employment and a reduction in use of public income supports such as social assistance and disability payments. A comparison of OF participants' sources of income prior to starting OF and at the time of the interview indicated that:

• The proportion of OF participants who relied on public income supports decreased substantially between the pre- and post-program period. Whereas 36 percent of participants named social assistance as a source of income prior to starting OF, this declined to 22 percent at the time of the interview. There were few differences in terms of the proportion receiving other public benefits such as CPP-Disability and Workers Compensation.
• Disability insurers, whether private or public, did not benefit from OF. A similar proportion of OF participants named disability insurance as a source of income both prior to and after participation in OF (six and four percent respectively).
• The proportion of OF participants who had employment earnings as a source of income increased between the pre- and post-program period. Prior to participating in OF, 17 percent reported employment earnings. At the time of the interview, this proportion increased to 49 percent. An increase in the number of individuals earning employment income has a positive benefit to government in terms of taxable income, though the amount of this benefit depends on the amount of individuals' earnings, presence of other sources of income and the tax rate.

With respect to earnings gains, the multivariate analysis of earnings data indicated that participation in OF does not have a significant impact on the percentage change in wage rate before and after the program. However, because OF participants had an advantage over the comparison group in terms of percentage of weeks working in the post-program period, this would be translated into an overall earnings advantage. Based on a 14.6 percent greater number of weeks worked found in the multivariate analysis, this translates into seven more weeks working a year. If the mean weekly earnings of OF participants of $303 is used, and with a population of 9,815¹⁹ OF participants, then on an annual basis, the total increase in earnings as a whole would be approximately $20.8 million more than the non-participants.

Costs

Unit costs for delivery of individual program interventions were not available. However, an overall cost per participant may be calculated based on the total funds dedicated to the project and the total number of participants served. In the previous section, the total population of OF participants was calculated to be between 9,800 and 12,100. This is roughly based on participant lists covering the first two years of the program. Funding over this period would equal $60 million. The cost per participant would, therefore, be between $4,950 and $6,100.

Program Leveraging

One original intent of the program design was that OF would be used in addition to other employment or income assistance programs for persons with disabilities. The survey of OF project sponsors indicated that 63 percent of organizations that deliver partnered programs received financial and/or in-kind contributions from their partners (45 percent receiving a financial contribution and 47 percent an in-kind contribution). These figures are similar to those cited in the formative evaluation. Note that in Quebec, leveraged funding is a condition of participation in OF, with most complementary funding coming from the province.

The close collaboration among project sponsors and with Human Resources Centres of Canada (HRCCs) was viewed as contributing to the effective use of OF and other funds to cover the often numerous and diverse needs of clients. Many key informants believed, however, that OF is often the sole funder, providing an additional source employment assistance for participants excluded from other programs or who require particular kinds of interventions.

The barriers to leveraging funds according to program deliverers were twofold: the absence of available programs from which to leverage funds, and differences in participant eligibility criteria across programs that limited the extent to which program funds can be used in a complementary way. To the extent that OF could increase funds leveraged from other programs, the overall cost-effectiveness of the program would be improved. OF funding, then can be used to increase intensity of interventions for existing clients or be used to extend service to more clients.

This summative evaluation of Opportunities Fund for Persons with Disabilities (OF) has focused on the implementation and results of the program since its inception in 1997. The issues examined in the evaluation included rationale for the program; program design and implementation; program impacts, and cost-effectiveness. The issues were addressed through multiple lines of evidence such as survey data, key informant interviews, focus groups, comparison group and econometric analysis, and documentation review. Following are the key findings and implications of the evaluation.

5.1 Relevance

The continuing need for a program such as OF was strongly supported by the evaluation. There is no evidence that the environment has changed significantly since the inception of the program. Persons with disabilities in Canada continue to have lower levels of income, experience higher levels of unemployment, and are more likely to be receiving public income support than the population overall. Demographic factors, namely, an aging population, suggest that many issues related to employment and disability will become more pressing in the future. OF, therefore, is both valid and necessary for equity reasons (to address systemic and attitudinal barriers) and efficiency reasons (to encourage self-sufficiency through employment).

5.2 Achievements

The evaluation of OF has provided a generally positive assessment of the program. Participants have accessed a broad and flexible variety of program interventions, reflecting the wide-ranging needs and barriers persons with disabilities face in the labour market. The use of individualized approaches to respond to varied barriers was viewed as a strength of OF and a key factor of success. Other aspects that were considered to be innovative and positive features of the program included: inclusion of the disability community; flexibility of the program; emphasis on partnership models for delivery; and focus on employment outcomes.

Overall, OF participants expressed a high degree of satisfaction with their OF intervention. Employers, project sponsors and the disability community also provided high approval ratings of various elements of the program. The program was found to be highly incremental, with the vast majority of participants, employers and project sponsors reporting that OF was very important for them in terms of their ability to participate in, or offer, interventions.

The self-assessment of OF participants, as well as the outcome data analysis, indicated that OF improved overall employability and had an incremental impact on participants' likelihood of obtaining employment and the proportion of time spent working in the post-program period. The rate of employment at the time of the survey (50 percent) exceeds the original target set for the program (40 percent). The evidence also indicated that jobs were sustainable over time. Further, the program was found to have important impacts on participants' quality of life.

No statistically significant incremental impacts were registered on wage rate or use of public income support. However, the advantage in terms of amount of time spent working translated into a positive earnings benefit for participants. Employers and organizations also confirmed having derived benefits from the program.

The impact of the program from the government's perspective derives from the increase in participants' total employment and, as a consequence, increased taxable income. Impacts in terms of a reduction in the draw on public income support could not be detected from the data collected in this evaluation.

5.3 Challenges

Program Management

The evaluation identified a number of challenges in program management and in the program policy and design. In terms of program management, implementation of the program targeting with respect to the Employment Insurance (EI)-eligibility criterion was found to be an issue, with a significant minority of OF participants falling outside the program eligibility criteria. From eight to 13 percent of participants appear to be eligible for employment benefits and support measures offered under EI Part II by virtue of their EI-claimant or reach-back status.

Program management in the area of monitoring and reporting of results was found to be weak. The availability of information on program participation, activities and results at the NHQ level is poor. The infrastructure and resources for data collection and roll-up are not currently in place, nor is there sufficient clarity in reporting expectations for third parties. Current performance measures were also criticized as being limited and lacking sensitivity to the unique employment issues and needs of persons with disabilities.

One final noted weakness in program management was a lack of consistency in the delivery of the program across Human Resources Centres of Canada (HRCCs). This was attributed to a lack of clarity in communications and initial inexperience in programming for persons with disabilities.

Program Policy and Design

While the program principles and guidelines were generally seen to have stood up well over the life of the program, some adjustments were suggested by key informants, particularly expanding the focus on "employment" by incorporating the broader notion of "employability". Employability recognizes achievements not only in terms of final employment outcomes, but also in terms of incremental steps toward attaining employment and reduction of barriers to employment.

Some of the other challenges of program policy and design have to do with gaps for particular client groups, including: those in remote communities; Aboriginal persons with disabilities; youth; those with invisible disabilities; and persons with multiple/severe disabilities. There were also identified gaps in the availability of interventions including: entrepreneurship; pre-employment skills training; disability supports; job maintenance support for those in the labour market; and employment in higher skill areas. Enhanced participation of employers and employer organizations, as well as increased attention to workplace based issues were also noted.

The above noted deficiency in program monitoring may require a solution at the level of program policy. Third party deliverers appeared unclear about the expectations for reporting, were under-resourced for this task and did not have the proper infrastructure and participant agreements in place to collect and forward the data to NHQ.

Finally, there is a need to look at a more comprehensive and cohesive labour market strategy for persons with disabilities that would encompass existing components such as OF, Employability Assistance of People with Disabilities (EAPD) and mainstream programs serving persons with disabilities. A key aspect of a labour market strategy would be ensuring that income and disability support programs do not work at cross-purposes with employment initiatives (e.g., by removal of disincentives to employment or seeking employment, and recognition of the sometimes intermittent nature of disabilities).

In sum, the OF program has many strengths in terms of program management and design which have been translated into employment and quality of life impacts for program participants. Ensuring that clients who are eligible for EI Part II programs avail themselves of EBSMs where appropriate, would extend limited program funds. There were several suggested areas where the program could elaborate upon its activities to address gaps in service or to selected client groups. Harmonization of OF with other employment and income support initiatives for persons with disabilities was identified as a key future consideration. Finally, ongoing management of the program would benefit from more rigorous and consistent monitoring and data collection. Future program evaluation activity would also be assisted by improved program administrative data.

Footnotes

1	In actuality, the survey gathered data from 1,800 individuals but 766 cases were dropped from the data set (see Section 1.3).
2	Note that participant survey results are weighted by region.
3	Comparison group survey results are weighted by region, prior EI and social assistance usage, and prior gross earnings.
4	Human Resources Development Canada, Formative Evaluation of the Employment Benefits and Support Measures, Final Report, 1998.
5	Based on Labour Force Survey data for 1999.
6	Possibly because the respondent did not recognize he/she has a disability or the disability no longer exists.
7	This may be because the respondent is reluctant to disclose the nature of the disability to the interviewer or the nature of the disability did not fit into the categories available.
8	The distribution of other types of disabilities in the overall population was as follows: mental (32 percent); hearing (25 percent); seeing (nine percent); and speaking (eight percent).
9	The 1991 Census data and the 1991 Health and Activity Limitation Survey (HALS), as well as some limited labour market data (available from the Labour Market Activity Survey and Survey of Labour and Income Dynamics), are either now dated or provide an incomplete picture of the employment needs of persons with disabilities.
10	The profile presented in this section is largely drawn from Mary Bunch and Cameron Crawford, Persons with Disabilities: Literature Review of the Factors Affecting Employment and Labour Force Transitions, Applied Research Branch, Human Resources Development Canada, 1998 and Cameron Crawford, Persons with Disabilities: Disability — Status Transitions and Labour Force — Activity Transitions, Applied Research Branch, Human Resources Development Canada, 1998.
11	The likelihood of disability increases with age (28 percent of those aged 55-64 have a disability compared to eight percent of those under 25).
12	See for example, In Unison: A Canadian Approach to Disability Issues A Vision Paper of the Federal/Provincial/Territorial Ministers Responsible for Social Services (1998). In 1999, the report Future Directions to Address Disability Issues for the Government of Canada: Working Together for Full Citizenship.
13	As a result, a custom database of program participants had to be compiled in order to assess participant outcomes.
14	Note that the measure of incrementality is solely based on the assessment of those participating in the program because a more rigorous methodology based on a reference group design was not feasible for this evaluation.
15	Note that participant results presented in this chapter are shown only for those who had completed their OF intervention at the time of the survey.
16	Note that those who were currently unemployed or not in the labour force at the time of the interview were asked about their most recent job.
17	Based on Census 1996 figures from Statistics Canada, inflated to 1999 dollars.
18	The regression coefficient (B) in models with a categorical dependent variable are transformed into a relative probability (compared to non-participants) using the following formula: Px(1-P)xBx100, where B (Beta) = the regression coefficient and P = the unweighted proportion in the bivariate results. For example, 45.6 percent of program participants (unweighted) were found to be currently employed. The regression coefficient associated with program participation was equal to 0.64. Therefore, using the formulas 0.456x(1-0.456)x0.64x100=15, it is found that OF increased participants likelihood to be currently employed by 15 percent.
19	As noted in Chapter One of this report, the survey sample was selected from a sub-population of OF participants that excluded those listed on National Employment Service System (NESS). There were 5,435 participants in this sub-population. The estimate of the true population of OF participants was based on the following steps:1) Addition of participants in third-party delivered projects where the project sponsor did not supply a list of participants. There were 297 eligible projects and lists of participants were received from 231, while 66 declined. To determine the number of potential respondents in the 66 cases, we used the average number of participants for the respondent projects ((4,118/231)*66) = 1,176). 2) Addition of 501 single-participant projects (adding these to the 1,176 yields 1,677. However, some of the cases would have shown up in the NESS database and would therefore be double counted. The overlap rate for the other projects was 17.6 percent and, using this as the deflator, the additional cases would be 1,382. Adding these to the original 5,435 yields 6,817.3) If non-EI eligible NESS cases are added, the total would be 6,817 plus 2,978 or 9,795 participants. 4) If EI eligible cases are included, the upper limit would be 12,100. The actual number of participants likely comes out somewhere in between these two limits, but in this analysis we use the more conservative lower limit.