Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

In May 1996, the Canadian government established a Federal Task Force on Disability Issues ("the Scott Task Force") to examine the role of the federal government in the area of disability. Among the issues addressed by the Task Force was the labour market integration of persons with disabilities. The Task Force recommended that the government ensure that " mainstream" labour market programs accommodate persons with disabilities. However, as many persons with disabilities have not had a strong labour market attachment, they are not eligible for employment programs available under traditional Employment Insurance (EI) and Human Resource Investment Fund (HRIF) programs. To address this gap, the Task Force identified a need to target labour market programming specifically at persons with disabilities. The Opportunities Fund for Persons with Disabilities (OFPD) was announced in February 1997 in response to this recommendation. The program is the responsibility of Human Resources Development Canada (HRDC), with the guidance of a national reference group.

Built into the Opportunities Fund (OF) is the obligation to regularly monitor the results of OF interventions. Toward that end, a three-phase evaluation plan was formulated. Ekos Research was commissioned by HRDC to conduct Phase I of the evaluation. This report describes the results of Phase I of that evaluation plan. Results from this phase will inform subsequent phases. These consist of a Mid-Term phase and a Summative phase.

II. Program Description

The objective of the OFPD is to assist persons with disabilities to prepare for, obtain and maintain employment or self-employment resulting in increased financial independence. The program is targeted to (self-identified) persons with disabilities who are legally entitled to work in Canada. Eligible individuals must be in need of assistance to prepare for or to obtain employment or self-employment and be prepared to commit to an action plan. OF clients must be ineligible for EI benefits.

There are a number of guidelines that have been established for the funding of OF projects. OF activities must help individuals with disabilities move into paid employment or self- employment, provide for the sharing of practices and experiences, and augment and complement existing programming funded through other sources. OF activities should also provide for the active participation of persons with disabilities and their organizations; lever funds from other public and private sector sources; involve employers, unions, and other partners to develop employment opportunities; provide post-placement follow-up with participants and employers; and fit into a balanced client strategy so that the full range of persons with disabilities are accessing the Opportunities Fund.

Activities can be similar in nature to employment activities funded by EI, such as targeted wage subsides, job creation and work experience partnerships, training and self-employment assistance, but expanded to accommodate persons with disabilities. OF also funds workplace accommodation costs and the provision of adaptive equipment and personal supports to facilitate access to and integration into the workplace. (These costs may also be funded in association with the above activities.)

Delivery of OF is decentralized mainly via HRDC regional offices, with a small proportion of the budget reserved for national projects. Based on their regional funding strategy, regional offices distribute all or a portion of the funds, at their discretion, to parent or local Human Resource Centres of Canada (HRCCs). HRCCs may, in turn, contract with third party organizations to deliver services to clients. There is considerable variety among communities in the way they have elected to manage OF funds. In Quebec, OF is managed by the Comit é d'adaptation de la main-d'oeuvre pour personnes handicapées (CAMO).

The total OF budget is $90 million, allocated equally over three fiscal years, 1997/1998 to 1999/2000. Ten percent of this amount is allocated to national projects. Each year, OF aims to reach 4000-6000 persons with disabilities. While OF was announced in February 1997, most program activity did not commence until October 1997 to permit time consultation, needs assessment and program design. As a result, $10 million has been reprofiled from 1997/98 to 1998/99 due to lower than expected expenditures.

III. Evaluation Issues

The main goal of Phase I of the Opportunities Fund evaluation is to assist program managers in understanding issues surrounding the design, implementation and delivery of the program in its first year, with a view to identifying areas for improvement.

Phase I of the evaluation addresses 12 evaluation questions which may be grouped under four headings: rationale, implementation, impacts, and cost-effectiveness.

Rationale. The major rationale questions addressed in this evaluation are whether there is a continued need for OF and whether the program reflects its basic underlying principles.

Implementation. Questions under implementation refer to whether the program was delivered according to regional and national implementation strategies developed on the basis of pre-program consultations and the extent to which approved projects reflect guiding principles. A final question was the extent to which proper information was being collected to permit future program evaluation activities.

Effects and impacts. At this formative stage, only early readings on impacts were feasible. The impact areas include labour-market skills, employment, earnings, the action plan, income-transfer dependence, and quality of life.

Cost-effectiveness. Under this heading, the evaluation sought to address the question of whether or not the program (projects) succeeded in leveraging funds from other program sources and the extent to which OF funds filled gaps and complemented other funding sources.

IV. Methodology

Phase I of the evaluation of OF consisted of five methodological components:

Literature and document review. Program documentation and literature on the labour market were examined as part of this review. As well, a summary of evaluative studies of lessons learned from prior experience of programs for persons with disabilities was reviewed.

Key informant interviews. In total, 22 key informant interviews were conducted to examine program rationale, implementation issues and effects and cost- effectiveness. Respondents included non-governmental organizations who are members of the national reference group; HRDC headquarters representatives; HRDC regional representatives; representatives from other similar programs aimed at persons with disabilities; non-funded applicants; and local HRCC representatives.

Case studies. To provide more detailed illustrations of program activities, five case studies were conducted; one of a national project, one of the Quebec umbrella organization, and three of regional/local projects in Victoria, Red Deer and Toronto. Each case study involved a review of documentation and three to five interviews with respondents involved in the project.

Survey of third party delivery organizations. In total, 253 interviews were conducted with organizations that provided work term placements for OF clients or that coordinated and managed services for clients. Issues covered in the survey interviews included program activity and evaluative information such as satisfaction with the program. A "faxback" form was designed to enable respondents to provide detailed feedback on such issues as gaps in programming for persons with disabilities, best practices, and suggestions for improvements to the Opportunities Fund.

Discussion groups. A total of 14 small discussion groups were conducted with participants and delivery agents/stakeholders. Groups were held in: Halifax, Rouyn/Noranda, Montreal, Ottawa, Toronto, Edmonton, Red Deer and Vancouver.

V. Evaluation Findings

The Opportunities Fund was intended to provide a response to a gap in labour market programming for individuals with disabilities that was created by reforms to EI and HRIF legislation. This gap was identified by the Scott Task Force on Disability Issues. According to the literature review and key informant interviews, there is a continued need for the OF program. There are both equity and efficiency arguments to support the integration of persons with disabilities into the labour force. The literature review noted further that as the labour market ages, the incidence of disabilities will increase, creating a greater demand for effective programs that will meet the needs of individuals with disabilities who are interested in labour market participation.

A more complicated issue is whether there is a need for the OF given the planned implementation of the Employability Assistance Program for Persons with Disabilities (EAPD). The EAPD, formerly the Vocational Rehabilitation of Disabled Persons Program, is being recast to focus on providing employment assistance for persons with disabilities. Unlike OF, however, the EAPD is a cost-shared program between the federal and provincial governments. Also, the EAPD is not expected to deliver programs in a fashion similar to OF (e.g., with the strong involvement of disability organizations). Nevertheless, some concerns were raised in this evaluation about the potential for overlap with the EAPD in the future when this program is fully implemented. As the EAPD unfolds, there will be a need in future to explore possibilities for linkages and harmonization between the two programs.

In evaluating OF, it is important to recognize that OF is not just a program, it is a process. The strong involvement of persons with disabilities and organizations representing persons with disabilities in the design and delivery of the program is an important feature. The OF reference group drawn from the Scott Task Force provides this leadership at the national level. The involvement of disability groups at the local level, while more uneven, is generally viewed as a positive and developing part of the program. The involvement of organizations for persons with disabilities creates a more sensitive and responsive environment for clients who may need more support in their ability to succeed in the labour market or to acquire the skills necessary to succeed in the labour market.

Some representatives from disability organizations suggested that there should be further involvement of this community in reviewing and approving OF projects. Other respondents noted, however, a potential conflict of interest in this position.

The flexibility and individualized approaches that are permitted under the OF program were also widely praised. The literature confirms the importance of holistic approaches to respond to the variations within the population of persons who are disabled as a key feature in the success of labour market interventions for persons with disabilities.

One issue that was noted by project sponsors was the lack of ability to serve clients who are EI eligible. While these individuals are eligible for labour market measures and benefits under EI (now largely delivered by the provinces), it should also be noted that some project sponsors viewed this exclusion as a weakness of the program and believed that in some cases EI eligible individuals were not getting the services they needed. The literature indicating that workers with disabilities respond better to programs that directly involve organizations for persons with disabilities and the fact that the earlier the interventions are implemented the more successful they will be suggest that these individuals would likely benefit from participation in OF. Alternatively, it would be prudent to examine the extent to which mainstream EI programming is fair, inclusive and accessible for persons with disabilities and, ideally, adopts the program elements that have been demonstrated to be successful in other settings.

Implementation

The evaluation results suggest that the guidelines established by the Opportunities Fund are viewed as appropriate by those involved in the program and the implementation of OF has largely been consistent with the guidelines. The focus of OF projects examined in this evaluation is clearly on improving the employability of the participants. Projects also were consistent in providing follow-up with clients and incorporating results targets. Sponsors noted, however, the need for patience in assessing outcomes and that depending on the nature of the disability, some clients may not be able to be fully self-sufficient. The project activity profile indicated further that there is a broad balance in the types of clients and severity of disability, though persons with severe disabilities were less likely to be present in wage subsidy programs and some key informants identified specific groups that were not yet well served by the program.

OF guidelines, which have proved more challenging to implement, include dissemination of findings and leveraging of funds. As the program matures and lessons learned on project implementation and success become more evident, a strategy to encourage dissemination of findings should become a greater focus for the program. The issue of leveraging is also important to increase the efficiency of use of OF funds and to enhance the opportunities available to clients.

One of the objectives and initial rationales for OF was to seek innovative strategies to integrate persons with disabilities with the labour market. Innovativeness was not incorporated into the program guidelines to ensure that "tried and true" practices were not ignored, though some key informants believed that to date there has perhaps been an over-emphasis on traditional approaches. This may be linked to pressures to be cost-effective and to maximize employment results.

The flow of information to HRCCs about the program and its dissemination to the community has been uneven. As well, the difficulty in balancing the desire for flexibility and responsiveness to local community needs and priorities with national guidelines and principles is visible in some of the qualitative data. There is a perception among some organizations that the guidelines are unclear or at least inconsistently applied. The insufficient availability of program information was highlighted as a weakness by project sponsors. According to some key informants, the result was that the projects approved were inconsistent among HRCCs.

The evidence gathered in this evaluation suggests that few changes to the design and delivery of the Opportunities Fund are required. Most project sponsors were impressed with the program design and satisfied with the length of time for approval, although some noted that the program was conceived and implemented in a short time frame. Many respondents valued the flexibility of the program and the ability of the program to provide a quick response to clients.

There were few suggestions for improvements; however, a number of project sponsors suggested that better information at the time of application would be an improvement. An ongoing, coherent communications strategy would be valuable. The planned Internet site, once developed, will provide some avenues for communication, as well as exchange among organizations. A substantial number also indicated that longer term funding would be beneficial. The three-year timeframe was viewed as being too short for programs to become well-established and effective. As mentioned earlier, some project sponsors indicated that the non-EI eligibility requirement was a weakness. As well, the literature suggests that some of the features of OF would be of benefit to all individuals with disabilities, regardless of the funding source for services and programs.

The limited amount of active promotion of the Opportunities Fund to clients and smaller community-based organizations raises concerns about the accessibility of the program. This is not necessarily an issue for individuals who have made a decision to enter the labour market, assuming that they are very likely to contact an HRCC or an organization representing persons with disabilities for job search assistance or related services. A more serious challenge may be for individuals who do not enter the labour market because they are not aware of the programs and services offered under OF.

Program Impacts and Cost Effectiveness

Leveraging of existing assistance programs may have been less than anticipated initially. Among the organizations who delivered programs and services, only one-third indicated accessing other programs and services and OF was often likely delivered in combination with social assistance rather than other labour market interventions. While the limited success in leveraging may again be the result of the relative recency of the program, there may also be a need to focus attention on building capacity among organizations to approach other government and corporate/union partners and developing appropriate communications to raise the profile of the program among potential partners.

The use of EI resources for job search/counselling was consistent with the intent to use OF funding effectively, however, only 10 percent of the organizations reported this use of EI. On the other hand, almost two-thirds of project sponsors reported providing employment counselling/job search advice to their clients themselves.

Two-thirds of the delivery organizations reported partnerships outside of their relationship with the federal government, including other non-profit organizations, the provincial government, and employers. Union involvement was relatively rare. The partnerships included activities ranging from advice to service delivery and direct funding of the project. The funding was more likely to be an in-kind contribution, but one in five organizations involved in the delivery of programs and services received a direct financial contribution from its partners.

Although a large proportion of projects were based on existing programs, virtually all sponsoring organizations indicated that the projects would not have proceeded without OF funding. Three-quarters of respondents reported that their project would not have taken place without OF funding.

It is too early to assess the impacts of OF on the skills acquisition, labour market and quality of life outcomes of the participants at this time; the majority of the current participants have not yet completed their program. Consequently, lessons learned about the effectiveness of the different types of interventions also cannot be assessed at this stage of the evaluation. One very promising finding was that the majority of the organizations that provided a work placement were very satisfied with the OF participant and nearly half intended to hire the participant in the future.

Future Monitoring and Evaluation Issues

Unfortunately, information on what works and lessons learned may be delayed or may not be available on an ongoing basis. Although most organizations intend to monitor their clients, a minority (14 percent) do not develop action plans and do not intend to follow-up with their clients (11 percent). Only about one in five organizations are using the Contact IV software to collect client information at this time. Indeed, only half of the project sponsors who were interviewed as part of this evaluation had heard of Contact IV. There were also reports of difficulties or delays in transferring local level data to the national system. Without a formal system to comprehensively track results, information available on client outcomes will not be consistent or comprehensive, limiting the usefulness of these data to assess what works and what doesn't. It should also be noted that at present the fields that are uploaded from Contact IV to NESS do not include participants' telephone number which presents a barrier to future contacts.

Evaluation research may not be able to fill this void. The decentralized delivery of the program coupled with the absence of complete national level data presents challenges in terms of directly contacting OF participants to assess the impacts of the program. The extent to which OF agreements with the sponsoring organizations include a provision for the collection of client contact information to be released for research purposes is not clear. During the course of this evaluation many project sponsors and HRCCs were unwilling to release participant information citing confidentiality concerns. The lack of participant-level contact information would severely limit the ability of future evaluation studies to assess the effectiveness of the interventions and the lessons learned.

The success of future evaluation activities will be enhanced by a strong communications strategy to ensure that HRCCs and project sponsors have a clear understanding of evaluation objectives, timing of activities and their responsibilities with respect to confidentiality/ release of information. As well, given some of the potential sensitivities around contacting persons with disabilities (e.g., use of self-identified equity information, intellectual capacity of some participants), up-front communications with participants will also be important to ensuring a high comfort level with the evaluation.

There are a number of options or strategies to ensure that the data requirements for the summative evaluation are met. In essence, the data requirements revolve around the need for accurate and comprehensive OF participant contact data. Options for compiling this information include:

1. Use of NESS data. Use of a single data source would simplify the evaluation exercise. However, use of the NESS assumes that the use of Contact IV be much more widespread among project sponsors than is currently the case and that these data are regularly uploaded to the NESS. The absence of participant telephone numbers is also problematic. Linkages with other administrative databases (e.g., social assistance, status vector or T1) may provide contact data for some cases. The remainder would have to be tracked manually through directory assistance (an expensive exercise that also opens the possibility for bias in underrepresenting participants who have relocated).
2. Obtaining Contact IV data from the regional/local level. While avoiding the problems associated with the NESS upload, this method also assumes that project sponsors are using Contact IV in a regular and consistent fashion and that these data are available at the HRCC level. This method also places the burden of data collection at the HRCC level, which may be difficult given current resource constraints.
3. Obtaining participant contact information directly from project sponsors. This final option involves contacts with each project sponsor and requesting that participant information be forwarded to an HRDC OF evaluation representative or outside consultant. This option would provide the most comprehensive participant information (including organizations that use Contact IV as well as those that do not). This process, however, is very time consuming. As well, based on the experience of this evaluation, proper communications and release of information guidelines must be in place to encourage project sponsors to forward this information.

While Phase I of the evaluation has provided useful descriptive information on program implementation and process issues, the mid-term and summative evaluations will be better positioned to collect information on program results. A survey of OF participants represents a crucial component of this exercise. From these data, the final evaluation results would contribute to an understanding of the profile of OF participants, the impacts and effects of the program, and the types of interventions that have proven most successful for persons with disabilities.

Last Modified: 1999-02-23	Important Notices