Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Evaluation of the Opportunities Fund for Persons with Disabilities (Phase I)

Final Report

Evaluation and Data Development
Strategic Policy
Human Resources Development Canada

November 1998

In May 1996, the Canadian government established a Federal Task Force on Disability Issues ("the Scott Task Force") to examine the role of the federal government in the area of disability. Among the issues addressed by the Task Force was the labour market integration of persons with disabilities. The Task Force recommended that the government ensure that " mainstream" labour market programs accommodate persons with disabilities. However, as many persons with disabilities have not had a strong labour market attachment, they are not eligible for employment programs available under traditional Employment Insurance (EI) and Human Resource Investment Fund (HRIF) programs. To address this gap, the Task Force identified a need to target labour market programming specifically at persons with disabilities. The Opportunities Fund for Persons with Disabilities (OFPD) was announced in February 1997 in response to this recommendation. The program is the responsibility of Human Resources Development Canada (HRDC), with the guidance of a national reference group.

Built into the Opportunities Fund (OF) is the obligation to regularly monitor the results of OF interventions. Toward that end, a three-phase evaluation plan was formulated. Ekos Research was commissioned by HRDC to conduct Phase I of the evaluation. This report describes the results of Phase I of that evaluation plan. Results from this phase will inform subsequent phases. These consist of a Mid-Term phase and a Summative phase.

II. Program Description

The objective of the OFPD is to assist persons with disabilities to prepare for, obtain and maintain employment or self-employment resulting in increased financial independence. The program is targeted to (self-identified) persons with disabilities who are legally entitled to work in Canada. Eligible individuals must be in need of assistance to prepare for or to obtain employment or self-employment and be prepared to commit to an action plan. OF clients must be ineligible for EI benefits.

There are a number of guidelines that have been established for the funding of OF projects. OF activities must help individuals with disabilities move into paid employment or self- employment, provide for the sharing of practices and experiences, and augment and complement existing programming funded through other sources. OF activities should also provide for the active participation of persons with disabilities and their organizations; lever funds from other public and private sector sources; involve employers, unions, and other partners to develop employment opportunities; provide post-placement follow-up with participants and employers; and fit into a balanced client strategy so that the full range of persons with disabilities are accessing the Opportunities Fund.

Activities can be similar in nature to employment activities funded by EI, such as targeted wage subsides, job creation and work experience partnerships, training and self-employment assistance, but expanded to accommodate persons with disabilities. OF also funds workplace accommodation costs and the provision of adaptive equipment and personal supports to facilitate access to and integration into the workplace. (These costs may also be funded in association with the above activities.)

Delivery of OF is decentralized mainly via HRDC regional offices, with a small proportion of the budget reserved for national projects. Based on their regional funding strategy, regional offices distribute all or a portion of the funds, at their discretion, to parent or local Human Resource Centres of Canada (HRCCs). HRCCs may, in turn, contract with third party organizations to deliver services to clients. There is considerable variety among communities in the way they have elected to manage OF funds. In Quebec, OF is managed by the Comit é d'adaptation de la main-d'oeuvre pour personnes handicapées (CAMO).

The total OF budget is $90 million, allocated equally over three fiscal years, 1997/1998 to 1999/2000. Ten percent of this amount is allocated to national projects. Each year, OF aims to reach 4000-6000 persons with disabilities. While OF was announced in February 1997, most program activity did not commence until October 1997 to permit time consultation, needs assessment and program design. As a result, $10 million has been reprofiled from 1997/98 to 1998/99 due to lower than expected expenditures.

III. Evaluation Issues

The main goal of Phase I of the Opportunities Fund evaluation is to assist program managers in understanding issues surrounding the design, implementation and delivery of the program in its first year, with a view to identifying areas for improvement.

Phase I of the evaluation addresses 12 evaluation questions which may be grouped under four headings: rationale, implementation, impacts, and cost-effectiveness.

Rationale. The major rationale questions addressed in this evaluation are whether there is a continued need for OF and whether the program reflects its basic underlying principles.

Implementation. Questions under implementation refer to whether the program was delivered according to regional and national implementation strategies developed on the basis of pre-program consultations and the extent to which approved projects reflect guiding principles. A final question was the extent to which proper information was being collected to permit future program evaluation activities.

Effects and impacts. At this formative stage, only early readings on impacts were feasible. The impact areas include labour-market skills, employment, earnings, the action plan, income-transfer dependence, and quality of life.

Cost-effectiveness. Under this heading, the evaluation sought to address the question of whether or not the program (projects) succeeded in leveraging funds from other program sources and the extent to which OF funds filled gaps and complemented other funding sources.

IV. Methodology

Phase I of the evaluation of OF consisted of five methodological components:

Literature and document review. Program documentation and literature on the labour market were examined as part of this review. As well, a summary of evaluative studies of lessons learned from prior experience of programs for persons with disabilities was reviewed.

Key informant interviews. In total, 22 key informant interviews were conducted to examine program rationale, implementation issues and effects and cost- effectiveness. Respondents included non-governmental organizations who are members of the national reference group; HRDC headquarters representatives; HRDC regional representatives; representatives from other similar programs aimed at persons with disabilities; non-funded applicants; and local HRCC representatives.

Case studies. To provide more detailed illustrations of program activities, five case studies were conducted; one of a national project, one of the Quebec umbrella organization, and three of regional/local projects in Victoria, Red Deer and Toronto. Each case study involved a review of documentation and three to five interviews with respondents involved in the project.

Survey of third party delivery organizations. In total, 253 interviews were conducted with organizations that provided work term placements for OF clients or that coordinated and managed services for clients. Issues covered in the survey interviews included program activity and evaluative information such as satisfaction with the program. A "faxback" form was designed to enable respondents to provide detailed feedback on such issues as gaps in programming for persons with disabilities, best practices, and suggestions for improvements to the Opportunities Fund.

Discussion groups. A total of 14 small discussion groups were conducted with participants and delivery agents/stakeholders. Groups were held in: Halifax, Rouyn/Noranda, Montreal, Ottawa, Toronto, Edmonton, Red Deer and Vancouver.

V. Evaluation Findings

The Opportunities Fund was intended to provide a response to a gap in labour market programming for individuals with disabilities that was created by reforms to EI and HRIF legislation. This gap was identified by the Scott Task Force on Disability Issues. According to the literature review and key informant interviews, there is a continued need for the OF program. There are both equity and efficiency arguments to support the integration of persons with disabilities into the labour force. The literature review noted further that as the labour market ages, the incidence of disabilities will increase, creating a greater demand for effective programs that will meet the needs of individuals with disabilities who are interested in labour market participation.

A more complicated issue is whether there is a need for the OF given the planned implementation of the Employability Assistance Program for Persons with Disabilities (EAPD). The EAPD, formerly the Vocational Rehabilitation of Disabled Persons Program, is being recast to focus on providing employment assistance for persons with disabilities. Unlike OF, however, the EAPD is a cost-shared program between the federal and provincial governments. Also, the EAPD is not expected to deliver programs in a fashion similar to OF (e.g., with the strong involvement of disability organizations). Nevertheless, some concerns were raised in this evaluation about the potential for overlap with the EAPD in the future when this program is fully implemented. As the EAPD unfolds, there will be a need in future to explore possibilities for linkages and harmonization between the two programs.

In evaluating OF, it is important to recognize that OF is not just a program, it is a process. The strong involvement of persons with disabilities and organizations representing persons with disabilities in the design and delivery of the program is an important feature. The OF reference group drawn from the Scott Task Force provides this leadership at the national level. The involvement of disability groups at the local level, while more uneven, is generally viewed as a positive and developing part of the program. The involvement of organizations for persons with disabilities creates a more sensitive and responsive environment for clients who may need more support in their ability to succeed in the labour market or to acquire the skills necessary to succeed in the labour market.

Some representatives from disability organizations suggested that there should be further involvement of this community in reviewing and approving OF projects. Other respondents noted, however, a potential conflict of interest in this position.

The flexibility and individualized approaches that are permitted under the OF program were also widely praised. The literature confirms the importance of holistic approaches to respond to the variations within the population of persons who are disabled as a key feature in the success of labour market interventions for persons with disabilities.

One issue that was noted by project sponsors was the lack of ability to serve clients who are EI eligible. While these individuals are eligible for labour market measures and benefits under EI (now largely delivered by the provinces), it should also be noted that some project sponsors viewed this exclusion as a weakness of the program and believed that in some cases EI eligible individuals were not getting the services they needed. The literature indicating that workers with disabilities respond better to programs that directly involve organizations for persons with disabilities and the fact that the earlier the interventions are implemented the more successful they will be suggest that these individuals would likely benefit from participation in OF. Alternatively, it would be prudent to examine the extent to which mainstream EI programming is fair, inclusive and accessible for persons with disabilities and, ideally, adopts the program elements that have been demonstrated to be successful in other settings.

Implementation

The evaluation results suggest that the guidelines established by the Opportunities Fund are viewed as appropriate by those involved in the program and the implementation of OF has largely been consistent with the guidelines. The focus of OF projects examined in this evaluation is clearly on improving the employability of the participants. Projects also were consistent in providing follow-up with clients and incorporating results targets. Sponsors noted, however, the need for patience in assessing outcomes and that depending on the nature of the disability, some clients may not be able to be fully self-sufficient. The project activity profile indicated further that there is a broad balance in the types of clients and severity of disability, though persons with severe disabilities were less likely to be present in wage subsidy programs and some key informants identified specific groups that were not yet well served by the program.

OF guidelines, which have proved more challenging to implement, include dissemination of findings and leveraging of funds. As the program matures and lessons learned on project implementation and success become more evident, a strategy to encourage dissemination of findings should become a greater focus for the program. The issue of leveraging is also important to increase the efficiency of use of OF funds and to enhance the opportunities available to clients.

One of the objectives and initial rationales for OF was to seek innovative strategies to integrate persons with disabilities with the labour market. Innovativeness was not incorporated into the program guidelines to ensure that "tried and true" practices were not ignored, though some key informants believed that to date there has perhaps been an over-emphasis on traditional approaches. This may be linked to pressures to be cost-effective and to maximize employment results.

The flow of information to HRCCs about the program and its dissemination to the community has been uneven. As well, the difficulty in balancing the desire for flexibility and responsiveness to local community needs and priorities with national guidelines and principles is visible in some of the qualitative data. There is a perception among some organizations that the guidelines are unclear or at least inconsistently applied. The insufficient availability of program information was highlighted as a weakness by project sponsors. According to some key informants, the result was that the projects approved were inconsistent among HRCCs.

The evidence gathered in this evaluation suggests that few changes to the design and delivery of the Opportunities Fund are required. Most project sponsors were impressed with the program design and satisfied with the length of time for approval, although some noted that the program was conceived and implemented in a short time frame. Many respondents valued the flexibility of the program and the ability of the program to provide a quick response to clients.

There were few suggestions for improvements; however, a number of project sponsors suggested that better information at the time of application would be an improvement. An ongoing, coherent communications strategy would be valuable. The planned Internet site, once developed, will provide some avenues for communication, as well as exchange among organizations. A substantial number also indicated that longer term funding would be beneficial. The three-year timeframe was viewed as being too short for programs to become well-established and effective. As mentioned earlier, some project sponsors indicated that the non-EI eligibility requirement was a weakness. As well, the literature suggests that some of the features of OF would be of benefit to all individuals with disabilities, regardless of the funding source for services and programs.

The limited amount of active promotion of the Opportunities Fund to clients and smaller community-based organizations raises concerns about the accessibility of the program. This is not necessarily an issue for individuals who have made a decision to enter the labour market, assuming that they are very likely to contact an HRCC or an organization representing persons with disabilities for job search assistance or related services. A more serious challenge may be for individuals who do not enter the labour market because they are not aware of the programs and services offered under OF.

Program Impacts and Cost Effectiveness

Leveraging of existing assistance programs may have been less than anticipated initially. Among the organizations who delivered programs and services, only one-third indicated accessing other programs and services and OF was often likely delivered in combination with social assistance rather than other labour market interventions. While the limited success in leveraging may again be the result of the relative recency of the program, there may also be a need to focus attention on building capacity among organizations to approach other government and corporate/union partners and developing appropriate communications to raise the profile of the program among potential partners.

The use of EI resources for job search/counselling was consistent with the intent to use OF funding effectively, however, only 10 percent of the organizations reported this use of EI. On the other hand, almost two-thirds of project sponsors reported providing employment counselling/job search advice to their clients themselves.

Two-thirds of the delivery organizations reported partnerships outside of their relationship with the federal government, including other non-profit organizations, the provincial government, and employers. Union involvement was relatively rare. The partnerships included activities ranging from advice to service delivery and direct funding of the project. The funding was more likely to be an in-kind contribution, but one in five organizations involved in the delivery of programs and services received a direct financial contribution from its partners.

Although a large proportion of projects were based on existing programs, virtually all sponsoring organizations indicated that the projects would not have proceeded without OF funding. Three-quarters of respondents reported that their project would not have taken place without OF funding.

It is too early to assess the impacts of OF on the skills acquisition, labour market and quality of life outcomes of the participants at this time; the majority of the current participants have not yet completed their program. Consequently, lessons learned about the effectiveness of the different types of interventions also cannot be assessed at this stage of the evaluation. One very promising finding was that the majority of the organizations that provided a work placement were very satisfied with the OF participant and nearly half intended to hire the participant in the future.

Future Monitoring and Evaluation Issues

Unfortunately, information on what works and lessons learned may be delayed or may not be available on an ongoing basis. Although most organizations intend to monitor their clients, a minority (14 percent) do not develop action plans and do not intend to follow-up with their clients (11 percent). Only about one in five organizations are using the Contact IV software to collect client information at this time. Indeed, only half of the project sponsors who were interviewed as part of this evaluation had heard of Contact IV. There were also reports of difficulties or delays in transferring local level data to the national system. Without a formal system to comprehensively track results, information available on client outcomes will not be consistent or comprehensive, limiting the usefulness of these data to assess what works and what doesn't. It should also be noted that at present the fields that are uploaded from Contact IV to NESS do not include participants' telephone number which presents a barrier to future contacts.

Evaluation research may not be able to fill this void. The decentralized delivery of the program coupled with the absence of complete national level data presents challenges in terms of directly contacting OF participants to assess the impacts of the program. The extent to which OF agreements with the sponsoring organizations include a provision for the collection of client contact information to be released for research purposes is not clear. During the course of this evaluation many project sponsors and HRCCs were unwilling to release participant information citing confidentiality concerns. The lack of participant-level contact information would severely limit the ability of future evaluation studies to assess the effectiveness of the interventions and the lessons learned.

The success of future evaluation activities will be enhanced by a strong communications strategy to ensure that HRCCs and project sponsors have a clear understanding of evaluation objectives, timing of activities and their responsibilities with respect to confidentiality/ release of information. As well, given some of the potential sensitivities around contacting persons with disabilities (e.g., use of self-identified equity information, intellectual capacity of some participants), up-front communications with participants will also be important to ensuring a high comfort level with the evaluation.

There are a number of options or strategies to ensure that the data requirements for the summative evaluation are met. In essence, the data requirements revolve around the need for accurate and comprehensive OF participant contact data. Options for compiling this information include:

1. Use of NESS data. Use of a single data source would simplify the evaluation exercise. However, use of the NESS assumes that the use of Contact IV be much more widespread among project sponsors than is currently the case and that these data are regularly uploaded to the NESS. The absence of participant telephone numbers is also problematic. Linkages with other administrative databases (e.g., social assistance, status vector or T1) may provide contact data for some cases. The remainder would have to be tracked manually through directory assistance (an expensive exercise that also opens the possibility for bias in underrepresenting participants who have relocated).
2. Obtaining Contact IV data from the regional/local level. While avoiding the problems associated with the NESS upload, this method also assumes that project sponsors are using Contact IV in a regular and consistent fashion and that these data are available at the HRCC level. This method also places the burden of data collection at the HRCC level, which may be difficult given current resource constraints.
3. Obtaining participant contact information directly from project sponsors. This final option involves contacts with each project sponsor and requesting that participant information be forwarded to an HRDC OF evaluation representative or outside consultant. This option would provide the most comprehensive participant information (including organizations that use Contact IV as well as those that do not). This process, however, is very time consuming. As well, based on the experience of this evaluation, proper communications and release of information guidelines must be in place to encourage project sponsors to forward this information.

While Phase I of the evaluation has provided useful descriptive information on program implementation and process issues, the mid-term and summative evaluations will be better positioned to collect information on program results. A survey of OF participants represents a crucial component of this exercise. From these data, the final evaluation results would contribute to an understanding of the profile of OF participants, the impacts and effects of the program, and the types of interventions that have proven most successful for persons with disabilities.

The Labour Market Directorate of Human Resources Development Canada (HRDC) has reviewed the Evaluation of the Opportunities Fund for Persons with Disabilities (Phase I). Overall, management is pleased with the positive findings contained in the report. As pointed out in the report, the majority (87%) of respondents was satisfied with the program, especially in the areas of amount of funding to the organizations, wage subsidies to the companies, HRDC's application process, reporting requirements and general support.

It is important to recognize that the Opportunities Fund is not just a program but a process. As a result of the Scott Task Force, there is a strong involvement of persons with disabilities and organizations representing persons with disabilities in providing leadership in the implementation of this program at the national level. This offers an innovative way of bringing sensitivity to a program established to meet the employment needs of persons with disabilities by involving them in the design and implementation. This is a key feature in the success of labour market interventions for persons with disabilities.

Few changes in design and delivery are recommended. Steps are already being undertaken to address the few weaknesses identified, especially in the area of program monitoring and data collection as well as marketing and sharing of innovative initiatives.

As stated, it is too early to assess the impacts of Opportunities Fund on the skills acquisition, labour market outcomes and quality of life of the participants at this time. Labour Market Directorate fully endorses the evaluation's conclusions and looks forward to Phase II of the evaluation process.

1.1 Background

In May 1996, the Canadian government established a Federal Task Force on Disability Issues ("the Scott Task Force") to examine the role of the federal government in the area of disability. The Task Force, which included broad representation from the disability community, undertook a series of consultations and hearings across Canada, as well as research studies. The resulting report was entitled Equal Citizenship for Canadians with Disabilities: The Will to Act.

Among the issues addressed by the Task Force was the labour market integration of persons with disabilities. The Task Force found that work was among the top concerns of Canadians with disabilities, contributing to a sense of belonging, accomplishment and dignity. Moreover, with the right legislative mechanisms in place, it was found that many persons with disabilities could be integrated into the labour market and make an economic contribution to society. The task force further noted that more important than the disability itself were environmental and economic barriers to employment faced by those with disabilities.

The Task Force recommended that the government ensure that "mainstream" labour market programs accommodate persons with disabilities. However, since many persons with disabilities have not had a strong labour market attachment, they are not eligible for employment programs available under traditional Employment Insurance (EI) programs. To address this gap, the Task Force identified a need to target labour market programming specifically for persons with disabilities.

The Opportunities Fund for Persons with Disabilities (OFPD) was established in February 1997 in response to this recommendation. The program is the responsibility of Human Resources Development Canada (HRDC) with the guidance of a national reference group. Among the major rationales for the program's implementation were the rising importance being placed on skills in the workplace and the large number of persons with disabilities who could benefit from assistance to (re-)enter the labour market, were this assistance available to them. Many Canadians with disabilities who are on social assistance would rather work, but do not normally qualify for public assistance integrating persons with disabilities into the workforce. The Opportunities Fund (OF) fills this gap.

Built into the OF is the obligation to regularly monitor the results of OF interventions. Toward that end, a three-phase evaluation plan was formulated. Ekos Research Associates Inc. was commissioned by HRDC to conduct Phase I of the evaluation of OF. This report describes the results of Phase I of the evaluation plan. Results from this phase will inform subsequent phases. These are the Mid-Term and the Summative phases.

1.2 Program Description

The objective of the Opportunities Fund Persons with Disabilities is to assist persons with disabilities prepare for, obtain and maintain employment or self-employment, resulting in increased financial independence.

Potential OF clients must meet certain criteria to qualify for assistance under the program. First, they must self-identify as disabled and wanting to work. Second, potential OF clients must be legally entitled to work in Canada. Third, they must be in need of assistance to prepare for or obtain employment or self-employment, though they do not necessarily have to be job-ready. Fourth, OF clients must be ineligible for Employment Insurance (EI) benefits. Finally, clients must commit to an action plan designed to assist them into employment or self-employment.

There are a number of guidelines that have been established for the funding of OF projects. OF activities must help individuals with disabilities move into paid employment or self- employment, provide for the sharing of practices and experiences, and augment and complement existing programming funded through other sources. OF activities should also provide for:

• the active participation of persons with disabilities and their organizations;
• leveraging of funds from other public and private sector sources;
• involvement from employers, unions, and other partners to develop employment opportunities;
• post-placement follow-up with participants and employers; and
• a balanced client strategy so that the full range of persons with disabilities are accessing the Opportunities Fund.

Another important OF guideline is that projects be accountable for results with respect to employment and cost-savings targets. Regarding employment, OF projects must state the number of clients who are expected to find employment or self-employment, with due consideration for the change in employment status, e.g., part-time employment is a success if the client had not been employed before the intervention. Each year, OF aims to reach 4000-6000 persons with disabilities, of which 40 percent are expected to be employed following the completion of their intervention. Regarding income support cost savings, there must be stated targets regarding expected cost savings to income transfer programs and private insurers.

What types of activities does OF fund? Activities can be similar in nature to employment activities funded by EI, such as targeted wage subsidies, job creation and work experience partnerships, training and self-employment assistance, but expanded to accommodate clients with disabilities. OF also funds workplace accommodation costs and the provision of adaptive equipment and personal supports to facilitate access to and integration into the workplace. (These costs may also be funded in association with the above activities.) In addition, OF funds implementation of action plans including prescribed EI Employment Assistance Services (EAS). However, OF does not fund the cost of the EAS services themselves, which are covered by the EI Act as services available to all (not just to EI clients).

What costs are covered under OF? For individual OF clients, living expenses, dependent care and the cost of attendant care may be covered. Also covered are costs associated with transportation, the cost of accommodation for training, the cost of training, as well as the costs of purchasing assistive equipment (e.g., hearing aid, large monitor) enabling persons with disabilities access to work. Income allowances may also be provided for in OF projects, as well as the cost of self-employment assistance, including training and advice. For employers, OF will cover the costs associated with modifying the workplace to accommodate participants and provide wage subsidies to employers as an incentive to hire participants. As for third party coordinators, OF covers the costs associated with implementing an action plan for participants through Employment Assistance Services under the EI Act and overhead costs (e.g., consultant and operational costs).

Delivery of OF is decentralized mainly via HRDC regional offices, with a small proportion of the budget reserved for national projects. Regional offices developed their own implementation strategy based on a national template which includes descriptions of the regional environment, the key partners, local and regional consultations that have been undertaken, the clients, priorities for activities, results targets, the service delivery mechanism, and the communication strategy. Based on their regional funding strategy, regional offices distribute all or a portion of the funds, at their discretion, to parent or local Human Resource Centres of Canada (HRCCs). HRCCs may, in turn, contract with third party organizations to deliver services to clients.

There is considerable variety among communities in the way they have elected to manage OF funds. Some communities, for example, have identified one organization locally to take the lead in delivering OF based on consultations and referrals from other organizations. Others have established or used pre-established partnership committees to guide the usage of OF funds. A third model is the distribution of OF funds through a mixture of individual and multi-service organizational contracts.

In Quebec, OF is managed by the Comité d'adaptation de la main-d'oeuvre pour personnes handicapées (CAMO). Founded in 1993, the CAMO is managed by many representatives including people from the three levels of government, NGOs, the business community and trade unions. CAMO basically determines the orientation of OF in Quebec and approves projects submitted by individuals and organizations in that province. This process is done in partnership with many stakeholders, at the provincial level and local levels, in cooperation with partnership committees from each of the Quebec administrative regions. With OF, the CAMO has hired project officers who establish links between the organization and the administrative regions. These officers play a key role in project selection and follow-up and are responsible for the linkages between the CAMO and local partnership committees.

National project activities include those under Employment Assistance Services and Labour Market Partnerships which cannot effectively be funded through one region and meet OF guiding principles. Priorities for national projects include having consumer control, addressing the needs of the "doubly disadvantaged", and harmonization with regional implementation strategies.

The total OF budget is $90 million, allocated equally over three fiscal years, 1997/1998 to 1999/2000. While OF was announced in February 1997, most program activity did not commence until October 1997 to permit time for consultation, needs assessment and program development and design. As a result, $10 million has been reprofiled from 1997/98 to 1998/99 due to lower than expected expenditures. Funds come out of the Consolidated Revenue Fund (CRF), which means earnings received by OF clients are insurable earnings. Of the annual budget of $30 million, 10 percent each year has been allotted for national projects. In addition, a matching amount ($3 million) each year comes out of EI Part II for pan-Canadian activities that are national or multi-regional in scope and support the integration of all individuals (regardless of EI status) into employment.

1.3 Evaluation Objectives and Issues

The main goal of Phase I of the Opportunities Fund evaluation is to assist in understanding issues surrounding the design, implementation and delivery of the program in its first year, with a view to identifying areas for improvement. The evaluation describes which activities are being delivered to generate intended outputs, to whom and how the interventions are being delivered, and in partnership with whom.

Phase I of the evaluation addresses 12 evaluation questions. The 12 evaluation issues are listed in Appendix A. They are briefly described here under four headings: rationale, implementation, impacts, and cost-effectiveness.

The major rationale questions addressed in this evaluation are whether there is a continued need for OF and whether the program reflects its basic principles' underpinning. To address these questions, we relied on program-related documentation and the views of key informant interview respondents. The logic of the program design was also examined given the lessons learned from other programs aimed at persons with disabilities.

The second set of questions is concerned with the implementation of the program. In answering these questions, we sought to determine whether the program was delivered according to regional and national implementation strategies developed on the basis of pre-program consultations. We also examined the extent to which approved projects reflect guiding principles. A final question was the extent to which proper information was being collected to permit future program evaluation activities. Several lines of evidence were used to address these issues, including discussion groups, key informant interviews, case studies, surveys and program documents.

For the third group of evaluation questions, effects and impacts, in this, the formative stage, we were able to obtain only early readings on impacts, as the program is still in its infancy. The impact areas include labour market skills, employment, earnings, the action plan, income-transfer dependence, and quality of life.

For the final evaluation issue, cost-effectiveness, this evaluation sought to address the question of whether or not the program (projects) succeeded in leveraging funds from other program sources. We also examined the extent to which OF funds filled gaps and complemented other funding sources, which is one of the OF guidelines.

1.4 Methodology

Phase I of the evaluation of OF consisted of five components: a literature and document review, key informant interviews, case studies, a survey of third party delivery organizations, and discussion groups. Each of these is briefly described in turn.

The review of literature and documentation served two purposes. First, the review set the context for the study by describing the basic economic and labour market environment in which the program was born in order to establish a rationale for the program. The literature review included government data analyses (particularly the Health and Activity Limitations Survey) and other independent studies. The second purpose of the review was to enable a thorough understanding of the program under study. For this, we reviewed program documentation including policy manuals, reports, statements and operational guidelines.

For the key informant interviews, a total of 22 interviews were conducted with different respondent types in order to get wide ranging views on the issues from all perspectives. The respondent types included non-governmental organizations who are members of the national reference group; HRDC headquarters representatives; HRDC regional representatives; representatives from other similar programs aimed at persons with disabilities; non-funded OF applicants; and local HRCC representatives. Among the issues addressed in the interviews were the rationale for the program; the implementation of the program and consistency with guidelines; effectiveness of partnerships and leveraging; and impacts of the program on clients and organizations.

Five case studies were conducted, one of a national project, one of the Quebec umbrella organization, and three of regional/local projects in Victoria, Red Deer and Toronto. Each case study was based on two information sources: (1) administrative data on finances, clients and employers, and impacts/results of the project, and (2) three to five one-to-two-hour interviews with project respondents, including the manager and clients of the project, the local HRCC official, and external stakeholders and partners. Among the issues addressed by the case studies were leveraging and partnerships; the effectiveness of program delivery and partnerships; short-term impacts on clients and organizations; the quality of follow-up on projects and clients; and lessons learned. Case study illustrations are presented throughout the text. Highlights of the case studies are presented in Appendix B.

For the telephone survey of third-party delivery organizations, 253 interviews (out of 520 organizations) were conducted with organizations that provided work term placements for OF clients or that coordinated and managed services for clients. Respondents were drawn from the OF projects database. Issues covered in the survey interviews included program activity (e.g., number of participants and partnerships created) and evaluative information such as satisfaction with the program, whether or not needs were met, and short-term employment and quality-of-life outcomes. A "faxback" form was designed to enable respondents to provide detailed feedback on such issues as gaps in programming for persons with disabilities, best practices, and suggestions for improvements to the Opportunities Fund.

For the discussion groups, 14 small discussion groups were conducted with participants and delivery agents/stakeholders to provide in-depth information on all evaluation topics. Groups were led, in either English or French, by Ekos moderators in: Halifax, Rouyn/Noranda, Montreal, Ottawa, Toronto, Edmonton, Red Deer and Vancouver. Sessions lasted approximately one to two hours. The groups ranged in size considerably, with an average of about six to eight per group.

While Phase I of the evaluation of the OF Fund has attempted to collect information using multiple lines of evidence, a number of caveats should be noted. First, the program has only recently entered the implementation stage. Issues related to impacts and effects and lessons learned could not be fully addressed at this stage. Second, the views of OF participants are only represented in a limited way through a small number of discussion groups. Participants' views on delivery and, in particular, impacts and effects are therefore very preliminary at this time.

1.5 Organization of Report

There are four additional chapters following this one. Chapter 2.0 addresses questions related to program rationale and program implementation, particularly with respect to the established program guidelines. Chapter 3.0 examines initial program delivery results, including, for example, take-up, project activities and satisfaction. Chapter 4.0 presents findings related to program impacts and effects and cost-effectiveness. It should be noted that, as this is Phase I of the evaluation and the program is in the very early stages of delivery, information related to these issues is preliminary. Chapter 5.0 presents a summary and formative evaluation conclusions.

With respect to program rationale issues, the focus of this evaluation has been on the extent to which the principles and approach of the Opportunities Fund reflect the original intention of the program and established best practices in this area. In addition, this chapter also examines the extent to which there is a continued need for the program based on the needs and profiles of persons with disabilities in Canada and current programming. The existing literature offers a strong rationale for programs to assist those with disabilities to enter the labour market and also supports many of the program features of the Opportunities Fund. This chapter also addresses implementation issues, in particular, adherence to the program principles in practices, targeting, outreach, leveraging, partnerships and monitoring and accountability.

2.1 Program Relevance and Rationale

(a) Program Relevance

As mentioned in the previous chapter, the catalyst for the Opportunities Fund was the Scott Federal Task Force on Disability Issues. During its consultations, employment was identified as a top concern and the Task Force noted the many barriers faced by persons with disabilities in attaining employment and independence.

The employment challenges experienced by Canadians with disabilities were also evident in the literature reviewed for this evaluation. For example, the labour force participation rate of persons with disabilities (56 percent) was considerably lower in 1991 than the respective rate for persons without disabilities (81 percent)¹. Conversely, the unemployment rate is higher among persons with disabilities, regardless of sex, age, education or occupation. Based on data from the 1991 HALS (Health and Activity Limitation Survey), Fawcett (1996) estimates that over half of persons with disabilities could have entered paid employment if barriers were removed.

Evidence of the consequences of marginalization of persons with disabilities in the labour market is the relatively high incidence of poverty. This is especially the case among those with severe disabilities. The literature suggests further that issues around employment for persons with disabilities will become even more important in the future (Fawcett, 1996). The incidence of disability rises with age, implying a rising incidence of disability in the future as the large baby boomer generation grows older, suggesting the continued need for programs to assist persons with disabilities to integrate into or remain in the labour force.

While the Scott Task Force asserted that all labour market programs should be inclusive for those with disabilities, it also identified a need for additional measures to meet the needs of persons with disabilities. The Task Force report noted that the reforms to Employment Insurance (EI) and the Human Resources Investment Fund (HRIF) resulted in a loss of eligibility among persons with disabilities for active employment measures. Because persons with disabilities often do not have the labour market experience to gain the insurable earnings necessary for access to EI-related measures, the Task Force recommended the creation of a fund to "provide innovative approaches to integrating individuals with disabilities into the labour force".

The Opportunities Fund was established to respond to the recommendations of the Scott Task Force around labour market integration. According to key informants and the literature, the OF is based on a rationale that includes both equity and efficiency arguments. On the one hand, the Fund attempts to address some of the attitudinal and physical barriers that have disadvantaged individuals with disabilities in the past from actively participating in the world of work. For equity reasons, individuals with disabilities should have fair access to programs and opportunities in the labour market and should be able to enjoy the sense of belonging, dignity and independence that is associated with work. In terms of efficiency, the Opportunities Fund is also intended to provide individual and societal benefits by increasing the financial independence and self-sufficiency among persons with disabilities through paid work. The inclusion of persons with disabilities also maximizes the labour market pool and the number of Canadians who are able to make a productive contribution to the economy and to society.

Key informants generally agreed that the federal government has an appropriate role to play in labour market programs for individuals with disabilities. Respondents noted that services available to persons with disabilities are uneven across provinces, with some provincial governments providing dedicated employment programming targeted to individuals with disabilities, while other provinces have very little available. The federal government was also perceived to have a responsibility under the Canadian Charter of Rights and Freedoms to ensure equity of opportunities for all citizens. Finally, respondents identified the federal government as having a leadership role to play in this area. This leadership was defined as providing resources, skills, moral persuasion and focus to encourage and supplement this type of programming.

The potential for duplication and overlap between the Opportunities Fund and other provincial or federal programs was addressed in the key informant interviews and stakeholder discussion groups. While respondents acknowledged that there may be some overlap between OF and provincial programs (due in part to confusion stemming from many recent changes in federal and provincial programming), consultation and exchange among those who deliver the program attempt to guard against duplication or competition for resources.

However, several respondents from within government noted that there was a potential for duplication in the future with the establishment of the Employability Assistance for Persons with Disabilities (EAPD) program (formerly the Vocational Rehabilitation of Disabled Persons program). While not yet fully implemented, the EAPD will focus on providing employability services to persons with disabilities on a cost-shared basis with the provinces. It is possible that the program will provide similar types of services to a similar target group, though the EAPD does not have many of the features embodied in the OF (e.g., involvement of disability groups). While some key informant and discussion group respondents from the stakeholder groups noted that the need for programming in this area is so great as not to preclude a number of different programs, nevertheless there may be a need to examine structured methods of co-ordination and linkages between these two programs in the future.

(b) Rationale for Program Design

There are several key features of the Opportunities Fund which distinguish it from other programs available to persons with disabilities including:

• the program was designed and implemented with the participation of persons with disabilities and organizations representing persons with disabilities;
• the interventions vary substantially depending on the sponsor and individual action plans developed with the clients;
• there is a focus on measurable employability outcomes and monitoring of the clients; and
• leveraging of resources and funding is required, including partnerships with all levels of government, non-governmental organizations (NGOs) and private sector companies as well as the effective use of EI funding (e.g., counselling and job search assistance).

Findings from the literature review, key informant interviews and discussion groups support these key program design features. The literature review², for example, indicated that the most effective labour market interventions are those that directly involve persons with disabilities. This involvement refers both to including persons with disabilities in determining the types of assistance required and making their own decisions regarding the purchase and direction of personal support, as well as having persons with disabilities, or their representative organizations, participate in the design and implementation of programs serving the workforce needs of persons with disabilities. The qualitative research also strongly supported the effectiveness of having direct involvement of persons with disabilities in the delivery of the program.

The OF has attempted to include the disability community through the establishment of a National Reference Group composed of representatives of national organizations devoted to disability issues. This reference group, drawn from the Scott Task Force Reference Group, has been integrally involved in the development of the program guidelines, as well as in overseeing the implementation of the program and its evaluation. The involvement of this group is also intended to ensure the ongoing relevance of OF over the life of the program.

At the local level, the involvement of the disability community is uneven and depends to some extent on the experiences and prior partnerships with the HRCC. Some respondents from the disability community felt there was an over-reliance on established or mainstream organizations and not enough outreach to local-level groups. Other key informants expressed concern about the expertise of HRDC/HRCCs to adequately serve the community of persons with disabilities, given the experience, sensitivity and knowledge that these respondents felt was required to provide effective services to persons with disabilities. Several respondents from the disability community felt it was important to further enhance their involvement at the local level in terms of building capacity within disability organizations to develop and deliver programs. Others also suggested greater involvement in setting priorities and approving projects at the local level (though other respondents noted the potential conflict of interest in this approach).

The literature review and key informant interviews provide support for the second key feature of OF � the use of individualized services and action plans. The literature clearly indicates that programs designed for persons with disabilities must recognize the great variation within the client base (HRDC, 1997). Moreover, in many cases, a holistic approach is required to effectively integrate persons with disabilities into the workforce. The needs of a person with a disability can be wide-ranging: employment may be impossible without assistance in a number of areas including transportation, housing, housekeeping, attendant care, income support and training. As a result, there is a need for flexible programming for clients with disabilities and the need for individualized interventions. Individual action plans ensure that clients are able to assess their needs and also can increase the self-esteem and confidence of participants as they begin to undertake action planning and interventions.

The focus on employment outcomes and monitoring of clients was found to be an important feature of effective interventions. Combining early intervention with case management and follow-up of clients with disabilities further enhances the value of the intervention. During the key informant interviews and stakeholder discussion groups it was noted that there is some friction between two of the guidelines: maintaining a balanced portfolio (that is, representing the full range of disabilities across OF projects and including clients who are comparatively less "job ready") and producing measurable results and maximizing the number of clients who become employed and self-sufficient. Stakeholder/ delivery organizations in some centres felt there was too much focus on the latter principle � employment results � at the expense of the former � achieving a balanced portfolio. This creates pressure to select clients who are more job ready and to avoid persons with more severe disabilities.

Just as the widely varying characteristics of the population of persons who are disabled and their circumstances must be considered when assessing what level and type of assistance is required, so must this variation be taken into consideration when measuring the impacts. Reasonable expectations for employment results must be maintained based on the employability of the clientele served by the program and the intervention when measuring success. While the overall expectations of the program are a balanced portfolio and reasonable employment targets, it is possible that this message has not been clearly communicated in all centres. That the guideline on employability does not specify a timeframe was viewed as positive by many respondents. This issue could also be resolved by establishing targets at the local level to ensure a balanced portfolio.

The interfacing of OF with other programs, particularly EI Part II, the final key feature of OF discussed here, is supported to the extent that there is an increase in the efficiency of use of OF funds. The use of EI Part II funds was intended to conserve OF resources by directing clients to use these Employment Assistance Services prior to the more active labour market services offered by OF. Persons with disabilities who are EI eligible are also directed to services provided under EI Part II and under HRIF³.

While this approach is intended to enhance the efficiency of OF, it may be prudent to examine the extent to which HRIF and EI Part II programming has proven to be fair, appropriate and accessible for persons with disabilities. The Scott Task Force noted that HRIF programs must be inclusive for those with disabilities. Ideally, this programming would reflect the principles identified in the literature as associated with success and which are reflected in the OF approach. Some key informants and discussion group participants believed that EI- eligible clients (including "reachback" clients who have used EI in the last three years) should be included under OF.

The extent to which "mainstream" services such as the Employment Assistance Services funded under EI Part II are accessible for persons with disabilities is particularly important given that this may be an early point of contact for clients with disabilities. Past evaluation research has shown that timing is critical to successfully integrating persons with disabilities into work and society. This was the lesson learned from the experience of the National Vocational Rehabilitation Project associated with CPP (Disability) in Canada, as well as the General Accounting Office in its examination of US programs (HRDC, 1997).

2.2 Program Implementation

(a) The Implementation Process

As described in the previous chapter, the implementation of OF is highly decentralized. Funding is provided to regions which have, in turn, developed models for the distribution of funds at the local level. Prior to allocating funds, all regions were required to develop a strategic plan outlining the key priorities for the area, planned activities, anticipated results and organizational involvement. Key informants agreed that projects were generally implemented in accordance with regional and national strategies.

Feedback both from designers and from those responsible for the implementation of OF indicated that where there were weaknesses in implementation. These were largely with respect to information dissemination. The program was underway very quickly and little time was afforded organizations to become familiar with the program. As well, there were some concerns about raising expectations about the program beyond the level of funding that was available. A mass mailing of 2,500 was conducted to established disability organizations to publicize the program and members of the national reference group were expected to promote the program among their constituents. However, the survey of sponsoring organizations revealed that the lowest levels of satisfaction were for the information initially available about OF. Key informants, as well, believed that the dissemination of information and outreach to organizations could have been better at the outset of the program.

Inadequate promotion of the program was also identified as a weakness by OF participants. In the discussion groups, many participants mentioned that awareness of the program was poor and these individuals expressed concern that many persons with disabilities who could benefit from OF were not currently being reached by the program. In particular, individuals with a disability who are not involved with an established organization or who are not well represented by their association may potentially be unaware of the program.

Some stakeholders/delivery organizations believed that there was a lack of structured information/training on the program to help HRCCs in program delivery. Combined with the considerable flexibility and discretion in the program, this was viewed by some as having had consequences in terms of the application of the program guidelines. The individuals noted that there were discrepancies across locations in terms of the interpretation of the guidelines and the types of projects approved. Some key informants found these inconsistencies to be major problems, while others regarded them as symptoms of the "growing pains" one typically sees in the early stages of new programs.

Another aspect of information dissemination is sharing of experiences and practices among organizations participating in OF. Some stakeholder/delivery organizations noted that they met informally to share information at the community level and some groups noted that they had plans in place for sharing lessons learned, such as through the Internet. At this stage of the program, there have been very few opportunities for sharing information about lessons learned, and under the national projects, finding success stories. However, based on the key informant interviews and the input received from the fax-back forms, it is too early for the sponsors to have enough experience with OF to comment on lessons learned. It should be noted that OF is currently exploring options for sharing of practices using an Internet site. A feasibility study, including community consultations, is currently underway.

In the survey of sponsor organizations, respondents were asked to rate the extent to which a number of the guiding principles of OF described their project (Exhibit 2.1).

Half of the eight principles which were probed received very high ratings (85 percent to 92 percent agreement), including active participation of persons with disabilities and their organizations in delivery, helping individuals with disabilities move into employment, activities producing measurable results, and activities adding to and complementing existing programming. Strong but less enthusiastic ratings were provided for sharing of practices and experiences and post-placement follow-up of clients and employers. The most challenging guidelines to implement revolved around establishing partnerships with the private sector. Working with partners such as employers and unions and leveraging funds from other public and private sector sources received the lowest ratings with 68 and 50 percent of respondents respectively stating that these guidelines described their project. The views expressed by delivery agents in the discussion groups are generally consistent with these survey results.

One of the recommendations of the Scott Task Force was for a fund which would provide for innovative ways to integrate persons with disabilities into the labour market. While the OF is intended to be innovative in process, the notion of innovation is not included in the program funding guidelines due to concern about the potential to exclude "tried and true" approaches that have been found to be effective in the past. Several key informants from organizations delivering the program have noted, however, that the program has, in some cases, focused on familiar practices at the expense of new or creative strategies to encourage employability. This may be linked in some centres to the perceived pressure to achieve employment targets.

(c) Leveraging and Partnerships

When queried about the use of OF in combination with other assistance programs such as EI or Workers Compensation, 58 percent of project sponsors indicated no other assistance programs were accessed compared to 34 percent who reported delivering OF in combination with other programs.

When asked about the other programs that had been used, 69 percent of project sponsors reported provincial government programs, including Social Assistance (Exhibit 2.2).

Most of this provincial government involvement in terms of assistance programs likely preceded the OF funding and does not necessarily represent leveraged funding. Approximately 10 percent of the organizations delivering services reported using EI to complement the OF. (Note that these data are based on a small number of cases.) While the intent of OF was to use EI Part II resources where possible, relatively few organizations reported delivering their programs in combination with EI.

There has been only limited involvement from employers, unions and other partners to develop strategic employment opportunities. When asked more broadly about establishing partnerships to deliver the Opportunities Fund, 65 percent of project sponsors stated partnerships had been established to deliver programs and services. In terms of the types of partners, the most frequent response was other non-profit/voluntary organizations representing persons with disabilities (70 percent), followed by the provincial government (40 percent), other non-profit organizations (34 percent) and employers (25 percent) (Exhibit 2.3). There has been a low level of union involvement, both as partners and in the development of strategic employment opportunities.

Two-thirds stated that partner contributions included advice or expertise, 44 percent were involved in service delivery, and 32 percent provided employment experience (Exhibit 2.4).

Initially, 46 percent of project sponsors indicated they had received funding through their partners. However, when probed again specifically for in-kind versus a financial contribution, the percentage reporting either financial or in-kind funding rose to 55 percent, with 31 percent receiving a financial contribution and 45 percent receiving an in-kind contribution from their partner.

The statistics surrounding the amount of the financial contribution have a wide margin of error due to the small percentage who received a financial contribution and the large percentage of recipients who did not know or would not state the amount received (27 percent). Just under half (47 percent) of those providing a dollar estimate stated the dollar amount was under $25K and 30 percent reported amounts exceeding $50K.

Based on the evidence from this evaluation, the program appears to be reaching the intended clientele. Across all sponsors and delivery agents, there is a wide range in the nature and severity of the disabilities of the OF clients (one of the guidelines of the program). The only issue which was raised in the evaluation with respect to targeting was the possible misinterpretation of the meaning of "balanced portfolio". Some key informants suggested that the principle of a balanced portfolio is sometimes misinterpreted to mean that initiatives with a cross-disability focus should be given priority. This interpretation is incorrect as the intent is to achieve a balanced portfolio across all projects, not in each project.

Several representatives from stakeholders/delivery organizations and some HRCC representatives believed that the eligibility criteria could be extended to include EI- eligible clients and those who are currently employed but who require assistance to continue working. These respondents felt that often these individuals could benefit and be better- served by the coordinated assistance provided by OF. Although these individuals are eligible for EI Part II services, several respondents believed that EI-eligible individuals had not been able to access resources under EI Part II.

One of the key issues that was to be addressed at this formative evaluation stage was the extent to which sufficient information is being maintained for program accountability and to permit the conduct of the summative evaluation study which plays an important role in accountability. There are three types of information that are maintained on the Opportunities Fund program. These are:

• Participant-based data such as profile data on the client and types of activities and outcomes. This information is held corporately in the National Employment Services System (NESS) and is intended to be collected through a combination of the NESS (for clients who are directly case-managed by the HRCC or small-scale projects) and the department's Contact IV software application that is used by third party delivery organizations for case management. Fields include, for example, client information, action planning, case management, interventions and results. A portion of the Contact IV data is uploaded to the corporate level on NESS. Note that the portion of data that is uploaded does not include the participant's telephone number.
• Project-level or contract information such as sponsoring organizations that enter into agreements with HRDC to deliver services and the types and amounts of contributions. This information is held on the CJSII database within NESS.
• Financial information such as Opportunities Fund budgets, commitments and actual expenditures. This information is held in the "FISMARK" system.

At this stage of program implementation, the availability of program and monitoring data has been irregular. Contact IV is a relatively new platform that is being used by HRDC and, as a result, the procedures are unfamiliar. While technical problems have been a barrier to the smooth implementation of Contact IV in some cases, concerns were also noted by respondents about the extent of delays in receiving information at the corporate level. The forwarding of data from third parties to HRCCs and then from HRCCs to NESS is not occurring as quickly as had been hoped. These delays are largely attributed to resource constraints and competing priorities, particularly at the HRCC level.

The survey of project sponsors also indicated that a relatively small proportion of the organizations that deliver OF programs and services would appear to be using Contact IV to track information on their clients (Exhibit 2.5).

Only half of surveyed organizations were aware of Contact IV (49 percent) and less than half of those aware of the package stated they were using the software to maintain information on their OF clients (43 percent). Overall then, 21 percent of organizations that coordinate and deliver OF programs and services are using Contact IV. The reasons for not using the software were diverse but the most common reason was inertia (not set up yet � 40 percent). Discussion group participants also mentioned concerns around confidentiality of the information and the extent to which Contact IV was able to collect information on the qualitative or non-labour outcomes of participation in OF. The reviews of those organizations that had implemented the software were mixed. 35 percent were dissatisfied and 39 percent were satisfied. (Note that these data are based on a small number of cases.)

The purpose of this chapter is to present a descriptive profile of activities which have been sponsored by the Opportunities Fund to date, including program clientele and interventions. The data presented rely largely on the survey of project sponsors conducted as part of the evaluation. Note that within this survey, two distinct sponsor groups were identified: organizations providing work term experiences under OF (usually a targeted wage subsidy program), and organizations that coordinated or brokered services to multiple clients. The two groups responded to many of the same items on the survey; however, separate modules were also developed for each group to reflect their varying roles in the program.

HRDC was the main source of awareness of OF (45 percent) among the sponsoring organizations. This may have been a result of the initial mailing to organizations which introduced the program or through outreach at the local level. Other sources of awareness of the program included: professional associations (19 percent); colleagues (13 percent) and clients (12 percent) (Exhibit 3.1). Those respondents who heard about the program through another disability organization could be categorized under the "professional association" or "colleague" response categories.

Referrals were the most common method that project sponsors used to attract potential clients; 49 percent cited referrals from other organizations representing persons with disabilities, 33 percent had referrals from the HRCCs, and 13 percent from social services/welfare. Over one-third did not use any specific marketing to promote their OF programs and services to potential clients (Exhibit 3.2). In the discussion groups, several project participants learned about their OF-funded project through a newspaper/magazine ad.

(c) Motivations for Participation

Organizations

For the majority of organizations who provided a work term under OF, the main reason for participating in the program was a conviction to help persons with disabilities (52 percent).

The work placements, however, were also meeting a need for the organization itself. Overall, 29 percent of organizations stated that the main reason for participating in the program was a work requirement or the need for new recruits. One in ten organizations cited the wage subsidy was the main reason for their participation. Within this group, private sector organizations were more likely than non-governmental organizations to name the wage subsidy as the primary reasons for participation (17 percent compared to 2 percent of NGOs). NGOs on the other hand were more likely than private sector employers to state that a work requirement was the primary reason for participation (Exhibit 3.3).

Participants

For participants, the main reason for participating in the Opportunities Fund was to improve their employment prospects. Those who participated in the discussion group discussions described a wide variety of services and activities in order to achieve their employment goals. These activities ranged from simple purchase of equipment or workplace modifications to skills upgrading and training to work term experiences with the help of a job coach. Several of the participants were pursuing self-employment.

The survey results presented in Exhibit 3.4 provide a broad profile of the organizations involved in the Opportunities Fund.

As mentioned above, the database of OF contracts was composed of two types of organizations, those that provided work experience to OF clients and organizations involved in coordinating or brokering services for OF clients. Based on the survey responses, the groups were quite evenly split with 48 percent of respondents falling into the former category and 52 percent into the latter.

The majority of organizations that contract with HRDC under the Opportunities Fund are non- profit sector organizations (62 percent). Thirty percent of organizations classified themselves as private sector and 6 percent were public sector organizations (e.g., colleges) .

The regional profile of project sponsors indicates that 20 percent were located in the Atlantic provinces, 18 percent in Quebec and 21 percent in Ontario. Twenty-one percent of respondents were located in the Prairies and 20 percent were from British Columbia.

Fifty-nine percent of organizations had agreements with HRDC that were less than $10,000 in value. Forty-one percent of agreements were over $10,000. Two-thirds of third-party organizations had a single agreement with HRDC under the Opportunities Fund. Fourteen percent of organizations reported having two agreements and 13 percent reported having three agreements or more. Among organizations that coordinated and delivered programs and services, 63 percent stated the projects were new and 36 percent stated the projects were based on an existing program.

While Phase I of the evaluation could not include a survey of OF participants, some information on the profile of individuals using the program was gathered through the survey of project sponsors. Since the profile data was obtained at the organization level, these data may not necessarily reflect the profile of individual participants. This information is presented below.

Nature and Severity of the Disability

For delivery organizations that provided a work term, the most frequently cited type of disability of the employee(s) placed was mental (41 percent), followed by mobility (35 percent) and agility (17 percent) (Exhibit 3.5). Hearing impairment or sight impairment was cited by 14 percent and 10 percent of the organizations respectively. Note that the nature and severity of the disabilities reported are based on the self-reported perceptions of the sponsors.

In terms of severity of the disability, 46 percent of work term organizations assessed the severity of the disability of their employee as mild, 39 percent reported the disability as moderate and 9 percent stated it was severe. Compared to the distribution of clients for organizations that coordinated and delivered programs and services, individuals with severe disabilities are substantially underrepresented (approximately by half) in work placements and those with mild disabilities are overrepresented by a factor of nearly two. The lower participation of those with severe disabilities in work terms may be due to the risks to existing income supports involved in establishing employment.

For organizations that coordinated or delivered programs and services, the most common disability for their client group was mental/intellectual disabilities (72 percent), followed by mobility (53 percent), agility (29 percent), sight impairment (29 percent), hearing impairment (26 percent) and speaking impairment (18 percent) (Exhibit 3.6). Another two percent of the organizations stated they served all client groups. Some key informants noted that some types of disabilities were underrepresented in local OF programming to date.

Just over 20 percent of the organizations that delivered programs and services had additional criteria for participation in their OF project. Of the small number of projects with additional program criteria, the most common criteria were related to sex (52 percent), age (26 percent), aboriginal status (18 percent), and vocational rehabilitation (12 percent).

The largest percentage of clients (46 percent) served by organizations that coordinated programs and services were assessed based on the perceptions of the project sponsor as having a moderate disability. Approximately 26 percent of the clients served had mild disabilities and 21 percent were reported as having severe disabilities.

The survey examined the status of OF clients in terms of their prior work experience and use of social transfers. Among organizations who were providing work term experiences, only 15 percent reported that their OF employee had worked for them in the past.

The majority of project sponsors reported that none of their participants were EI recipients (75 percent) or they did not know the EI status of their participants (12 percent). Thirteen percent reported that some of their participants were EI clients. Based on the survey results, it would appear that most project sponsors were adhering to the program criterion that the participants not be EI eligible. This is a crude indicator, however, since the question only asks about EI recipients and does not tap the full range of EI eligible clientele. This issue is best addressed through the analysis of administrative data since third party organizations may not fully understand or know the EI status of their clientele.

Our expectation that the majority of OF participants would be social assistance recipients (SAR) was not fully confirmed. While 14 percent of project sponsors indicated that they did not know whether any of their clients were social assistance recipients, 45 percent stated they had no SAR participants and 42 percent indicated that at least some of their participants were SARs. The status of this large portion of non-EI/non-SAR clients is unclear. Based on discussion groups with participants, a portion of clients may be Workers' Compensation or CPP/ Disability claimants or possibly have other sources of support such as family. An accurate profile of OF clients will not be possible until a survey of participants is conducted during the later stages of the evaluation.

Note that this high incidence of participants who are not social assistance recipients may pose significant problems for the design of the mid-term and summative evaluations, particularly in identifying an appropriate comparison group for OF participants. The primary sampling frame for the comparison group may still be based on a social assistance recipient database, however, alternative sources may have to be sought for comparison cases for non-SAR/ non-EI OF participants.

3.2 Benefits and Measures Offered

For delivery organizations that provided a work term, the main types of services or activities offered were on-the-job training (78 percent) and the workplace experience (71 percent). Formal training was relatively rare (13 percent) (Exhibit 3.7). The average work term was over six months (29 weeks) (Exhibit 3.8). A majority of work terms were less than six months in duration (56 percent) and one in five were less than three months. The work placements with private sector firms tended to be shorter than work placements in organizations from the non-governmental sector (an average of 26 weeks versus 32 weeks).

Among organizations that coordinated and delivered programs and services, the focus of activities was on improving the employability of participants through services related to finding employment or improving skills and experience. The most common services offered were job placements/work experience (64 percent), employment counselling/job search advice (64 percent), and training programs (including pre-employment programs) (63 percent) (Exhibit 3.9). The next most common activities included targeted wage subsidies (39 percent), labour market information (39 percent) and assistance for self-employment (35  percent).

The delivery of services such as employment counselling and labour market information was almost exclusively provided by the organizations funded under OF themselves (as opposed to other partners). In terms of training, three-quarters of the funded organizations stated they provided the training themselves, while 20 percent reported delivery by a college or university. A similar proportion of the funded organizations were also responsible for the delivery of self-employment assistance (72 percent), with the second largest category being other NGOs. The activities least likely to be undertaken directly by the OF-funded organization were job placement/work experience and the targeted wage subsidy. Approximately 61 percent of the funded organizations stated they provided the job placement/work experience to the participants themselves, while 35 percent indicated employers. The delivery of the targeted wage subsidy was largely undertaken by the funded organization (59 percent) however the HRCC was indicated as the delivery agent by 23 percent of the organizations that delivered programs and services. It is unclear whether respondents meant that their own organization had hired an OF employee or they were simply acting as brokers for job placements with external employers.

(a) Project Costs

Reflecting the flexibility of the OF program, organizations that coordinated and delivered OF used their funds to cover a wide variety of costs (Exhibit 3.10). The kind of costs covered under OF that were mentioned most frequently by surveyed project sponsors were participant wages at a job (55 percent), special devices or support related to the disability (50 percent) and tuition/course fees (46 percent). The next most frequently mentioned responses were: transportation (36 percent); other expenses such as child care (20 percent); and staff salaries (17 percent). Detailed information on the proportion of OF funding that is being used for administrative versus program costs was not collected in this evaluation.

Virtually all organizations providing work terms received wage subsidies through the Opportunities Fund (Exhibit 3.11). Training costs were reported as a cost covered by OF by 18 percent of the organizations and 15 percent reported OF covering the cost of special devices or supports for participants. Only 6 percent reported other costs such as child care expenses and 1 percent reported transportation costs as being covered by the program.

The length of time for sponsors to receive approval for their project was relatively short. (Exhibit 3.12). More than one-half of respondents received their approval in one month or less (61 percent) and only 14 percent reported their approval taking four months or more. Approval time increased with the value of the contract. Virtually all project sponsors found this approval period to be acceptable (86 percent). Stakeholder groups frequently noted a key benefit of the program was the ability to get up and running quickly with little red tape.

Length of Time to Implement Project

Most organizations also reported a relatively short implementation phase; approximately 25 percent reported that it took one week or less between approval of their project and enrolling participants in their project. Just over half (56 percent) reported the implementation as taking one month or less. Less than one quarter took four months or more to implement (Exhibit 3.12).

Virtually all respondents reported that their projects served one community/municipality (53 percent) or region (40 percent). Only 3 percent were regional in scope and another 3 percent of projects were national in scope.

Almost all organizations coordinating or delivering programs and services developed individual plans for all clients (83 percent) or at least for some clients (2 percent). Fourteen percent of organizations claimed never to have developed individual action plans with clients.

Almost all organizations that coordinated or delivered programs and services intend to monitor clients. Eleven percent indicated they did not intend to conduct any follow-up. Over half of the organizations conducting a follow-up (56 percent) stated they would conduct this follow-up every three months, 15 percent stated the interval would be more than three months, and another 14 percent stated the follow-up would depend on the individual needs. Unfortunately, these follow-up data will not be maintained centrally, representing a lost opportunity for monitoring and lessons learned analyses.

Note that in the discussion groups, some stakeholders/delivery organizations mentioned that there was confusion in some cases as to which organization was responsible for follow-up with clients. These respondents also noted that there were often few resources to conduct frequent and/or extended follow-up contacts.

Nearly three-quarters (72 percent) of the organizations that coordinated or delivered programs and services acknowledged their agreements with HRDC included targets for the results of their project. Of those that had targets, 89 percent indicated these were employment targets and 30 percent indicated targets to reduce reliance on social assistance/income transfers.

The majority of surveyed project sponsors have either one (43 percent) or no participants currently (12 percent). Nearly one-third expect to have no more than one participant (Exhibits 3.13 and 3.14). As can be expected, the most common response for private sector organizations was to expect only one participant (51 percent) (given that their involvement was more likely to be limited to providing a work term), while for NGOs only 21 percent expected only one participant.

The decision to delay the participant survey at this formative evaluation stage was vindicated by the fact that the vast majority of the participants have not completed their program � 70 percent of project sponsors indicated that at the time of the survey no participants had completed their intervention. Any outcomes at this stage are still too early to assess.

Project sponsors' overall rating of satisfaction with the Opportunities Fund was very high: 87 percent were satisfied with OF overall and 67 percent were very satisfied. Only 4 percent expressed any dissatisfaction with the program.

Respondents were asked if they experienced any challenges or difficulties in implementing their OF project. Approximately two-thirds did not indicate any specific difficulties. Fifteen percent indicated communications or information problems (e.g., guidelines unclear), 12 percent indicated funding delays, and 6 percent stated that the program implementation had been rushed. When asked to suggest improvements to the program, 44 percent did not offer any improvements, 24 percent suggested more information on the program, and 16 percent suggested longer term funding.

Virtually all project sponsors indicated they would participate in the program again (96 percent). Nearly all of the organizations (96 percent) that provided a work placement would recommend participating in OF to another organization. Only 2 percent would not.

Reflecting the overall satisfaction ratings, the ratings of various aspects of the program were also relatively high (Exhibit 3.15).

The highest average ratings were for the amount of funding provided, both the overall project funding and the amount of the wage subsidy. The next highest ratings were for the support provided by HRDC, monitoring and reporting requirements, the application process, and the flexibility of the program for experimentation. The types of activities and costs funded by the program were among the next highest ratings, as was the aptitude and skills of the work term employee.

The lowest ratings all related to information dissemination. After adjusting for the "don't know/no response", 70 percent were satisfied with the opportunities to share information and experiences with others working in the field. The lowest rating was for the availability of information about OF. Only 62 percent were satisfied with the availability of information, while 24 percent were dissatisfied. Private sector organizations had the lowest ratings; 33 percent were dissatisfied with the availability of information while this figure was 19 percent for NGOs. A lack of clear information and communications on the OF was also identified as a weakness by stakeholder/delivery organizations.

OF participants who were contacted for this evaluation expressed a high degree of satisfaction with the program. While most individuals were in the early stages of their intervention, they were very satisfied with the flexibility of the program to fund the kinds of services and activities they needed to become employed. For some, participation in programs where other participants were disabled raised their level of comfort. Participants had found the application process devoid of the kind of "red tape" that was often perceived to be associated with a government program. Finally, the follow-up was perceived to be highly valuable, providing participants with access to an individual on whom they could rely for support or advice.

In terms of program weaknesses, several participants noted some difficulties in accessing OF without jeopardizing existing income support and benefit coverage (e.g., from social assistance or CPP Disability). In several of the discussion groups, an issue was also raised around the coordination of activities and programs related to persons with disabilities. Participants suggested that it would be useful to have a single point of contact where individuals could be made aware of their options available through different programs. Also, insufficient program information and a lack of sensitivity on the part of some trainers (working for organizations which traditionally train people who are not disabled) were identified as weaknesses.

In this chapter, we present the findings on preliminary impacts and effects in the areas of labour market skills, action planning, quality of life, and to a lesser extent, employment, earnings and income transfer dependence. These findings are drawn from the survey of third party delivery organizations and employers, discussion groups with participants and stakeholders/delivery agents, and key informant interviews. At this formative stage it is appropriate to focus on short-term impacts such as skills acquisition, but premature to assess longer-term impacts on employment and quality of life outcomes, which will be fully addressed in the subsequent summative evaluation.

4.1 Post Program Employment

(a) Hiring Expectations

Nearly half of the organizations in the survey that provided a work term stated that they plan to hire the Opportunities Fund work placement employee in the future (48 percent).

As illustrated in Exhibit 4.1, the main reason organizations did not plan to hire the work placement employee was a lack of work (31 percent). Nearly 20 percent stated there were barriers related to the disability and another 18 percent stated the individual was not interested. Few organizations identified dissatisfaction with the worker as the reason for not hiring (12 percent) and only 11 percent indicated the budget was too low.

(b) Employment and Job Characteristics

Of the sponsoring organizations that stated that they intend to hire the work placement employee, 61 percent indicated that this would be a full-time job, while 36 percent stated the job would be part- time.

4.2 Employer Impacts

(a) Employer Benefits

The most frequently cited benefit for organizations providing work placements was a skilled employee (49 percent), followed closely by increasing awareness of disability issues (38 percent) (see Exhibit 4.2). Contrary to what might be expected, private sector firms were less likely to cite obtaining a skilled employee as a benefit than NGOs who provided a work placement under OF (though this difference is not statistically significant). These NGOs were more likely to identify raising awareness of training or human resource issues as a benefit (31 percent) than private sector organizations (11 percent). Modifications to the workplace were claimed as a benefit only by the NGOs providing a work placement, though only 13 percent of these organizations reported this benefit.

Approximately 17 percent of the organizations reported lower labour costs (wage subsidies) as a benefit. Although there was a significant difference in the percentage of private sector organizations and employers from the non-governmental sector who provided a work placement citing the wage subsidy as the main reason for participation, there is no significant difference in the percentage reporting this as a benefit.

4.3 Participant Impacts

In the discussion group discussions, participants in OF-funded projects identified a variety of preliminary beneficial impacts � impacts related to their job skills and employment prospects/outcomes, as well as to the quality of their lives. Key impacts include the following:

• improved self-esteem and self-confidence, as well as increased motivation and a more positive outlook on life in general (e.g., programs provide a reason to "get out of bed in the morning" and people are hopeful that they can become productive members of society);
• better quality of life because of higher income and rewarding work activity;
• setting of goals � enabling one to "dream again" � and preparation of a "return to work" action plan;
• improved job readiness skills, life skills and job search skills;
• improved knowledge and skills related to employment in a range of fields (e.g., information/computer technology, data entry, word processing, work in a call centre, sociology, television/film production, home- based business skills);
• job experience (through internships, work placements, wage subsidies or volunteer work) and good job prospects at the completion of training programs;
• actual employment for graduates of training programs (e.g., work in a call centre) and for people receiving workplace accommodations and related support (e.g., financial assistance for work-related transportation, equipment and assistive devices); and
• reduced dependency on government social assistance.

Stakeholders, delivery agents and HRDC/HRCC officials consulted in key informant interviews and discussion groups observed preliminary impacts similar to those listed above. It remains to be seen, however, if these positive indications at this early stage of the Opportunities Fund will translate into enduring employment for participants.

4.4 Incrementality

One measure of incrementality of OF is whether the projects funded under the program were new or were based on existing projects. As noted earlier, just over one-third of the projects were based on existing programs. This is just one measure of the incrementality of the funded project, however. For example, the funded project may be based on an existing program but the funding may have been used to extend the project to a new clientele, to provide new services within an existing program framework or to extend the timeframe for the existing program.

When asked directly in the survey, the overwhelming response was that the project funding was incremental. Nearly three-quarters of the respondents indicated very strongly that the project would not have proceeded without the OF funding, whereas only 12 percent indicated any strong likelihood that the project would have taken place without the funding.

Evaluation Findings

Program Rationale

The Opportunities Fund was intended to provide a response to a gap in labour market programming for individuals with disabilities that was created by reforms to EI and HRIF legislation. This gap was identified by the Scott Task Force on Disability Issues. According to the literature review and key informant interviews, there is a continued need for the OF program. There are both equity and efficiency arguments to support the integration of persons with disabilities into the labour force. The literature review noted further that as the labour market ages, the incidence of disabilities will increase, creating a greater demand for effective programs that will meet the needs of individuals with disabilities who are interested in labour market participation.

A more complicated issue is whether there is a need for the OF given the planned implementation of the Employability Assistance Program for Persons with Disabilities (EAPD). The EAPD, formerly the Vocational Rehabilitation of Disabled Persons Program, is being recast to focus on providing employment assistance for persons with disabilities. Unlike OF, however, the EAPD is a cost-shared program between the federal and provincial governments. Also, the EAPD is not expected to deliver programs in a fashion similar to OF (e.g., with the strong involvement of disability organizations). Nevertheless, some concerns were raised in this evaluation about the potential for overlap with the EAPD in the future when this program is fully implemented. As the EAPD unfolds, there will be a need in future to explore possibilities for linkages and harmonization between the two programs.

In evaluating OF, it is important to recognize that OF is not just a program, it is a process. The strong involvement of persons with disabilities and organizations representing persons with disabilities in the design and delivery of the program is an important feature. The OF reference group drawn from the Scott Task Force provides this leadership at the national level. The involvement of disability groups at the local level, while more uneven, is generally viewed as a positive and developing part of the program. The involvement of organizations for persons with disabilities creates a more sensitive and responsive environment for clients who may need more support in their ability to succeed in the labour market or to acquire the skills necessary to succeed in the labour market.

Some representatives from disability organizations suggested that there should be further involvement of this community in reviewing and approving OF projects. Other respondents noted, however, a potential conflict of interest in this position.

The flexibility and individualized approaches that are permitted under the OF program were also widely praised. The literature confirms the importance of holistic approaches to respond to the variations within the population of persons who are disabled as a key feature in the success of labour market interventions for persons with disabilities.

One issue that was noted by project sponsors was the lack of ability to serve clients who are EI eligible. While these individuals are eligible for labour market measures and benefits under EI (now largely delivered by the provinces), it should also be noted that some project sponsors viewed this exclusion as a weakness of the program and believed that in some cases EI eligible individuals were not getting the services they needed. The literature indicating that workers with disabilities respond better to programs that directly involve organizations for persons with disabilities and the fact that the earlier the interventions are implemented the more successful they will be suggest that these individuals would likely benefit from participation in OF. Alternatively, it would be prudent to examine the extent to which mainstream EI programming is fair, inclusive and accessible for persons with disabilities and, ideally, adopts the program elements that have been demonstrated to be successful in other settings.

The evaluation results suggest that the guidelines established by the Opportunities Fund are viewed as appropriate by those involved in the program and the implementation of OF has largely been consistent with the guidelines. The focus of OF projects examined in this evaluation is clearly on improving the employability of the participants. Projects also were consistent in providing follow-up with clients and incorporating results targets. Sponsors noted, however, the need for patience in assessing outcomes and that depending on the nature of the disability, some clients may not be able to be fully self-sufficient. The project activity profile indicated further that there is a broad balance in the types of clients and severity of disability, though persons with severe disabilities were less likely to be present in wage subsidy programs and some key informants identified specific groups that were not yet well-served by the program.

OF guidelines which have proved more challenging to implement include dissemination of findings and leveraging of funds. As the program matures and lessons learned on project implementation and success are more evident, a strategy to encourage dissemination of findings should become a greater focus for the program. The issue of leveraging is also important to increase the efficiency of use of OF funds and to enhance the opportunities available to clients.

One of the objectives and initial rationales for OF was to seek innovative strategies to integrate persons with disabilities with the labour market. Innovativeness was not incorporated into the program guidelines to ensure that "tried and true" practices were not ignored, though some key informants believed that to date there has perhaps been an over-emphasis on traditional approaches. This may be linked to pressures to be cost-effective and to maximize employment results.

The flow of information to HRCCs about the program and its dissemination to the community has been uneven. As well, the difficulty in balancing the desire for flexibility and responsiveness to local community needs and priorities with national guidelines and principles is visible in some of the qualitative data. There is a perception among some organizations that the guidelines are unclear or at least inconsistently applied. The insufficient availability of program information was highlighted as a weakness by project sponsors. According to some key informants, the result was that the projects approved were inconsistent among HRCCs.

The evidence gathered in this evaluation suggests that few changes to the design and delivery of the Opportunities Fund are required. Most project sponsors were impressed with the program design and satisfied with the length of time for approval, although some noted that the program was conceived and implemented in a short time frame. Many respondents valued the flexibility of the program and the ability of the program to provide a quick response to clients.

There were few suggestions for improvements, however, a number of project sponsors suggested that better information at the time of application would be an improvement. An ongoing, coherent communications strategy would be valuable. The planned Internet site, once developed, will provide some avenues for communication, as well as exchange among organizations. A substantial number also indicated that longer term funding would be beneficial. The three-year timeframe was viewed as being too short for programs to become well-established and effective. As mentioned earlier, some project sponsors indicated that the non-EI eligibility requirement was a weakness. As well, the literature suggests that some of the features of OF would be of benefit for all individuals with disabilities, regardless of the funding source for services and programs.

The limited amount of active promotion of the Opportunities Fund to clients and smaller community-based organizations raises concerns about the accessibility of the program. This is not necessarily an issue for individuals who have made a decision to enter the labour market, assuming that they are very likely to contact an HRCC or an organization representing persons with disabilities for job search assistance or related services. A more serious challenge may be for individuals who do not enter the labour market because they are not aware of the programs and services offered under OF.

Leveraging of existing assistance programs may have been less than anticipated initially. Among the organizations who delivered programs and services, only one-third indicated accessing other programs and services and OF was often likely delivered in combination with social assistance rather than other labour market interventions. While the limited success in leveraging may again be the result of the relative recency of the program, there may also be a need to focus attention on building capacity among organizations to approach other government and corporate/union partners and developing appropriate communications to raise the profile of the program among potential partners.

The use of EI resources for job search/counselling was consistent with the intent to use OF funding effectively, however, only 10 percent of the organizations reported this use of EI. On the other hand, almost two-thirds of project sponsors reported providing employment counselling/ job search advice to their clients themselves.

Two-thirds of the delivery organizations reported partnerships outside of their relationship with the federal government, including other non-profit organizations, the provincial government, and employers. Union involvement was relatively rare. The partnerships included activities ranging from advice to service delivery and direct funding of the project. The funding was more likely to be an in-kind contribution, but one in five organizations involved in the delivery of programs and services received a direct financial contribution from its partners.

Although a large proportion of projects were based on existing programs, virtually all sponsoring organizations indicated that the projects would not have proceeded without OF funding. Three-quarters of respondents reported that their project would not have taken place without OF funding.

It is too early to assess the impacts of OF on the skills acquisition, labour market and quality of life outcomes of the participants at this time; the majority of the current participants have not yet completed their program. Consequently, lessons learned about the effectiveness of the different types of interventions also cannot be assessed at this stage of the evaluation. One very promising finding was that the majority of the organizations that provided a work placement were very satisfied with the OF participant and nearly half intended to hire the participant in the future.

Unfortunately, information on what works and lessons learned may be delayed or may not be available on an ongoing basis. Although most organizations intend to monitor their clients, a minority (14 percent) do not develop action plans and do not intend to follow-up with their clients (11 percent). Only about one in five organizations are using the Contact IV software to collect client information at this time. Indeed, only half of the project sponsors who were interviewed as part of this evaluation had heard of Contact IV. There were also reports of difficulties or delays in transferring local level data to the national system. Without a formal system to comprehensively track results, information available on client outcomes will not be consistent or comprehensive, limiting the usefulness of these data to assess what works and what does not. It should also be noted that at present the fields that are uploaded from Contact IV to NESS do not include participants' telephone numbers, which presents a barrier to future contacts.

Evaluation research may not be able to fill this void. The decentralized delivery of the program coupled with the absence of complete national level data presents challenges in terms of directly contacting OF participants to assess the impacts of the program. The extent to which OF agreements with the sponsoring organizations include a provision for the collection of client contact information to be released for research purposes is not clear. During the course of this evaluation many project sponsors and HRCCs were unwilling to release participant information citing confidentiality concerns. The lack of participant-level contact information would severely limit the ability of future evaluation studies to assess the effectiveness of the interventions and the lessons learned.

The success of future evaluation activities will be enhanced by a strong communications strategy to ensure that HRCCs and project sponsors have a clear understanding of evaluation objectives, timing of activities and their responsibilities with respect to confidentiality/ release of information. As well, given some of the potential sensitivities around contacting persons with disabilities (e.g., use of self-identified equity information, intellectual capacity of some participants), up-front communications with participants will also be important to ensuring a high comfort level with the evaluation.

There are a number of options or strategies to ensure that the data requirements for the summative evaluation are met. In essence, the data requirements revolve around the need for accurate and comprehensive OF participant contact data. Options for compiling this information include:

1. Use of national NESS data. Use of a single data source would simplify the evaluation exercise. However, use of NESS assumes that the use of Contact IV be much more widespread among project sponsors than is currently the case and that these data are regularly uploaded to the NESS. The absence of participant telephone numbers is problematic also. Linkages with other administrative databases (e.g., social assistance, status vector or T1) may provide contact data for some cases. The remainder would have to be tracked manually through directory assistance (an expensive exercise that also opens the possibility for bias in underrepresenting participants who have relocated).
2. Obtaining Contact IV data from the regional/local level. While avoiding the problems associated with the NESS upload, this method also assumes that project sponsors are using Contact IV in a regular and consistent fashion and that these data are available at the HRCC level. This method also places the burden of data collection at the HRCC level, which may be difficult given current resource constraints.
3. Obtaining participant contact information directly from project sponsors. This final option involves contacts with each project sponsor and requesting that participant information be forwarded to an HRDC OF/evaluation representative or outside consultant. This option would provide the most comprehensive participant information (including organizations that use Contact IV as well as those that do not). This process, however, is very time consuming. As well, based on the experience of this evaluation, proper communications and release of information guidelines must be in place to encourage project sponsors to forward this information.

While Phase I of the evaluation has provided useful descriptive information on program implementation and process issues, the mid-term and summative evaluations will be better positioned to collect information on program results. A survey of OF participants represents a crucial component of this exercise. These data would permit analyses of any differences among participants based on key socio-demographic and labour market characteristics such as age, sex, region, aboriginal and visible minority group status and prior work experience. From these data, the final evaluation results would contribute to an understanding of the profile of OF participants, the impacts and effects of the program, and the types of interventions that have proven most successful for persons with disabilities.

Evaluation Questions

Rationale

1. Does (the implementation of) the Opportunities Fund (OF) reflect the basic principles underpinning the program?

2. Did the implementation of the OF reflect the plans set out in the regional and national implementation strategies?

3. Were the early guidelines for the development of implementation strategies preserved?

4. Did the program design and implementation provide progress monitoring and clientele support after the completion of the intervention?

5. Is the proper information being collected to enable the summative evaluation?

6. Did the OF assist individuals in improving their labour market skills?

7. Did the OF assist individuals in obtaining employment or reemployment?

8. Did the OF assist individuals in improving their earnings?

9. How effective is the OF client action plan?

10. To what extent did OF assist individuals in reducing their dependence on income transfers?

11. To what extent has the participation in OF impacted on the quality of participants' lives?

Cost-Effectiveness

12. Did the program succeed in leveraging funds from other program sources?

Case Study Highlights

Case studies provide vivid illustrations of the kinds of activities that have been implemented under OF. For Phase I of the evaluation of OF, five case studies were conducted. These included:

• the umbrella organization in Quebec, the Comité d'adaptation de la main- d'oeuvre pour personnes handicapées (CAMO);
• a national project, Creating Employment Opportunities for People who have an Intellectual Disability; and
• regional/local projects in Victoria, Red Deer and the Toronto area.

Aspects of each of these cases are highlighted below. A more detailed description of each of these case studies is available under separate cover.

Building capacity to create employment opportunities.

The Canadian Association of Community Living is leading a national project under OF to build the capacity of local organizations and their communities to partner with business to create sustainable employment opportunities. One of the key strategies is the development of a one-day workshop for Association of Community Living members. The workshop, titled "Ready...Set...Go!" provides participants with practical guidelines and suggestions for approaching potential business partners and building long-term successful relationships.

Needs assessment and harmonization.

In Quebec, the implementation of OF is being preceded by a needs assessment process in each administrative region in the province conducted by the CAMO. The needs assessment will contribute to an understanding of the needs and priorities of each area and to refine the OF funding criteria on this basis to cover any gaps. The needs assessment is based on existing statistical data at the regional level and a qualitative assessment of perceived needs conducted by an external consultant. It is hoped that this process will ensure that OF funding is properly harmonized with existing programs sponsored by the Office des personnes handicap�es du Qu�bec (OPHQ) and to ensure that needs are being met.

Delivery in partnership.

In the region of Peel, Halton and Dufferin in Ontario, the Centre for Worklife and Assessment was established to deliver OF. The mandate of the organization is to ensure that the employment needs of persons with disabilities are met. The Centre is a virtual organization � program activities are undertaken in the community or "brought to" the client in their home. Office space is provided as an in-kind contribution by the HRCC and other partners. The Centre was established by the Coalition for Persons with Disabilities composed of service delivery organizations, persons with disabilities and other individuals in the community. This group remains actively involved in guiding the delivery of OF. As well, a consortium of local employers plays a key role. This group is involved in creating employment and training opportunities, promoting the program to other employers, and securing sponsorships and in-kind contributions.

Active outreach and promotion.

In Red Deer, Alberta, OF is delivered by a coordinator based at Employment Placement and Support with the help of an Advisory Group representing 12 local stakeholder organizations. Over 40 clients have been assisted under OF. In this rural area, active outreach has been an important factor for the success of the project and the coordinator has played a key role in the promotion of OF. Simple, tailored marketing materials were developed for the program. Strategies to promote the program have involved consultation meetings with stakeholders, delivery of program brochures, meetings with service providers and coordinators in each area and promotion through existing networks.

Individualized and flexible approaches to support employment.

In Victoria, BC, OF is delivered in collaboration with a local employment agency, Techniques Employment Services. Over 30 clients have participated in OF. Clients have a wide range of disabilities and an equally broad variety of interventions have been implemented, ranging from small-scale, simple measures (such as a job finding club) to more intensive approaches (such as education/training expenses or purchase of assistive equipment). Assistance for self-employment has been an important and innovative intervention. The length of activities may vary from one month to 18 months. A return to work action plan is prepared with each client and clients are encouraged to take as much responsibility for this plan as possible.

Footnotes

1	Fawcett, G. Living with Disability in Canada: An Economic Portrait, Ottawa: Human Resources Development Canada, Office for Disability Issues, 1996.
2	Human Resources Development Canada, Evaluation and Data Development, Lessons Learned from Evaluation of Disability Policy and Programs, Ottawa, 1997.
3	Note that with the establishment of Labour Market Development Agreements between the federal government and most provinces and the devolution of labour market responsibilities over the next several years to the provincial level, this program landscape may change considerably.