|
|
     |
||
 |
     |
 |
|
Formative Evaluation of
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Table of Contents]](/web/20060126222803im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif) |
| Shelley Borys | Principal Investigator and Project Manager |
| Norman Leckie | Senior Consultant |
| Bill Blois | Research Analyst |
| Benoît Gauthier | Senior Advisor |
| Ken Innes | Database Manager |
| Lina Danileviciute | Database Analyst |
| Mark Anderson | Survey Manager |
| Whitehorse-Based Subcontractors: | |
| Russell Graham, Graham and Associates | Phase I Interviews |
| Donna Larsen, DataPath Systems | Phase II Interviews and Focus Group |
While Ekos Research accepts responsibility for any errors contained in this report, we would nonetheless like to acknowledge the important contribution of a number of individuals to the successful completion of this project. The members of the Joint Evaluation Committee (JEC) for the Formative Evaluation were, in no particular order, Derek Brackley of Human Resources Development Canada (HRDC), British Columbia/Yukon Region; Brent Slobodin and Carol Leef of the Department of Advanced Education; Leslie Levy of HRDC, Whitehorse; Lyle Thompson and Tim Brady of Health and Social Services; and Jay Alam of HRDC National Headquarters. The JEC provided valuable advice and guidance to the research team throughout the project. We would also like to thank all respondents to the surveys, interviews, focus groups and the case study who took time out of their busy schedules to participate in this evaluation. Without their cooperation, this research would not have been possible.
|
Under the terms of the Canada/Yukon Labour Market Development agreement (LMDA), the federal and territorial governments agreed to jointly manage active labour market measures for the unemployed in the territory. This LMDA came into being as a result of the federal government's commitment, announced in November 1995, to devolve responsibility for labour market development to the provinces and territories. This commitment was made in response to the expressed desire by territorial and provincial governments to assume greater control over employment development programs. This report presents the results of a formative evaluation of Employment Benefits and Support Measures (EBSMs) under the terms of the Canada/Yukon LMDA.
The Canada/Yukon LMDA, which came into effect on January 24, 1998, sets out the individual EBSMs themselves. Canada and the Yukon are jointly responsible for the design, management and evaluation of the EBSMs, which mirror those outlined in the Employment Insurance Act. Four employment benefits were the focus of the current evaluation: Targeted Wage Subsidies, Self-Employment, Job Creation Partnerships, and Training Purchases/ Enhanced Feepayers. The support measures considered here are Local Labour Market Partnerships and Employment Assistance Services. Another employment benefit (Skills Development Employment Benefit) and a support measure (Research and Innovation) are briefly profiled, but were not included in the context of this evaluation.
Multiple methods were used to collect data for this formative evaluation. Specifically, quantitative methods (i.e., surveys) were used to obtain early results which could be applied to conclusions about the full population of participants in EBSMs. Qualitative methods (i.e., key informant interviews, focus groups and a case study) as well as a literature review, were used to gather particular in-depth perspectives on the research issues.
Overall, the EBSMs were perceived to be functioning quite well, although various aspects of the implementation and delivery of the EBSMs require improvement. In general, the areas for improvement concern communications, information sharing and resource issues. The main findings and conclusions from the formative evaluation are described below.
LMDA programs and services were generally perceived to be consistent with EI legislation and the priorities of the Yukon and were felt to be meeting the needs of clients, employers and communities. In instances where programs and services were felt to be less relevant (for example, Human Resource Centre of Canada [HRCC] not responding to clients' emotional needs, program eligibility, timing of wage subsidies), these perceptions could often be traced to a lack of understanding of the role played by EBSMs in the range of employment programming in the Yukon (for example, increased use of third-party providers in service delivery, targeting EI clients and need for programming in the off-seasons). Nonetheless, a number of factors were identified which may decrease the relevance of programs and services, including gaps in services provided and clients served, aspects of the design and delivery of programs (for example, emphasis on volume of clients served, definitions of success as simply employment), the need for coordination of program delivery, the need for more partnering (for example, with First Nations), communications problems (for example, information sharing between government, third parties and stakeholders), and resource issues (for example, need for training, supports for partnership).
Partnering efforts have shown improvement since the LMDA was first implemented. While partnering efforts among governments seem to be working at senior management levels, more partnering may be required among and within the government partners, as well as among senior and less senior levels in terms of communication and information sharing. Respondents to this evaluation felt that the greatest improvements in partnering would be realized through increased communication and consultation with partners, be it better reporting of the results of joint planning exercises and evaluations or more coordination of effort and consultation in the design, delivery and funding of programs and services. The development of common priorities among the three partner departments, as well as the creation of appropriate supports for partnering activities, may also improve this aspect of the LMDA.
A range of partnerships exist among government, employers and the community, including community consultations and informal networks among third-party delivery agents. Respondents generally felt that these relationships would also improve through increased communication and information sharing.
Overall, LMDA programs and services tend to be somewhat complementary to other non-LMDA programming delivered in the Yukon. Although no evidence of duplication was found, the primary area in which overlap was perceived to exist was in the types of services delivered through Starting Points and Job Finding Clubs (for example, resume writing, job search, etc.). Although some respondents felt similar client groups were targeted by different programs, these sources of overlap are not likely to represent major problems in that the duplication of programming was occurring with non-LMDA targeted client groups (similar programming for youth and First Nations). In other cases, there is some question about whether the example represented a true overlap in programming (i.e., similar client group targeted but different eligibility requirements). One form of duplication, the use of multiple interventions, was perceived to have positive consequences for participants. Administrative overlaps may also be occurring and were felt to be due primarily to the lack of information sharing between governments and third parties which has required all parties to collect the same information for the same clients. Suggestions to reduce administrative overlap were focussed on better communication and coordination among all groups.
From the perspective of clients, the delivery of services under the LMDA is thought to be good because of the relevant content of programming and the convenience of services (for example, no waits, staff well-prepared). Nonetheless, there is room for improvement of some aspects of service delivery, particularly the quality of services such as counselling and job search advice, and information on the types of programs and services available. Survey evidence also showed that a large majority of clients are accessing self-serve resources, and that this is typically occurring at an Human Resources Centre of Canada (HRCC) or through the Internet. The use of return-to-work action plans was only moderate, although there was high rate of completion of action plan activities.
Respondents provided mixed feedback on the extent to which programs and services are sufficiently flexible to adapt to the needs of the community when they are used as designed. Although a number of third-party deliverers felt programs were sufficiently flexible, it was clear that some lack a thorough understanding of the rules and guidelines for program administration.
The delivery of programs and services relies heavily on the use of third parties and they were perceived to be doing a good job. A primary obstacle to effective service delivery among third parties was the lack of information sharing to allow delivery agents to manage clients effectively. Positive changes to service delivery that have occurred under the LMDA include more autonomy among third parties, partnerships to support third-party service delivery and a more client-centered approach.
Non-Human Resource Development Canada (HRDC) government respondents generally felt that case management was not working, although HRDC respondents were more positive concerning this aspect of service delivery. Part of the negative perception may be that some respondents, particularly non-HRDC respondents, may have misunderstood how and by whom case management was to be done (and therefore felt it was not being done). Nonetheless, all respondents identified weaknesses of the case management approach, including data systems problems (i.e., Contact IV) which make it difficult to properly manage clients and the labour intensity of the approach.
LMDA programs and services have not been well promoted in the community. Many clients felt ill-informed of the nature of the programs and program objectives prior to their participation and there was no clear understanding of all program options, both LMDA and non-LMDA, available to clients. Similarly, there was a perception that third-party delivery agents needed to be better informed of the range of programs and services available in the territory and that there was a need to promote programs among employers. Although some advertising is taking place, word of mouth was felt by some to be the most effective means of promoting EBSMs, especially since governments lack funds dedicated to their promotion. In contrast to the perceptions of clients who felt they were ill-informed about the nature of programs, third parties felt the information they provided to clients accessing EBSMs was adequate and included information on other programs and services.
Among respondents to the qualitative components of the evaluation, mixed perceptions exist concerning the availability and quality of Labour Market Information (LMI). While some respondents reported a wealth of LMI, others felt that it is not readily accessible and that what information does exist is of limited usefulness because it is not specific to the Yukon and is never or rarely community specific. Overall the results suggest low awareness and/or access to the LMI rather than a lack of such information.
Current monitoring and data collection systems were found to be inadequate. The sharing of client information is generally perceived to be insufficient to support client service and program development. Further, a comparison of administrative files used in the development of the participant survey sample frame indicated that the existing tracking and monitoring systems might be failing to capture the full universe of EBSM program participants.
A number of programs and characteristics of service delivery were perceived to contribute to clients assuming more responsibility for their employment situation. Programs were felt to be moving clients off income support but it was recognized that a number of other factors, such as the availability of jobs, must be present before a reduced reliance on income support can be achieved and that a number of barriers to self-reliance exist.
Over four fifths of survey respondents indicated that the programs have had at least a moderate positive impact on all the aspects of their employment situation about which they were asked. Participants were also more likely than comparison group members to rate the employment services they received as very important in helping them to obtain their current job. Further, the rates of employment among participants increased dramatically from the pre- to post-intervention periods. It is important to bear in mind, however, that too little time has elapsed for an accurate assessment of the impact of EBSMs on clients to be made. The incremental results will be pursued at the summative stage of the evaluation.
EBSMs may lead to several positive impacts in the community. These include the reduction of out-migration, the provision of services to the full range of clientele who require employment services and the development of small businesses.
Changes to the accountability targets for the Canada/Yukon LMDA suggest that these targets are flexible and sensitive to the changing conditions, realities and restraints of the territory. The accountability targets have fallen just short of being met. The targets for the number of clients served, the proportion of participants who were active Employment Insurance (EI) claimants, and the number of participants returning to work were all very close to being met. The target for unpaid benefits resulting from EI clients returning to work, however, was considerably short of being met.
A number of potential measurement problems were identified in the administrative data and methods used by Human Resources Development Canada (HRDC) to track and measure LMDA success. These included not all EBSM participants being in the administrative dataset, the underestimation of returns to work, and the overestimation of unpaid benefits. These problems cast doubt on existing measures of success based on these data and indicate the need for HRDC to re-assess its methods in this respect.
|
| TO: | The Joint Management Committee Canada/Yukon Labour Market Development Agreement |
| FROM: | Labour Market Information Sub Committee Canada/Yukon Labour Market Development Agreement |
| SUBJECT: | RESPONSE TO THE FORMATIVE EVALUATION OF EMPLOYMENT BENEFITS AND SUPPORT MEASURES UNDER TERMS OF THE CANADA/YUKON LABOUR MARKET DEVELOPMENT AGREEMENT |
The report documents of the Formative Evaluation of Employment Benefits and Support Measures (EBSM) Under Terms of the Canada/Yukon Labour Market Development Agreement (LMDA) were received from EKOS in December 2000. Derek Brackley, chair of the Canada/Yukon Evaluation Committee, formally presented the reports to the Joint Management Committee (JMC) during the January 2001 meeting. A sub-committee of the Labour Market Information (LMI) Committee was struck to review the reports and prepare a formal response.
The EKOS reports find "overall, the EBSMs were perceived to be functioning quite well...(however) various aspects of the implementation and delivery of the EBSMs require improvement." (p.i, Executive Summary, Final Overview Report). It is these necessary improvements that this response will address.
Six recommendations made by EKOS relate to the following:
It is important to note that many of the details included in the recommendations from the EKOS report had been identified in Dr. Marilyn Mohan's 1999 LMDA Consultations in the Communities, and ELEM's (Laurie MacFeeters) 1999 LMDA Consultations in Whitehorse. Responses based on those local consultations are underway.
It is acknowledged by the Labour Market Information (LMI) Committee that major initiatives will require resources not currently available. These activities will need to be funded as needs are identified.
When another evaluation is planned, an approach more tailored to the uniqueness of Yukon's demographics should be considered.
1. Relevance
2. Communications Within and Among Government Partners
Conduct more information meetings within government departments, with government staff and among partners.
3. Communications with Third Parties
4. Communications with the General Public
5. Service Delivery Practices
6. Monitoring and Data Collection
|
Under the terms of the Canada/Yukon Labour Market Development Agreement (LMDA), the federal and territorial governments agreed to jointly manage active labour market measures for the unemployed in the territory. This LMDA, like its counterparts with the provinces and the Northwest Territories, came into being as a result of the federal government's commitment, announced in November 1995, to devolve responsibility for labour market development to the provinces and territories. This commitment was made in response to the expressed desire by territorial and provincial governments to assume greater control over employment development programs. This report presents the results of a formative evaluation of employment benefits and support measures under the terms of the Canada/Yukon LMDA.
Subsequent to the announced commitment to devolve responsibilities, in May 1996 the federal government made formal offers to the provinces and territories to assume such responsibility. Then in July 1996, it enacted the Employment Insurance (EI) Act. In addition to bringing about changes to the unemployment insurance regime, the objectives of the EI Act were (and are) as follows:
The EI Act has two parts. Part I provides for passive income support (EI benefits) to those temporarily out of work. Part II provides for active employment benefits to enable unemployed persons to return to work. Under the Act, the administration of Part I benefits will be retained by the federal government, which will also be responsible for delivering active labour market programs that are national in scope, such as those concerned with interprovincial/territorial mobility or the operation of the labour exchange. On the other hand, under Part II, provincial and territorial governments are expected to take greater responsibility for the provision of active benefits and measures through agreements with the federal government, provided that these programs and measures are "similar" to those outlined in the Act and that they meet specified guidelines (see below). Provinces and territories that assume responsibility for these active measures will be provided appropriate funds from the EI account.
The Act outlines eight types of labour market programs (referred to as employment benefits and support measures) for EI clients, which are to serve as the models for corresponding territorial and provincial programs. They are as follows:
As well as being similar to these types of programs, territorial and provincial benefits and measures must meet certain principles and objectives. These principles and objectives are as follows:
The Canada/Yukon LMDA came into effect on January 24, 1998. It is a comprehensive agreement that specifies the details of the arrangement made by the federal and the Yukon governments regarding the management and delivery of employment benefits and support measures (EBSMs). Under the terms of the Agreement, Canada retains responsibility for the delivery of Employment Insurance (EI) benefits; for aspects of labour market development that are national in scope such as inter-jurisdictional mobility, sector councils and national labour market information and exchange; and for the financing and delivery of EBSMs and the National Employment Service (NES) in the Yukon. On the other hand, the Yukon will, under the Agreement, work with Canada to design, manage and evaluate the EBSMs through partnership arrangements between the governments and third parties. A Joint Management Committee (JMC), consisting of representatives of the Yukon and Canada, oversees the LMDA's implementation.
According to the LMDA, the arrangement between the Yukon and Canada is to take place according to principles that echo those set out in the EI Act, as outlined above. These include flexible and innovative approaches recognizing the association between economic and labour market development policies; enhancement of client service and efficiency; increased program harmonization and reduced program overlap; joint decision-making in terms of planning, design and targeting, and partnership among stakeholders and sharing of information; local flexibility; reduction of dependence on public income support; commitment by clients in terms of achieving goals, identifying needs, and contributing financially; equity principles; and implementing EBSMs within an accountability framework.
The principle of accountability is to be met by building a results-based evaluative framework into the LMDA to enable the setting, tracking and reporting of results targets. Success will ultimately be measured in terms of savings realized by government from reduced dependence on income support (EI and Social Assistance) owing to greater labour market attachment among the unemployed who use the EBSMs. The actual short-term performance targets, to be tracked in this and subsequent (including the summative) evaluations, have been set in 1998/99 as follows: (1) 759 EI eligible clients served by the EBSMs, 60 percent of whom are to be active claimants; (2) 318 EI eligible clients returned to work; and (3) $2,350,000 cost savings (unpaid benefits) from EI claimants returning to work before the end of their claim. It is recognized that these targets could be modified for subsequent years as conditions change.
The extent to which EBSM targets are met can be measured only in a preliminary manner in this formative evaluation. Only through evaluation over the longer term, based on data gathered over an extended period of time, can definitive assessment of results indicators be carried out.
Another important aspect of the LMDA and the EBSMs concerns the eligible clientele. Entrance criteria differ across benefits and measures, as the descriptions of the individual benefits and measures will show. In most cases, it is EI clients who are covered by the LMDA and who were included in this evaluation. According to the LMDA, an EI eligible client is defined as an unemployed person requesting assistance who (1) is an active EI claimant, or (2) had a benefit period that ended within the previous three years, or (3) had a benefit period which began within the last five years and was paid maternity/parental benefits, subsequently withdrew from the labour force and would like now to re-enter the labour force. The latter two groups are referred to as "reachback" clients.
The LMDA sets out the individual employment benefits and support measures themselves, of which Canada and the Yukon are jointly responsible for the design, management and evaluation. These benefits and measures mirror those outlined in the EI Act referred to above. Four employment benefits were the focus of the current evaluation: Targeted Wage Subsidies, Self-Employment, Job Creation Partnerships, and Training Purchases/Enhanced Feepayers. The support measures considered here are Local Labour Market Partnerships and Employment Assistance Services. Another employment benefit (Skills Development Employment Benefit) and a support measure (Research and Innovation) are briefly profiled, but were not included in the context of this evaluation. The total actual budget for these EBSMs was $3,135,000 in 1998/1999. The total proposed 1999/2000 budget for these EBSMs stands at $3,535,000, the same as that proposed for 1998/99 but an increase of $516,000 over 1997/98 budget proposals. A brief description of each EBSM follows.
i) Training Purchases and Enhanced Feepayers
The objective of this benefit was to encourage the unemployed to acquire skills through education and training for purposes of ending dependence on EI benefits and entering employment. Among the components of Training Purchases were seats in training programs offered by Yukon College or other approved training institutes that were purchased by the local Human Resource Centre of Canada (HRCC Whitehorse). Under this benefit, the HRCC or its partner paid for course costs, although in some cases some of the course costs were shared with the individual. Under a sunset clause agreement with the federal government, funding was not available as of March 1999 and courses that extended beyond June 30, 1999 would not be purchased. Training is now supported under the Skills Development initiative (see also Skills Development Employment Benefit), whereby EI eligible claimants request financial assistance while they are on such training. Support is negotiated based on needs, circumstances, and financial ability to pay for one's own courses.
ii) Skills Development Employment Benefit (SDEB)
This benefit replaces the Purchase of Training. It provides negotiated financial support to EI eligible clients taking training. Feepayers pay for their own training and are authorized to collect income benefits while taking training. Enhanced Feepayers negotiate financial assistance, which may include insurance benefits, tuition and other costs, while they are being trained.
iii) Job Creation Partnerships (JCP)
The objective of Job Creation Partnerships is to create meaningful work experience opportunities through temporary employment that will lead to long-term employment opportunities. Under this benefit, delivery agents and local community partners combine their efforts to create the jobs, with HRCC Whitehorse contributing up to 100 percent of the remuneration (the EI benefit or allowance payment). Eligible partners include businesses, not-for-profit organizations, and public health and educational institutions. These partners are responsible for making a material contribution to their project. Employers are encouraged to provide a top-up to meet the prevailing wage rate, and the maximum term of the project is 52 weeks.
iv) Self-Employment (SE)
This employment benefit aims to create self-employment by providing financial assistance for a fixed period to individuals with sound business ideas to develop their business plans. A third party contracts with an HRCC to deliver the benefit. Contractors may be businesses, local community organizations, individuals, public health and educational institutions and non-federal governments. These contractors are charged with evaluating business plan proposals, assessing client suitability, recommending Self-Employment proposals to the HRCC, providing participants with business plan development and implementation assistance, providing technical and related advice, and directing clients to other supports where appropriate. Agencies are reimbursed for administrative and support costs. Clients of this employment benefit will continue to receive EI benefits up to 52 weeks, and benefits may be "topped-up" to a locally determined flat rate.
v) Targeted Wage Subsidies (TWS)
The objective of Targeted Wage Subsidies is to help workers who are at risk of long-term unemployment, lack experience or face other barriers to employment to find a job and gain work experience. Workers with disabilities, for example, or young people with limited work experience are able to benefit from on-the-job work experience. This is done by providing a temporary wage subsidy to eligible employers to encourage them to offer jobs to such individuals. Potential employers are evaluated on the quality of the work experience offered and the likelihood of the job continuing. The subsidy, which does not need to be paid at a steady rate during a subsidy period of typically 26-30 weeks, normally does not exceed 60 percent of the total wages paid to the individual and can be used for up to one year and a half for disabled clients as needed. Eligible employers include businesses, non-profit organizations, specified municipal and provincial or territorial governments, and public and educational institutions.
vi) Employment Assistance Services (EAS)
This is the Part II measure that supports contracts with third parties such as private sector organizations or community groups to help unemployed individuals obtain and maintain employment. The types of activities covered under Employment Assistance Services include employability skills enhancement, career counselling, self-assessment workshops and job search assistance. Two examples of LMDA programs delivered under EAS are the Job Finding Club and Starting Points. Starting Points is a self-assessment service whereby participants develop an employability action plan that identifies employment strengths and barriers and is available to all residents, but has been used primarily by those facing lifestyle, educational, or demographic employment barriers. Available free of charge to all unemployed Yukoners, the Job Finding Club has monthly three-week sessions that offer job-finding strategies and services such as interview preparation and résumé preparation.
EAS agreements are arranged with sponsors based on community consultations and client needs. Businesses, non-profit organizations, band/tribal councils, and public and education institutions may qualify as eligible sponsors. While agreements can last up to 52 weeks and may be renewed, individual participation tends to be short in duration (i.e., one day to three weeks).
vii) Local Labour Market Partnerships (LLMP)
Under this support measure, Human Resource Centres of Canada and local delivery agents may form partnerships to develop strategies that encourage people to return to work. This measure can be used to assist those persons subjected to a mass layoff or closure so that they can quickly become re-employed. The LLMP can facilitate partnership development, leadership development, labour market research, and communications and promotion. Eligible sponsors (businesses/employers, not-for-profit organizations, non-federal governments, bands/tribal councils, public health and educational institutions, and/or individuals or groups of people) may receive financial assistance for overhead costs through contribution agreements.
Partnerships imply joint responsibility and accountability, and can be negotiated, with an annual renewal, to operate for up to three years. The LLMP includes the HRDC program that had been known as Industrial Adjustment Services.
viii) Research and Innovation
This measure considers national-level projects that, among other objectives, target applied research with the potential for large-scale distribution. Activities and experimentation are conducted to address labour market development, policy, and design issues. This support measure is delivered nationally and is not examined in this evaluation.
The objectives of the evaluation of the LMDA in the Yukon are five-fold:
Given that the immediate focus is on the formative stage of the evaluation (i.e., early on in the process), the objectives were focused on the shorter-term outcomes of the implementation of the LMDA. In broad terms, the purpose of the formative evaluation is to supply information indicating what improvements, if any, are required to LMDA design, delivery and infrastructure that would permit it to better meet its objectives. The formative evaluation also, however, provides a foundation for the future summative evaluation by collecting baseline information (for example, early outcomes, profile information) on participants and non-participants. In addition, the evaluation examined the extent to which current management information systems are sufficient for conducting the summative evaluation, which will provide information on the longer-term impacts of the programs and respond to questions regarding their cost-effectiveness.
In this section, we briefly describe the methodologies employed for the formative evaluation. Multiple methods were used to capitalize on the strengths of each EBSM. Specifically, quantitative methods (i.e., surveys) were used to obtain early results, which could be applied to conclusions about the full population of participants in EBSMs. Qualitative methods (i.e., key informant interviews, focus groups and a case study) as well as a literature review, were used to gather particular in-depth perspectives of the research issues. The research for the formative evaluation was carried out in two phases.
i) Phase I of the Formative Evaluation
The primary work of Phase I of the formative evaluation consisted of conducting a series of six key informant interviews; developing a matrix of evaluation issues, questions, indicators, sources and methods; and completing the literature review. A key output of Phase I was an implementation plan for Phase II. The key informant interviews were designed to obtain background, contextual information and informed opinion that would contribute to the design of the implementation strategy for Phase II.1 Interviews were held with representatives of HRDC and the Yukon Departments of Advanced Education and Health and Social Services. All respondents were members of at least one of the three LMDA committees: Management, Labour Market Information, and Evaluation.
ii) Phase II
A total of four focus groups were held: two with EBSM participants and one each with employers and stakeholders.2 The recruitment of employers and stakeholders relied on lists provided by the Joint Evaluation Committee. Clients of the EBSMs were asked during the survey whether they would be interested in participating in a focus group and the lists of interested survey respondents were used to recruit focus group participants.
The stakeholder focus group was held in Whitehorse on December 6, 1999. A total of six persons attended the group and represented a wide range of stakeholders, including program providers and administrators as well as those focused on First Nation needs and disability needs. All were third-party contractors.
The employer focus group was held with employers who had utilized an LMDA program in the past year. The group was held in Whitehorse on December 6, 1999. Four employers participated in the group and each had been involved with the Ready Aim Hire program.
Three groups were recruited for clients; a total of two groups were held and 10 clients participated. One group was held on December 12 in Whitehorse and the second was held on January 20, 2000. The clients' experience with HRDC programs ranged dramatically. Several had quite a detailed experience level, and had participated in multiple programs (Starting Points, Ready Aim Hire, Career Planning, courses at Yukon College); while others had only participated in Starting Points; and one had only started but not completed Starting Points.
Interview respondents were identified with the assistance of the Joint Evaluation Committee. Interviews with respondents based in Whitehorse were conducted in person and by telephone.3 Respondents from other locations were interviewed by telephone. Feedback on the evaluation questions was solicited from two different respondent groups through the key informant interviews, as follows:
Case Study
One case study was conducted of a specific EBSM: LLMP. The case study included five interviews with program managers and staff involved in the planning, implementation and delivery of LLMPs as well as major community partners.
Surveys of Participants and Comparison Group
A single participant data file was developed from ten separate participant files from various sources and together, these files contained 829 unique LMDA participant cases for individuals who participated in LMDA employment programs and services at any time between January 24, 1998 and August 1, 1999. This file was then merged with five administrative data files from which additional participant information was extracted. In addition, a data file containing SAR information was merged with the existing participant data file; too few participants had received only SA, however, to warrant a comparison group of people who had only received SA. Following the removal of all cases without valid phone numbers, start and end dates for EI benefits, and start and end dates for most recent interventions, the final data file consisted of 769 individuals. The distribution of participants by program type is presented in Table 1.1, which also presents the response rate to the survey.
The comparison group claimant sample was drawn from a file of EI claims that were active in 1998 and dormant EI claims (i.e., for individuals who were eligible for EI but who had not activated their claim) from 1994 to 1998 which was provided by HRDC. This produced a file of 18,525 claimants and reachbacks4 from which to draw the comparison group sample. To develop a comparison group for claimants, the comparison group data file was matched to the population of participant claimants on the basis of the period of time during which they participated and the amount of time from the start of a claim to when the participant's intervention came to an end. The comparison group sample for reachbacks was developed separately. Near, mid and far reachback comparison group cases were also sampled to match the population of participant reachbacks on the basis of the period of time during which the participants participated.5 The final comparison group sample frame consisted of 1,320 cases from a population of 18,525.
The participant survey yielded an overall response rate of 51.2 percent and a refusal rate of 6.6 percent. The comparison group survey yielded a response rate of 36.86 percent and a refusal rate of 9.4 percent.
To capture a profile of the Yukon work force, we turn to a discussion of the territory's labour market. This section provides an overview of the labour force participation and unemployment. Next, we discuss occupations and earnings, and an occupational outlook for the Yukon is presented. Finally, an overview of the importance of seasonality is included, with a link to EI claims in the Yukon.
| EAS | TWS | SE | Training | JCP | Enhanced Feepayers | Total Participant | Comparison Group | |
| Initial sample | 352 | 21 | 85 | 124 | 80 | 107 | 769 | 1,320 |
| (less) Unused sample | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| (less)Attrition Number not in service Duplicate number Quota Filled No knowledge of topic/ineligible Language barrier (not English/ French) | 104 | 5 | 16 | 35 | 29 | 26 | 215 | 461 |
| 7 | 0 | 0 | 0 | 0 | 0 | 7 | 0 | |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 17 | |
| 9 | 0 | 1 | 0 | 0 | 4 | 14 | 23 | |
| 1 | 0 | 0 | 0 | 0 | 1 | 2 | 2 | |
| Functional sample | 231 | 16 | 68 | 89 | 51 | 76 | 531 | 817 |
| Other number retired (not due to attrition) No answer/busy (<16 calls before end of fieldwork) Unavailable for duration of survey Retired/called 16+ times Other/illness | 109 | 0 | 3 | 1 | 0 | 0 | 113 | 383 |
| 3 | 1 | 3 | 3 | 2 | 0 | 12 | 6 | |
| 0 | 5 | 12 | 28 | 9 | 15 | 69 | 16 | |
| 16 | 1 | 4 | 2 | 5 | 2 | 30 | 34 | |
| Non-response Refusal Incomplete refusal 1 | 8 | 1 | 5 | 5 | 2 | 7 | 28 | 73 |
| 2 | 1 | 0 | 1 | 1 | 2 | 7 | 4 | |
| Total non-response | 10 | 2 | 5 | 6 | 3 | 9 | 35 | 77 |
| Total completed | 93 | 7 | 41 | 49 | 32 | 50 | 272 | 301 |
| Refusal rate | 3.5% | 12.5% | 7.4% | 6.7% | 5.9% | 11.8% | 6.6% | 9.4% |
| Response rate | 40.3% | 43.8% | 48.2% | 55.1% | 62.7% | 65.8% | 51.2% | 36.8% |
| Margin of error |
 8.9% |
32.7% |
11.6% |
11.0% |
14.4% |
10.3% |
4.9% |
5.6% |
| 1. Incomplete refusals refer to interviews during which the respondent refuses to continue part way through. | ||||||||
The Yukon's labour force participation rate7 is higher than the national rate, but so is its unemployment rate. In 1996, over eight in ten (81.4 percent) members of the Yukon working age population (15 years and over) were in the labour force, compared to under two thirds (65.5 percent) in Canada. Yukon residents' labour force participation and employment rates are higher across all key age brackets: youth (ages 15-24), prime working-age (25-54), and pre-retirement cohorts (55-64). Participation and employment rates particularly are higher for women, especially those in the latter age cohort.8 Also of note is that the participation rate in the formal economy is lower and the unemployment rate up to three times higher among First Nations residents.9
Reflecting somewhat the larger relative labour force in the Yukon and increased seasonal employment, while national unemployment levels declined to 8.6 percent in January 1999 (1.1 percent lower than January 1998), the Yukon unemployment rate increased to 14.7 percent (3.4 percent higher). Canada's unemployment rate declined steadily over 1998. During this time, however, the Yukon rate was more volatile, experiencing a peak of 17.5 percent in April, declining each month to a low of 8.9 percent in September, and subsequently increasing monthly to the 14.7 percent rate. The unemployment rate averaged 13.1 percent in 1999.
Since the implementation of new Employment Insurance (EI) legislation, the number of EI claimants has generally declined throughout Canada since 1995. In the Yukon, however, these numbers increased from 2,004 claimants in 1995 to 2,127 claimants in 1997. About three quarters of claimants in the Yukon tend to be between 25 and 49 years of age and about 60 percent of claimants are male. Approximately 90 percent of claimants spend between zero and 29 weeks on claim (the majority of whom spend between one and 29 weeks). As is the case in other parts of Canada, the number of claims increases in the winter months and is at its lowest in the late summer.
With respect to the nature of jobs that Yukon residents occupy, first, about one third of workers in the Yukon are employed by government. Second, Yukon private sector employment is concentrated in small businesses; about nine in 10 private sector workers are employed in the over 2,000 businesses with fewer than 15 people. In fact, only "a handful" of private sector firms employ more than a hundred people.10 Third, in 1996, one in eight Yukon labour force members were self-employed, which matches the national self-employment rate. Fourth, an important aspect of the Yukon labour market is its seasonality, with rapid expansions of employment during the summer months (by both residents and nonresidents) in response to increased business in the resource-based and tourism industries during those months. Finally, less than one in five (19 percent) employed labour force members in the Yukon worked on a mostly part-time basis (less than 30 weekly hours) during 1995, compared to 23.4 percent of the overall Canadian employed labour force. This is reflected in the fact that Yukoners were more likely to work longer weeks than the average Canadian (41.8 mean weekly hours versus 38.8 for Canada overall).
The longer workweeks partly explain why Yukon workers earn more from their jobs than Canadians do overall. In 1995, mean annual earnings were $28,511 in the Yukon compared to the national average of $26,474. However, even after controlling for hours worked, Yukoners still earned more: mean full-time, full-year earnings were $42,786 compared to $37,556 nationally. One reason for the difference is the relatively high minimum wage in the territory ($7.20 an hour, the highest in the country). Another is the need to compensate workers for working in remote locations, which explains why earnings tend to be higher in areas outside Whitehorse and in resource-extraction industries.
Future employment opportunities, according to the Yukon WorkFutures Employment Forecast, tend to be across occupations requiring lower educational attainment. Occupations such as cleaners and janitors, retail salespersons, retail trade managers, and secretaries, where grade ten or completed high school is required, are those that are expected to be most in demand in the near future. Employment percentage growth, however, is concentrated in the accountants and auditors category — where completion of a post-secondary program is typically a prerequisite for formal employment — as well as in the carpenters and cabinetmakers profession that requires less formal education. Self-employment occupations are concentrated among retail trade managers and food service/accommodation managers. Industry job growth projections, on the other hand, point to government as the employment growth leader, followed by trade, mining, and hotels and restaurants. Low job growth potential is expected in finance and primary industries.
Seasonality is both a major issue and a fact of life in the territory. Geography, climate, and convenience/production cycles are all elements of seasonality as it relates to employment. These factors are all very important when assisting seasonal workers, who by their nature are more dependent on EI than non-seasonal workers. Dependency on income assistance is particularly high in years when a resource is depleted or the weather is particularly inclement.
Of the five major sectors of the Yukon economy, three industries are seasonal to some degree: mining, construction (road and building), and in particular tourism. These tend to be private sector industries that peak in the summer when employable workers are in demand and training is limited by time constraints. Seasonality may increase the importance of self-employment as seasonal workers attempt to supplement their income with their own earnings. The peak seasons' importance is accentuated by the potential of extremely cold weather in the early spring and early autumn, which may limit many operations' ability to extend their services to the so-called "shoulder" seasons.
The effects of seasonal employment extend beyond statistics as communities adjust to the onset of autumn. Residents may hold more than one job or work in more than one field to build sustainable levels of income.11 Businesses that rely on seasonal industries may lay off workers or close altogether. Seasonal workers may search for employment outside of the Yukon. While we might expect that professions in construction, mining and tourism would be subjected to shifts as the harsh winter draws near, even people working in writing, translation, and public relations are more likely than the average Yukoner to be employed part-year. The impact of seasonality inevitably extends to employment insurance claims; fully 75 percent of Whitehorse claims, for example, are received in the fall and winter.12 With much of the Yukon labour force unemployed for six months of the year,13 seasonal employment is both a fact of life in the territory and a significant economic concern.
An assessment of the problems associated with the seasonality issue should consider the 1995 findings of the Working Group on Seasonal Work and Unemployment Insurance.14 The Working Group found that concerns about seasonal employment as it relates to Employment Insurance include the mindset of workers aiming to achieve the minimum days of work in order to collect EI income benefits, the frequency of job rotation which EI has helped encourage, employers not investing in equipment because of a lack of productivity time to get a return on the investment, and an absence of stakeholders thinking of markets outside of their borders. One of the primary findings of the Working Group is that seasonal workers require skills upgrading, basic education, and reorienting towards emerging industries, particularly as they tend to have less education than do non-seasonal employees.
The Working Group heard that there tends to be skepticism associated with government promoting skills upgrading. The Working Group expressed concerns that EI claimants may receive training that is not strategically linked to employment opportunities,15 indicating that training must be linked to employment opportunities of the future. Concerns also emerged regarding the ability of training to improve seasonal workers' employment prospects, and it was recognized that it is not realistic to expect these workers to find off-season employment where jobs do not exist. They recommended that training may be viable where opportunities do exist, such as by having employers view seasonal workers as permanent employees, smoothing seasonal peak demands where there is a large enough labour pool, and developing partnership options where employees move between/within firms.
Despite these suggested measures, the Working Group determined that only a more diversified industry base will lead to increased employment opportunities. There is also particular concern where training programs produce an over-supply of skilled labour in communities. As such, it was suggested that local upcoming job needs should be determined through research and only the number needed should be trained.
This report presents the results of the formative evaluation of EBSMs under the terms of the Canada/Yukon LMDA. The following six chapters present results organized by the major evaluation issues:
The final chapter contains the conclusions of the evaluation.
|
In this chapter, we present evidence concerning the relevance of Employment Benefits and Support Measures (EBSMs) in terms of their consistency with the Employment Insurance (EI) Act, the Labour Market Development Agreement (LMDA) and Yukon Territorial Government (YTG) priorities, their relevance to the needs of clients, employers and the community and their reach and accessibility. Evidence of the relevance of the EBSMs is drawn primarily from key informant interviews, focus groups and survey findings.
Among those key informants who were aware of the LMDA, LMDA programs and services were perceived to be consistent with the EI legislation. A number of third-party respondents remarked that small modifications have been made to programs in order to better meet the needs of the Yukon and that these changes have contributed to a sense of consistency of programs with the priorities of the YTG. These modifications primarily involved methods of dealing with small-market size issues and adjustments to meet the needs of specific community groups, and are discussed in more detail in Section 3.3(d).
Overall, third-party providers, clients and employers felt the programs and services were meeting their respective needs. Aspects of the service delivery approach which were felt to be particularly relevant to the needs of the clients included the negotiation of financial assistance to clients (i.e., based on client need, fosters more client involvement thus enhanced commitment) and services that address all aspects of clients' employment situations (i.e., from soft skills such as communication and pointers on appearance to job search skills and matching). Among government respondents, program strengths were thought to include flexibility in program eligibility, staff expertise in setting up programs and dealing with available resources, a good connection with the community, and more job coaching available to clients.
One aspect of the service delivery that some government respondents perceived to be a strength was the friendlier atmosphere at Human Resources Development Canada (HRDC), although this view contradicts that held by clients and other government respondents who reported poor client relations at HRDC (for example, lack of one-to-one counselling). Clients perceived the service delivery from federal government facilities to be "rough" and "cold" because of the environment at the Human Resource Centre of Canada (HRCC) (i.e., roped-off lines to stand in), the heavy reliance on computers (i.e., some clients are intimidated and uncomfortable using computers) and the perceived lack of ability and/or preparedness of staff to deal with clients' emotional needs. As well, some third-party respondents felt that these factors have led to less personalized services and may discourage clients from participating. It is important to note, however, that one-on-one services to clients are contracted to third parties, a fact that may account for this general perception of service delivery from the HRCC.
The reliance on third parties to deliver services was also felt to increase the relevance of programs for clients because these organizations have closer connections to the community and have an understanding of local needs; and clients may be less intimidated by their smaller scale and more personal, hands-on approach. Nonetheless, stakeholders and third parties noted several factors which may impact on program relevance and effective service delivery, including:
Employers felt the relevance of programs was limited by the fact that many participants leave following the end of programming (i.e., employers were looking for long-term employees) and the cost and time associated with bookkeeping for the programs. In addition, a number of employers felt that the programs were less relevant because most programs take place in winter which is a slower time for most employers. It is important to note, however, that programs would be less beneficial to clients if they were delivered during peak seasons, as clients require assistance the most during the off-season when jobs are scarce. One area in which employers felt the programs were working well for them had to do with the quality of the workers. In general, employers felt the relevance of programs could be improved if they could hire participants during peak seasons, if HRDC could respond faster as to whether funding is available to cover an employee, if the period of funding for job placement programs (for example, Targeted Wage Subsidies [TWS]) were made longer, if more clients were eligible for programs and services, and additional funding was available to help employers keep staff through maternity or sick leave.
In terms of their relevance to the community overall, government respondents generally felt that the mix of programs and services provided under the LMDA was fairly consistent with the local community labour market and economic development plans. Recent community consultations have led to recommendations on what programs and services are required most by the community. Thus, respondents noted that it is common for less successful programs to be modified, more successful programs to receive more funding and for some programs to be dropped or redirected if the market cannot support them. Examples of recommendations made on the basis of the community input include ensuring that appropriate budgets were in place to support TWS, the addition of a computer training component to Starting Points programs, components to address life skills issues in all EBSMs, increased support for Local Labour Market Partnerships (LLMPs) and new Employment Assistance Services (EAS) projects to address community-specific needs.
The survey results provide an indication of the degree to which different groups within the population are able to access the LMDA programs. A review of survey findings shows that the demographic composition of EBSM participants is very similar to that of the comparison group, suggesting that the EBSMs are being made available to all groups that require services.16 In fact, the survey findings suggest that Aboriginal and disabled clients account for a higher proportion of program participants than comparison group members (i.e., LMDA eligible but not participating). Similarly, a number of key informants felt that all equity groups were being served and reported that a number of programs exist to meet the needs of different equity groups, including the Challenge program (for persons with disabilities), Starting Points program (which some felt is more accessible to First Nations groups and is also delivered in French), and a proposed Older Worker Project.
Qualitative evidence nonetheless suggests that some gaps in the client groups served, as well as in the types of programs and services offered, are perceived to exist. The perceived gaps in client groups served include persons with disabilities, seasonal workers, older respondents, non-Whitehorse residents and First Nations. In addition, lower labour market attachment among some equity groups may mean that they have lower rates of Employment Insurance eligibility and are thus less likely to be eligible for EBSM programs and services. It is likely that some of these gaps, however, have more to do with the range or volume of services specifically designed for these client groups than with the availability of programming, as programs do exist to meet the needs of some of these client groups (for example, Challenge Program for clients with disabilities, Older Worker Project, etc.). It is also important to note that some of the perceived gaps, such as non-EI eligible clients and, in particular, previously self-employed, under-employed, the employed wanting to quit for training and those returning to the workforce (i.e., homemakers), may go beyond the scope of the LMDA. It is important to acknowledge that the eligibility for programs and services under the LMDA is restricted by the EI Legislation; thus commentary about gaps for non-EI eligible individuals may speak more to general labour market issues and needs rather than specifically to the design of the LMDA programming.
Perceived gaps in the types of programs and services provided included basic high school education and basic life skills training, as well as too much emphasis placed on trades-related training and not enough on other job skills such as office work and professional services. As well, some third-party deliverers felt that, although programs and services within Whitehorse were available in a timely fashion, this was not the case in outlying communities, where some programs (for example, Job Finding Clubs (JFC), resume writing, etc.) might be delivered only once or twice a year, or not at all. Market size and resource issues are likely the source of difficulties in trying to deliver programming in outlying communities, although government key informants noted that efforts will likely be made to address the needs of clients in these locations (for example, through the EAS program as well as a YTG initiative to increase access to technology in all communities). It is also possible that poor marketing and promotion of programs and services may have contributed to the perception of gaps because clients are simply unaware of the range of services offered.
The needs of clients, employers and communities were perceived to have been well met by LMDA programs and services, and the use of third-party delivery agents and community consultations were thought to contribute to the relevance of the EBSMs. In some instances, however, a lack of understanding of the role played by EBSMs in the range of employment programming in the territory (for example, increased use of third-party providers in service delivery, targeting EI clients and need for programming in the off-seasons) may have contributed to perceptions that EBSMs were less relevant in some areas (for example, HRCC not responding to clients' emotional needs, program eligibility, timing of wage subsidies). Nonetheless, a number of factors were identified that may decrease the relevance of programs and services, including gaps in services provided and clients served, aspects of the design and delivery of programs (for example, emphasis on volume of clients served, definitions of success as simply employment), the need for coordination of program delivery, the need for more partnering (for example, with First Nations), communications problems (for example, information sharing between government, third parties and stakeholders), and resource issues (for example, need for training, supports for partnership). Qualitative evidence also suggests that some gaps in the client groups served, as well as in the types of programs and services offered, are perceived to exist.
|
In this chapter, we review evidence related to the design and delivery of Employment Benefits and Support Measures (EBSMs). Specific issues addressed in this chapter include partnership, duplication and overlap, satisfaction with services, use of resources and services, bilingual services, flexibility, use of third-party delivery agents, client targeting, and case management. The discussion of the design and delivery of EBSMs draws on evidence from the key informant interviews, focus groups and surveys.
The evaluation reviewed the extent to which EBSMs have been designed and delivered in partnership both among government departments, as well as among government, employers and the community (i.e., third-party delivery agents and stakeholders). For the purpose of this discussion, it is important to operationally define the concept of partnerships. Partnership can be defined as a relationship involving close cooperation among parties having specified and joint rights and responsibilities and in which there is:
Many respondents to the evaluation perceived partnerships with third-party providers to exemplify a community partnership arrangement. This view may have merit as the distinction between third parties and community partners is often blurred in the territory, primarily because many community organizations are often also contracted by the government to deliver services under the Labour Market Development Agreement (LMDA). An examination of the criteria of partnering suggests that third-party contracts do represent some form of partnership in that third parties are involved voluntarily; have shared authority for the delivery of programs and responsibility to provide accountability information; and invest materials, expertise and information that may not have been available otherwise. Nonetheless, evidence pertaining to the relationship between government and third parties is presented in the discussion of the role of third parties in the delivery of services in Section 3.3(e).
Qualitative evidence revealed that while respondents occupying more senior management levels perceive partnerships to be strong and useful, respondents in less senior positions perceive partnership to be much weaker. Among the latter group, the least cooperation is perceived to occur between Health and Social Services (H&SS) and the other two partners. A primary weakness of the partnering process is felt to be poor communication of the results of partnering efforts (i.e., the regular management meetings) to staff within the respective governments so that staff could act on these results. Nonetheless, all respondents, regardless of position, have seen improvements over the last year in terms of partnership among levels of government (i.e., improved communication and the feeling of working more as a team) and much evidence of partnering (for example, agreements between HRDC and H&SS to divide the costs of training for Social Assistance Recipients [SAR] reachback clients, Yukon Territorial Government [YTG's] micro-loan to assist under-employed Yukon residents to access the Self-Employment program, the Labour Market Sub-Committee, etc.).
Weaknesses in partnering were perceived to exist with respect to client flows (i.e., lack of adequate information about other available services, duplication and overlap of information gathered on clients and repeat use of training programs by some clients) and the process of funding programs and services (i.e., reporting on funding is done after departmental decisions about funding have already been made and the three departments have little input into the budgets of the other organizations). Further, while some respondents felt that joint planning is working well among the three LMDA partners, others note there is not enough reporting and that the results of joint planning exercises are not constructed as a team. Another drawback of joint planning was felt to be the timeliness of the approval process as it is sometimes difficult for the Joint Management Committee (JMC) to meet.
Barriers to effective partnering among the three primary partners (HRDC, Advanced Education, H&SS) primarily involved differing priorities that interfere with communication and the ability to modify programs to increase their relevance for communities, as well a lack of appropriate supports for staff involved in partnerships (for example, administrative resources). Suggestions to improve partnering primarily involved methods to increase information sharing, including more formal meetings, more communication between management and front-line staff, meetings among front-line staff from the three departments (HRDC, H&SS and Advanced Education), better methods of sharing client information, more consultation with service providers, and a better match between federal and YTG agendas.
The partnership process with community agencies was generally thought to be working well, although some respondents felt that this was not necessarily due to the LMDA. The types of partnership initiatives that were perceived to be occurring included projects or communication with Yukon College, service providers and employers; community consultations such as the Labour Market Sub-Committee and Mana Research consultations;18 and internal consultations done directly with the community by government departments and service providers. Community consultations engendered a feeling of partnership among members of the community; feedback from community representatives formed the basis of a number of adjustments to the planning and implementation of the EBSMs; and the consultation process allowed participants to learn what other organizations were doing in the community. One recommendation from these consultations was that members of the community should come together more often to address their common needs and H&SS has volunteered to organize these networking forums.
Employers also perceived their relationship with HRDC to be quite positive although generally they have only limited contact with HRDC staff. Some employers felt HRDC worked at cross-purposes with them because they felt HRDC did not trust them to keep program participants on after the funding (for example, employers felt distrust as well as pressure from HRDC to keep employees on) and because HRDC provides employers with funding for only 20 weeks. Some suggested that more communication and follow-up from HRDC would help both clients and employers to succeed in the program.
Third-party key informants reported that a strong informal network for referring clients to other delivery agents has evolved among delivery agents under the LMDA. Third parties also cooperate to deliver services, such as public resource stations, although ownership issues sometimes detract from the spirit of cooperation as some delivery agents will wish to have ownership of projects in order to take credit for them when it is time to account for resource expenditures.
One focus of the evaluation was to determine the extent to which duplication and overlap was occurring in the delivery of EBSMs to clients. For the purposes of the discussion of overlap and duplication within the context of EBSMs, overlap refers to common services delivered or clients served by two or more agencies or programs but where the full range of services or clients is not entirely similar. Duplication refers to services provided and clients served by another agency or program that are virtually identical to that provided or served by an EBSM.
With respect to the programs and services themselves, no examples of duplication were provided. Nonetheless, third parties and government respondents provided a number of examples of perceived overlaps, including similar programming and services for youth and First Nations' groups (i.e., multiple funding sources), although it should be noted that First Nations and youth programming does not fall within Part II funding and are more accurately portrayed as issues related to "Pan-Canadian" programming than the LMDA; similar client groups targeted by the Department of Indian Affairs and Northern Development and H&SS, but different administration styles and eligibility requirements; the content of programs and services offered through Starting Points and H&SS (for example, basic job skills, resume writing, interview skills, etc.); and services provided through Job Finding Clubs and Starting Points (for example, resume writing).19
Focus groups with clients revealed that clients generally did not use services from more than one government department (i.e., HRDC or H&SS) although clients expressed some confusion about who delivers what as well as around the options available to them. Consistent with the qualitative evidence, when survey respondents were asked if they had received services from an organization that had not yet been discussed (i.e., other than self-serve resources or in-person assistance), only 16 percent indicated that they had. A number of third-party respondents felt that the use of multiple interventions was often beneficial for clients, particularly when the interventions are provided in an appropriate order (i.e., address most basic employment needs first). These benefits included reinforcement of learned information as well as variety with respect to the presentation of information. Survey evidence provided support for these qualitative findings although the preliminary nature of the survey findings, given the short period of time that has elapsed since the interventions, suggests that long-term impacts on employment outcomes will only be observed at the summative stage. Among the positive outcomes that were observed for these participants was the finding that respondents who had used multiple interventions were more likely to have developed a return-to-work action plan with an employment counsellor and were more likely to be employed in year-round jobs in their current or most recent position.
EBSM participant survey respondents were asked to rate their satisfaction with various aspects of the programs and services they received. As shown in Exhibit 3.1, overall, satisfaction ratings were very encouraging for all aspects of the services received by participants. Respondents were satisfied (responded with a 5, 6 or 7 on a 7-point scale) with all aspects of the programs and services they received, notably the quality of the education or training (90 percent), the level of knowledge of the staff (86 percent) and the quality of the job with their employer (83 percent). Respondents were slightly less likely to be satisfied with the resources or services they used on their own (79 percent), how quickly they were able to receive the programs and services they needed (78 percent), the quality of such services, as career counselling and job search (73 percent) and the information on the types of programs and services available (73 percent).
Consistent with the survey findings, findings from the focus groups with clients revealed high levels of satisfaction with the programs in which they were involved. Clients' explanations of their satisfaction with programs concerned both the focus or content of the programs (i.e., helped create resumes, understand on-the-job behaviour and gave them direction), as well as characteristics of service delivery (for example, no waits to get in, staff were well prepared). Some focus group respondents suggested that their high levels of satisfaction may in part be attributable to their low expectations upon entering the programs, given their limited knowledge of the programs in advance of participation. It is also possible that high satisfaction is related to the fact that, by the time of the evaluation, clients had had a chance to experience the positive benefits of any learning that took place through the programs.
LMDA program participants who responded to the survey were asked to indicate what resources or services they used on their own to help them get back to work. Respondents were most likely to indicate using newspaper advertisements (73 percent) and job bank kiosks or job board listings at a Human Resource Centre of Canada (HRCC) (60 percent) on their own. Respondents were least likely to have used word-of-mouth or their friends (five percent). Respondents were most likely to have accessed resources or services on their own at an HRCC (70 percent) or on the Internet (32 percent) (Exhibit 3.2).
When asked about the first place they received in-person assistance (i.e., from an HRCC, public library, outreach office or elsewhere), more than half of the respondents indicated that they had spoken with staff from the organization where they first received in-person assistance in order to help them plan their strategy to return to work (56 percent). Thirty-one percent of these respondents indicated having developed an action plan with an employment counsellor from the organization where they first received in-person assistance. When asked if they had completed the activities in their action plan, fully 87 percent of these respondents indicated that they had. The most common reasons for non-completion of action plan activities among the remaining 12 percent of these participants were that they changed direction and were no longer interested in pursuing the action plan (29 percent) or they found a job (15 percent).
An important evaluation issue involved whether LMDA programs and services are being delivered in both official languages. Evidence from the formative evaluation suggests that this requirement has been well met. All LMDA program participants surveyed for the evaluation were asked if they had been able to receive information about employment programs in the official language that they prefer and fully 97 percent of respondents indicated that they had been able to. Of the three percent (n=7) of respondents who had been unable to receive this information in their preferred official language, three had been unable to receive information in French and three had been unable to receive information in English (one respondent did not answer this question). The inability to receive information in English is most likely due to a lack of written material rather than verbal information.
Key informant interviews with government respondents suggest that services are never or seldom requested in French, although arrangements to provide services in French can be made if necessary.
Third-party deliverers generally felt that programs and services provided sufficient flexibility to respond to community needs. One third-party respondent noted that under the LMDA there are fewer regulations detailing what can and cannot be done. The requirement for much more reporting under the LMDA, however, may have hampered program flexibility because there is more paperwork and red tape, thus less time to be innovative in the delivery of services. As well, closer scrutiny of program delivery may mean less willingness among delivery agents to adapt programs to the needs of the community and one delivery agent felt this inflexibility may also apply to HRDC staff.
Some respondents (including government and third parties) felt that the EBSMs were very flexible under the guidelines and rules for their administration and that this flexibility was exemplified by Local Labour Market Partnerships (which can be delivered to meet the needs of any client group), as well as by the fact that funds are set aside in order to respond quickly to community needs (for example, the mine closure in Faro). Although a number of third-party respondents felt they have been able to adjust programs to meet community needs, it was also clear that some lack a thorough understanding of the rules and guidelines for program administration. For instance, one respondent was under the mistaken impression that it was inappropriate to refer EI eligible clients who wish to access services but who are not currently on claim to HRDC, rather than simply disqualifying them.
Government key informants generally felt that a major strength of the service delivery of EBSMs was the relationship between HRDC and third-party contractors and were quite happy with this aspect of service delivery. These respondents felt the use of third parties has been effective and has helped to ensure that programs are delivered well.
Third-party delivery agents provide a range of services to clients, including one-on-one and employment counselling, workshops, advertising employment, matching services, information kits, loans, training and job skills and business planning. As well, third parties participate in a management meeting held twice a year, and provide feedback to HRDC concerning the design and delivery of EBSMs. Delivery agents felt their most important contribution is hands-on personal service, such as directing people to needed services, motivating clients and building their self-confidence.
In addition to the services that they already provide, third-party delivery agents felt services would improve if they were able to pay clients to attend class, deliver home-school programs (for example, distance education)20 and access both personal and administrative information at the HRCC in Whitehorse (rather than Vancouver). As well, stakeholders mentioned the need for more sharing of information in order to properly manage clients and felt that, because clients do not always undergo a one-on-one interview with HRDC staff, they may not be aware of the various options available to them.
Among improvements to the service delivery process that have occurred since the LMDA was first implemented, third-party respondents reported improvement with respect to the communication of job postings and job opportunities (i.e., HRDC e-mails job postings to third parties) and communication among front-line staff. Positive changes were also felt to have occurred as a result of the LMDA itself. For instance, one third-party delivery agent noted that prior to the LMDA there was nothing to allow HRDC to contract out negotiated financial assistance or employment counselling services and that the ability to do so under the LMDA has allowed the third party to deliver a wider range of services. Another positive change was felt to be the greater partnership under the LMDA, such as with HRDC, YTG (in general), H&SS, Advanced Education, and Yukon College, which provides positive support to enable third parties to take on their additional responsibilities. In particular, these partnerships were thought to involve more parties in the delivery of EBSMs, which has led to less duplication of effort and a broader knowledge base from which to draw for the delivery of services (for example, knowledge of community needs and events, evaluating how the programs are run, etc.). Another advantage of the LMDA was that the approach to service delivery is more client-centered and thus better suited to their needs. Furthermore, given the (erroneous) perception among third parties that program budgets had been cut without a concurrent reduction in programming, they felt that programs were being administered more efficiently.
According to third parties, weaknesses of the service delivery structure included a lack of standard forms and software used by different departments and the fact that programs are not ongoing in the community. As well, third-party contractors felt their relationship with HRDC and H&SS could improve with respect to long-term planning and funding issues. One suggestion was to incorporate other non-LMDA programs that fall outside of Part II funding to reduce overlap, paperwork and program gaps. Another suggestion was for more partnering and networking between third parties and government so that delivery agents could provide feedback and recommendations on how programs could be modified to better meet the needs of clientele.
Most respondents (including government, third parties and stakeholders) felt there were administrative overlaps in the delivery of services to some client groups. Examples include:
Suggestions to reduce administrative overlap were focussed on better communication and coordination among all groups (i.e., government, service providers, employers and First Nations), such as the pooling of all program monies through a single agency, regular formal meetings with front-line staff and contractors, joint labour-management committees21 (which are already occurring) and the sharing of client assessment and background information with all relevant parties (it should be noted, however, that the sharing of client information is hampered by privacy issues).
Most government respondents did not feel that specific client groups are being targeted. Of the few that did, they felt targets were set for reasons other than identified needs and favoured groups such as persons with disabilities, seniors, youth and older clients.
One reason that targets may not be set for LMDA programs and services in the territory concerns the fact that the population of the Yukon may not be able to support targeting services to select client groups. As one respondent pointed out, there are so few clients in all potential target groups in different areas, with the exception of First Nations people, that the number of target group clients would be insufficient to justify delivering services to just the one group. As such, the need to target specific client groups may not exist in the Yukon as it does elsewhere in Canada.
Case management is a client-driven approach which tailors assistance to each individual's needs until an employment result is achieved. Case management supports the individual's goal of obtaining and keeping employment by negotiating individually tailored action plans, coordinating interventions in a partnership environment, and ensuring individual tracking, follow-up and support, when required, until an employment result is achieved.
Case management plays an important role in the delivery of EBSMs as it seeks to ensure optimum results from EBSM investments by determining the most appropriate mix of EBSMs, enabling service quality and affordability; demonstrating results in terms of employment and savings to the EI account; and providing data to support local planning, the improvement of interventions and the achievement of better results. As well, case management supports the principles which govern the delivery of EBSMs, such as client-driven delivery, where there is individual commitment and empowerment, and partnership, where different partners deliver interventions and services to a common client.
Non-HRDC government respondents generally felt that the case management approach was not working. They felt it was limited by an outdated computer system (Contact IV) and questioned whether most deliverers were able to keep up-to-date information. Among those aspects of the case management approach that were felt to interfere with its effectiveness, government respondents noted:
Respondents from H&SS and Advanced Education felt that, in general, case management is not being done. This perception is not consistent with the views of HRDC staff who felt that, with the exception of data systems issues and time constraints, case management is working well. Senior management with HRDC note that case management is done with all clients but that it may often involve only signing a release-of-information form; thus it is possible that this is not perceived to be case management by some staff. As well, non-EI eligible clients were not being monitored until recently.
Partnership . There has been improvement in partnership efforts since the LMDA was first implemented, although more partnering may be required among government partners, as well as among senior and less senior levels in terms of communication and information sharing. In the view of respondents to this evaluation, increased communication and consultation with partners would result in improvements to partnerships. The development of common priorities among the three partner departments, as well as the creation of appropriate supports for partnering activities, may also improve this aspect of the LMDA. A range of partnerships exists among government, employers and the community as well, including community consultations and informal networks among third-party delivery agents. Improved communication and information sharing were considered to be appropriate ways of ameliorating these partnerships.
Duplication and Overlap . Overall, LMDA programs and services tend to be somewhat complementary to other non-LMDA programming delivered in the territory. The primary area in which overlap was perceived to exist was in the types of services delivered through Starting Points and Job Finding Clubs. Less weight should be accorded to other examples of overlap as these concerned non-LMDA targeted client groups or, in other cases, dubious examples of overlap (i.e., similar client group targeted but different eligibility requirements).
Service Delivery . Due to the relevant content of programming and the convenience of services, most clients felt the delivery of services under the LMDA was good. Areas where service delivery was felt to require improvement, however, included the quality of services, such as counselling, job search advice, and information on the types of programs and services available. A large majority of survey respondents are accessing self-serve resources, primarily at an HRCC or through the Internet, and while the use of return-to-work action plans was only moderate, there was a high rate of completion of action plan activities.
Although a number of third-party deliverers felt programs were sufficiently flexible to adapt to the needs of the community when they are used as designed, it was also clear that some lack a thorough understanding of the rules and guidelines for program administration.
Third parties were perceived to be doing a good job. The lack of information sharing, however, was felt to be a barrier to allowing delivery agents to manage clients effectively. Nonetheless, a number of positive changes to service delivery were perceived to have occurred under the LMDA, including more autonomy among third parties, partnerships to support third-party service delivery and a more client-centered approach.
The lack of information sharing between governments and third parties has led to administrative overlaps by causing all parties to collect the same information for the same clients, delivery agents to provide similar assessment services, clients to complete similar forms each time they access services from different delivery agents, and employers to be approached by different service providers. Suggestions to reduce administrative overlap were focussed on better communication and coordination between all groups.
While non-HRDC government respondents generally felt that case management was not working, HRDC respondents were more positive concerning this aspect of service delivery. The contention by some non-HRDC respondents that case management does not occur or work across government departments suggests that some respondents may lack a clear understanding of how and by whom case management is supposed to be done. Respondents were more consistent in identifying weaknesses of the case management approach, including data systems problems (i.e., Contact IV) which make it difficult to properly manage clients, as well as the labour intensity of the approach.
|
In this chapter, we present findings related to communication and information issues for the Labour Market Development Agreement (LMDA). Specific topics addressed include the promotion of programs and services, labour market information and monitoring and data collection. Evidence for this discussion was derived from key informant interviews, focus groups and a review of administrative data files.
Overall, qualitative evidence concerning the promotion of Employment Benefits and Support Measures (EBSMs) shows that this aspect of service delivery is fairly weak. Some EBSM participants felt they were well informed through Human Resources Development Canada (HRDC) information sessions and by HRDC staff, while others felt ill-informed upon entering programs and only did so because they were told to, without really understanding the nature of their program participation or the objectives of the program itself. There was also general confusion among clients as to who provided what services and most agreed that they did not know how to investigate other options on their own.
Stakeholders and government officials echoed the views of participants and felt that clients lacked an overall understanding of the options available to them under the LMDA, as well as through non-LMDA programs and services. Similarly, government respondents felt third-party providers needed to be better informed of the range of programs and services provided by other agencies in order to enhance the overall complementarity of programs and services offered in the territory, although an informal network has already emerged among third-party deliverers. These respondents also noted a need to promote programs among employers in order to increase their awareness and rates of participation in programs. Employers themselves expressed very limited awareness of available programming and had heard of the programs in which they participated through HRDC recruitment phone calls or through word of mouth.
Among the methods currently used to promote programs, government and third-party respondents reported advertising on the premises of the Human Resource Centre of Canada (HRCC) and the offices of third-party agents and advertising on the radio and in newspapers. Several respondents felt that word of mouth is the most effective means of promoting EBSMs. Efforts are being made to improve the communication and promotion of EBSMs, as HRDC and its partners have met to develop ideas to improve the situation. One barrier to effective promotion is that no budget exists for government departments to promote programs that they are responsible for delivering (for example, Targeted Wage Subsidy [TWS].
Third parties felt that the information they provide to clients is quite adequate, and includes information on sources of funding for training and colleges, assistance to develop budgets, locating labour market information to research career occupations, face-to-face information about the rationale and logistics of programs (for example, how to apply/register, etc.), information and referrals to other programs being delivered in the community, pamphlets describing the services provided by other delivery agents, and verbal descriptions of how to fill out application forms (one respondent suggested that a step-by-step guide to assist clients through the entire application process would be most useful).
Qualitative evidence provided mixed feedback on the extent to which sufficient labour market information (LMI) was available to support initiatives to clients and employers. A number of third-party delivery agents and government officials felt that this aspect of service delivery was quite strong and reported a number of sources of LMI, including: phoning LMI officers at HRDC, Yukon and B.C. WorkFutures pamphlets, Community Profiles, information from the Yukon Territory Bureau of Statistics, word of mouth and local knowledge, relevant LMI resources carried by other third-party providers, occupation searches by phoning or visiting employers in industries of interest, and a wealth of LMI on the Internet (for example, Yukon Work Futures, Yukon Community Profiles, and the Yukon Bureau of Statistics website). Although electronic sources of LMI may be less accessible, they are more easily updated than hard copy (print versions of Yukon WorkFutures had not been updated as recently as the website because the cost of doing so made this activity unfeasible), most communities have electronic access; and a number of supports are in place to help interested users access and use these resources (for example, training for teachers, students and staff, seminars for government employees).
Other respondents felt that there is still room for improvement to be made in terms of labour market information. A number of clients felt that third parties delivering Starting Points did a much better job than HRDC staff of providing LMI, that labour market information (LMI) is not readily available, and that they have to dig for it, and that the available LMI is often not specific to the Yukon and is never community specific. The contradictory opinions expressed by respondents with respect to LMI suggest that the issue may have less to do with the availability of LMI and more to do with a need to promote these information sources and/or improve users' ability to access this information.
Stakeholders and third parties generally agreed that insufficient client information was shared in order to support client service and program development. The lack of such information has led to administrative duplication because all delivery agents must do assessments and reporting separately, and this lack has hampered the ability of third parties to deliver services to clients in an effective manner.
In addition, some evidence suggests that weaknesses exist in the collection of client information. As mentioned in the methodology section, 10 separate participant files were received for the development of the participant survey sample, six hard copy files and four electronic files, one of which was a spreadsheet from the strong.C./Yukon HRDC regional office that was extracted from the National Employment Service System (NESS). A review of the hard copy files revealed that two thirds of the hard copy cases did not appear in the HRDC spreadsheet and that few of the cases from the spreadsheet appeared in the Action Plan file. Together, these findings suggest that the existing tracking and monitoring systems may be failing to capture the full universe of EBSM program participants. As well, Phase I key informants noted that entry of participant and results data can be delayed by operational constraints; and since much of these data; depend on contractors for entry, this delay may lead to problems of data timeliness and reliability.
Finally, evidence from key informant interviews completed in Phase I of the evaluation also revealed that HRDC internal systems were adjusted in 1998 and these adjustments were thought to have the potential to affect comparisons between pre- and post-LMDA results.22
Promotion of Programs and Services . There has been inadequate promotion of LMDA programs in the community. The nature and objectives of the programs were unclear to many clients prior to their participation and there was no clear understanding of all program options available to clients. Third-party delivery agents also felt that they needed to be better informed of the range of programs and services available in the territory and that there was a need to promote programs among employers.
Labour Market Information . Respondents held mixed views on the extent to which labour market information (LMI) was available and on the quality of this information. While some respondents felt that a great deal of LMI was available, others considered the information to be inaccessible and of limited usefulness because it is not often specific to the Yukon or to communities. Overall the results suggest low awareness and/or access to LMI rather than a lack of such information.
Monitoring and Data Collection . Current monitoring and data collection systems were found to be inadequate. Respondents felt that there was insufficient sharing of client information to support client service and program development. As well, evidence gathered in the development of the participant survey sample frame indicated that the existing tracking and monitoring systems might be failing to capture the full universe of EBSM program participants.
|
This chapter presents evidence from the formative evaluation concerning the success of Employment Benefits and Support Measures (EBSMs) to date. It is important to bear in mind that the results presented in this chapter provide only an early glimpse of the success of EBSMs. As relatively little time has elapsed since the Labour Market Development Agreement (LMDA) was first implemented, it is likely that many of the impacts and successes that will ultimately be observed for the EBSMs have not yet begun to present themselves.
Additionally, measurements of program successes in this formative evaluation are obtained through a quasi-experimental design. Quasi-experimental designs provide an alternate means for examining causality in situations not conducive to experimental controls and where there may be few, if any, alternative approaches to examine the phenomenon of interest, such as the impact of program participation on employment behaviour and labour market attachment among participants in LMDA programs. While these designs were developed to control as many threats to validity as possible (for example, pre-existing differences between the two groups which compete with the variable of interest [i.e., program participation] to explain any observed differences, otherwise known as confounding factors), they lack at least one of the other two properties that characterize true experiments, randomization and a control group. Although the current evaluation does employ a comparison group, participants and comparison group members alike were not randomly assigned to their respective groups and as such, the inherent differences between members of these two groups (i.e., differences that prompted some to participate and others not to participate) present an alternative explanation to any observed program impacts. Although multivariate analyses can be used to simultaneously control for numerous potentially confounding factors, it is nonetheless early (i.e., too little time has elapsed since the LMDA was first implemented) to assess the incremental impacts of the EBSMs. As such, the descriptive findings related to program impacts are presented for illustrative purposes only. The calculation of incremental results will be pursued as part of the summative evaluation.
Third-party and government respondents suggested a number of programs and characteristics of service delivery that may contribute to clients assuming more responsibility for their employment situation, including the Self-Employment program (possibly because it provides very definite timelines for the client to succeed), training and certification, basic education, vocational life skills, job skills, post-training follow-up and assistance, the provision of advice, creation of self-worth, the availability of Labour Market Information (LMI) and client involvement in the assistance. Another third-party respondent felt that, on the whole, the simple acquisition of additional employment-related skills, be it job search or specific training, is likely to decrease clients' reliance on income support. As well, government respondents felt that programs are moving clients off income support, but that a number of other factors are needed in order for a reduced reliance on income support to be realized, including the availability of jobs that pay higher than income support amounts, interventions that reach clients earlier on in their employment history, more coordination between programs, more community partners, more personalized contact, more focus on motivation and long-term guidance.
Barriers that were perceived to exist and that may impede client self-reliance included a lack of daycare funding and funding for other non-employment related needs (for example, rent, food, etc.), the loss of disability benefits or social assistance (SA) when clients participate in the EBSMs, the inaccessibility of services outside of Whitehorse, the need for lifestyle changes (i.e., adapting to a lifestyle that includes employment, scheduling, etc.) and clients' lack of ability to plan ahead in order to meet labour market demands. Another barrier to self-sustainability involves the general mindset of individuals who have come to rely on income assistance (EI or SA), as some clients tend to remain on EI until their claim has run out and look for work only when they have nothing to lose by quitting a new job. Finally, some training deliverers felt the rate of completion of training courses was much higher when delivery agents could pay clients for attending classes as if they were salaried employees.
The use of Negotiated Financial Assistance (NFA) was felt by some third-party deliverers to increase the commitment of participants to their particular program because the amount of assistance is based on individual client need and the client is more active in the development of their employment plan. NFA was thought to contribute to clients taking primary responsibility for identifying their employment needs and sharing in the cost of assistance but was felt to have less of an impact on reducing reliance on income support (for example, EI or SA) or increasing employment because environmental factors, such as high unemployment, were felt to have a stronger influence on these outcomes. One respondent felt that NFA does not pose a barrier to training because clients who are unable to contribute anything to the cost of training may receive the full amount of their training.
Qualitative evidence from third parties provided mixed results in terms of the extent to which clients are required to take responsibility for preparing a return-to-work action plan. While some respondents indicated that everyone was required to have an action plan, others noted that only a moderate proportion of clients developed action plans. Of those clients who develop action plans, survey and qualitative evidence suggests that the large majority (between 71 and 80 percent of program participants) follow the plan through to its end, a finding that some key informants felt may in part be attributable to the lack of jobs in the Yukon. As well, those third-party agents that required clients to develop an action plan indicated that they followed-up on the plans with clients and some also have detailed protocols for following up with clients involved in training.
Overall, the survey results show that roughly four fifths of survey respondents feel that the EBSMs have had at least a moderate positive impact on each of the impacts measured. The EBSMs were most likely to have been perceived to have had a large positive impact (responded with a 6 or 7 on a 7-point scale) on respondents' self-esteem (55 percent), satisfaction with their career path (50 percent) and job search skills (47 percent). Respondents were less likely to feel the EBSMs had a positive impact on their job satisfaction (42 percent) and ability to make appropriate career choices (40 percent) (Exhibit 5.1).
Focus groups with clients revealed that many felt the true success of the programs was in life-skills development and personal growth. The programs provided a meaningful purpose for clients and a support network with individuals in similar circumstances. On-the-job experience, training and job simulation programs were felt to contribute the most to the success of interventions. Job placement programs were felt to be least effective when the job placement was for too short a period (i.e., 20 weeks).
Survey respondents were asked to rate how important the employment services they received were in helping them to get their current or most recent job. As shown in Exhibit 5.2, 35 percent of participant survey respondents who were employed in the post-program period felt that the program was very important in this respect (responded with a 6 or 7 on a 7-point scale). Twenty four percent rated the program as somewhat important (responded with a 3 to 5 on a 7-point scale) and 41 percent rated the assistance as not important (responded with a 1 or 2 on a 7-point scale) in helping them to get a job. Employment Assistance Services (EAS) and Training Purchases participants were the most likely to feel that the employment program was very important in helping them find a job (39 and 36 percent, respectively), whereas Self-Employment and Job Creation Partnerships (JCP) participants were the least likely to feel this way (19 and 17 percent, respectively).
It should be noted that this question was addressed only to those respondents who had a job in the post-intervention period and to respondents who were not continuers (i.e., Self-Employment participants who did not continue operating their businesses or Wage Subsidies or Employment Partnership participants who were not hired on by the host employer in the post-program period). If it is assumed that these continuers (i.e., those whose business/program jobs continued beyond the program end-date) would have responded "very important" to this question, the rated importance of the employment services rises dramatically among Self-Employed (19 to 74 percent) and JCP participants (17 to 65 percent).
While participant ratings of the importance of their employment program are somewhat modest, a comparison of participant and comparison group survey responses reveals that, overall, participants were somewhat more likely than comparison group respondents to rate the employment services they received as very important in helping them to get a job (Exhibit 5.3). This implies an advantage for participants in LMDA programs relative to non-participants.
Rates of Employment
Table 5.1 presents various employment outcomes for participant and comparison group survey respondents. Overall, these results suggest that some advantage may exist for participant claimants compared to comparison group claimants.
The first panel of Table 5.1 shows that a large majority of both the participant and comparison groups indicated that they have had a job since the end of the program or reference date. The incidence of employment at some time in the post-program period was high for all program respondents and ranged from a low of 78 percent of Employment Assistance Services (EAS) participants to a high of 100 percent of Targeted Wage Subsidies (TWS) respondents. Participant claimants were more likely than comparison group claimants to indicate that they have been employed at some time in the post-program period (86 and 76 percent, respectively), although no difference in employment rates for reachbacks was observed between these two groups.
A review of employment rates according to respondents' socio-demographic profiles (not shown) shows that the incidence of employment was higher among non-equity group respondents.
Another way in which the impacts of LMDA programs were measured involved a comparison of employment status in the week prior to the intervention or reference date and employment status at the time of the survey. Generally, these findings show positive shifts in employment from the pre- to post-intervention periods for all EBSMs and larger shifts for participants relative to the comparison group.
As shown in Table 5.2, there was a larger overall increase in employment from the pre- to post-program periods among participants relative to the comparison group. It is important to note, however, that the smaller observed increases in employment rates among the comparison group are at least partly attributable to a ceiling effect caused by their higher rates of pre-reference date employment. Unemployment rates (ninth row of Table 5.2) declined for both participants overall and comparison group claimants at the time of the survey but the former exhibited considerably larger declines in unemployment rates (67 to 24 percent for claimants and 67 to 26 percent for reachbacks) while comparison group claimants showed a more moderate decline in their rates of unemployment (46 to 39 percent). In contrast to the other groups, the rate of unemployment among comparison group reachbacks rose slightly from the pre- to post-program period (12 to 18 percent), although the rate of unemployment in the post-program period was much lower among these respondents relative to the other groups.
| Participants by Claim Status | Comp. Group by Claim Status | |||||||
| EI Claimant | Reachback | EI Claimant | Reachback | |||||
| Pre | Post | Pre | Post | Pre | Post | Pre | Post | |
| Employed | 22 | 60 | 17 | 63 | 41 | 53 | 73 | 70 |
| Self-employed | 2 | 11 | 1 | 12 | 4 | 6 | 6 | 6 |
| Full-time year-round | 6 | 27 | 3 | 29 | 10 | 24 | 41 | 52 |
| Part-time year-round | 3 | 7 | 6 | 12 | 4 | 6 | 4 | 5 |
| Full-time seasonal | 8 | 5 | 1 | 1 | 9 | 5 | 9 | 3 |
| Part-time seasonal | 1 | 1 | 1 | 2 | 5 | 3 | 2 | 1 |
| Contract | 0 | 5 | 0 | 1 | 4 | 2 | 6 | 3 |
| Casual | 2 | 4 | 5 | 6 | 5 | 7 | 5 | 0 |
| Unemployed and looking | 67 | 24 | 67 | 26 | 46 | 38 | 12 | 18 |
| Student/in school | 2 | 10 | 0 | 6 | 1 | 2 | 5 | 2 |
| Out of Labour Force | 5 | 5 | 11 | 3 | 13 | 5 | 9 | 10 |
| DK/NR | 3 | 2 | 5 | 0 | 0 | 0 | 2 | 0 |
| Unweighted n | 152 | 78 | 162 | 139 | ||||
|
Source: Canada/Yukon LMDA Participant and Comparison Group Surveys |
||||||||
Positive shifts in employment were observed for all of the programs under study as well (sum of rows 1 to 7 in Table 5.3). All groups at least doubled their overall employment rate and the largest positive shifts in overall employment occurred for Self-Employment (SE) (19 to 81 percent) and TWS participants (23 to 100 percent). For all EBSMs except Self-Employment, the largest increase in employment was observed with respect to full-time, year-round jobs. For Self-Employment participants, the largest increase was observed with respect to the proportion of respondents reporting that they were self-employed. Unemployment was also much lower at the time of the survey, dropping dramatically for all EBSMs (ninth row of Table 5.3).
In this section, we present key evaluation findings as they relate to each of the EBSMs. Most of the findings presented here relate to program impacts and are based on results from the participant survey.
The Targeted Wage Subsidies (TWS) program is designed for workers who are at risk of long-term unemployment, lack experience or face other barriers to employment, to help them to find a job and gain work experience. The program provides a wage subsidy to eligible employers to encourage them to offer jobs to such individuals.
TWS program respondents were most likely to indicate that they were interested (responded with a 6 or 7 on a 7-point scale) in entering the workforce on a full-time or part-time basis (100 percent) in the week prior to starting their first employment program, suggesting that the program was well matched to participants' interests. Over half (58 percent) of TWS respondents felt they would not have been able to obtain the job in which they were placed had the wage subsidy not been available.
The rate of program completion among TWS respondents was good, as only two TWS participants indicated leaving their wage subsidy jobs prior to the completion of the full period of funding (22 percent). Seventy-eight percent of TWS program respondents were retained by their program employer at the end of the program23 and 90 percent of these respondents indicated that they were hired into the same job as when their job was funded. Those who were hired at the end of the program were hired into full-time year-round jobs (82 percent) or full-time seasonal positions (18 percent). Seventy-four percent of respondents who were hired at the end of the TWS program felt that the employment services they received were very important (responded with a 6 or 7 on a 7-point scale) in helping them to get this job and none of the respondents indicated that the program was not important.
The Self-Employment benefit aims to create self-employment by providing financial assistance for a fixed period to individuals with sound business plans to start their businesses. A third party is partnered with the HRCC to deliver this benefit. The contractor provides a business planning workshop and an independent advisor for participants.
Pre-intervention interest ratings show that SE program respondents were placed into an appropriate program, as these respondents were most likely to indicate that they were very interested in starting their own business (81 percent) at that time. Fifty-two percent of Self-Employment program respondents felt they would have been unable to start their business if the Self-Employment program had not been available.
Over four fifths (81 percent) of SE participants took a training program or workshop to help them start their business, although roughly one quarter of SE participants indicated that their businesses had not yet started. The majority of Self-Employment businesses operate year-round (88 percent) and 36 percent of respondents whose businesses have started indicated that their businesses employed someone other than themselves. Self-Employment participants earned an average of $483 per week and only 17 percent worked at other jobs in addition to their business. Ninety-one percent of Self-Employment program respondents whose business had started indicated that it was still in operation at the time of the survey.
The objective of the Job Creation Partnerships (JCP) program is to create meaningful work experience opportunities through temporary employment that will lead to long-term employment opportunities. Under this benefit, delivery agents and local community partners combine their efforts to create jobs, with HRCC Whitehorse contributing up to 100 percent of the remuneration (the EI benefit). Employers are encouraged to provide a top-up to meet the prevailing wage rate, and the maximum term of the project is 20 weeks.
Job Creation Partnerships program respondents' high levels of pre-intervention interest in entering the workforce on a full-time or part-time basis (88 percent) suggests that the JCP program was the appropriate employment program for the majority of participants. Nearly two thirds (65 percent) of JCP participants felt they would not have been able to obtain their program job had the EI subsidy not been available at the same time.
In their JCP program jobs, respondents were most likely to indicate that they worked for a private employer (60 percent). Relatively few JCP program participants indicated leaving the program prior to the completion of the full period of funding (28 percent) and the majority (58 percent) were retained by their host employer at the end of the program.24 JCP respondents who were hired on were most likely to have been hired into full-time year-round jobs (45 percent), followed by part-time year-round (19 percent), full-time seasonal (18 percent) and part-time seasonal jobs (seven percent) (11 percent did not respond). Fifty-three percent of the respondents hired at the end of the program felt the employment services they received were very important (responded with a 6 or 7 on a 7-point scale) in helping them to get the job. Only 15 percent felt they were not important.
One key informant who was familiar with the JCP program felt that the reporting and monitoring requirements for the program have resulted in less flexibility and an adherence to the strict rules of program delivery to ensure that accountability requirements are met. The activities that take place under JCP include the Student Training and Employment Program (STEP), Job Shadowing and Co-op programs. A broad range of organizations were known to participate, including YTG, the federal government, crown corporations, private businesses and non-governmental organizations (NGOs).
It was felt that the JCP program contributed to supporting community non-profit infrastructure by the simple fact that clients are placed with NGOs and because these placements sometimes involve developing projects to support this infrastructure. The overall impact of the program on the community was felt to be quite positive as it contributes to increased employment, encourages people to stay in the community, creates a learning opportunity for clients that is directly related to their professional training, results in jobs and savings and has potential social impacts since participants are less likely to become disaffected and marginalized if they are able to gain employment. It may also contribute to long-term employment at the local level because clients develop a relationship with the participating employer and develop a skill set that is customized to the employer's needs.
Training Purchases are not an employment benefit under the LMDA but a now phased-out training mechanism. The objective of the Training Purchases benefit was to encourage the unemployed to acquire skills through education and training for the purposes of ending dependence on EI benefits and entering employment. Training is now supported under the Skills Development Employment Benefit which supports individuals who make their own training arrangements.
Training Purchases respondents were most likely to indicate that they were interested (responded with a 6 or 7 on a 7-point scale) in entering a training or education program (81 percent) and in entering the workforce on a full- or part-time basis (80 percent) in the week prior to starting their first intervention. Fully 79 percent of these respondents felt they would not have been able to enter the training program if financial assistance for course costs had not been available.
Respondents were most likely to indicate that the training programs they took prepared them for jobs in trades, transport and equipment operation (34 percent); business, finance and administration (17 percent); academic upgrading (12 percent); or computers and IT (12 percent). The majority of Training Purchases respondents reported that they received their training from a community college (85 percent) and most training programs were less than six months in duration (44 percent). Over two thirds (71 percent) of respondents indicated that they were required to contribute to the cost of their training and the average contribution was $792. The majority of Training Purchases participants either completed their training program (77 percent) or were still in the process of finishing their training at the time of the participant survey (9 percent).
The Enhanced Feepayers program (replaced by Negotiated Financial Assistance) provided negotiated income support to EI eligible clients taking training and was intended to encourage the unemployed to acquire skills through education and training. Feepayers paid for their own training and collected Employment Insurance benefits while taking training.
Enhanced Feepayers program respondents were most likely to indicate that they were very interested (responded with a 6 or 7 on a 7-point scale) in entering a training or education program (87 percent) in the week prior to starting their first intervention. A large majority of Enhanced Feepayers participants (84 percent) felt they would not have been able to enter their training program if they had not been eligible to collect EI while taking the course.
Respondents were most likely to indicate that the training programs they took prepared them for jobs in health-related fields (15 percent); computers and IT (13 percent); trades, transport and equipment operation (13 percent); and business, finance and administration (12 percent). Most received their training from a community college (78 percent). The average length of training programs was 33.4 weeks and the average number of hours per week that respondents spent in their programs was 32.4 hours. More than half (58 percent) of the Enhanced Feepayers respondents indicated that they had contributed to the cost of their training, with an average contribution of $3,575. A large majority of Enhanced Feepayers participants had completed their training program (82 percent) at the time of the participant survey.
The Employment Assistance Services (EAS) measure supports contracts with third parties, such as private sector organizations and community groups, to help unemployed individuals obtain and maintain employment. The types of activities covered under EAS include employability skills enhancement, career counselling, self-assessment workshops and job finding clubs.
In the week prior to starting their first intervention (within the time period under investigation in this study), Employment Assistance Services (EAS) program respondents were most likely to indicate that they were very interested (responded with a 6 or 7 on a 7-point scale) in entering the workforce on a full- or part-time basis (91 percent).
The most frequently mentioned types of resources that EAS participants reported using on their own were newspaper advertisements (80 percent) and job bank kiosks or postings (68 percent). EAS respondents were most likely to have accessed self-service resources from a Human Resource Centre of Canada (HRCC) (77 percent), the Internet (29 percent) or a public library (22 percent). At the first place they received in-person assistance (i.e., from an HRCC or elsewhere), more than two thirds (70 percent) of the EAS participants who responded to the survey reported that they had spoken with staff to help them plan their strategy to get back to work. Forty-one percent reported that they had developed a return-to-work action plan25 with an employment counsellor and 84 percent of those who had developed an action plan indicated that they completed the activities in their plan.
A range of activities take place under the Local Labour Market Partnerships (LLMP), including labour market research for specific areas or groups (through community consultations, employer surveys and other methods); the development of labour market recovery plans (for example, in the situation of the Faro mine closure); the development of long-term training strategies for industry sectors (such as tourism); workshops, training initiatives and projects to create on-going employment aimed at various client groups (including women, First Nations and disabled persons); and program reviews to increase the efficiency and effectiveness of other employment programs.
All case study respondents agreed that the LLMPs have met local labour force priorities. The primary reason respondents felt this way was that the majority of the projects are focused on improving employment in some form or another, be it directly through the creation of employment opportunities, or less directly through training, the development of labour market information (LMI) through research, or supporting initiatives that have as part of their objectives the advancement of employment among certain groups. Labour force priorities are identified through a number of methods. In fact, the focus of the LLMPs themselves is often on identifying labour force priorities through primary and secondary labour market research.
All respondents felt that partnerships are a crucial ingredient for the success of LLMP projects and felt that the LLMPs have been quite successful in encouraging partnership. The partners involved in LLMPs represent almost every potential stakeholder in the territory and include employers, non-governmental organizations (NGOs), communities, community organizations, industry associations, sector councils, educational institutions (colleges and high schools) and tribal bands.
While most respondents felt that partners understood the concept behind LLMPs, they nonetheless felt that there was confusion concerning LLMPs and a need for promotion to increase awareness of the program. One respondent remarked that most partners are not even aware of the LLMP and that for many the program is simply a source of funding. Respondents reported that partners were generally satisfied with the processes to establish cost sharing, responsibilities and deadlines because they were involved in their development.
Respondents from community organizations were generally happy with the application process to obtain LLMP funding. All community respondents were satisfied with the support they received from HRCC staff, although they noted that HRDC may not have the necessary human resources to turn applications around in a timely fashion or to be proactive in terms of identifying projects that could be implemented to address community needs.
The impacts of the LLMPs on the community are diverse and include increased employment, improved employability due to enhanced skills, knowledge and mobility, a more highly skilled workforce for employers, direction and guidance for the development of labour market strategies, the receipt of funding for new projects to address LLMP identified needs, and improved partnership. Among the few weaknesses of LLMPs that were mentioned is low awareness of the program among stakeholders; and a few suggested there were insufficient human resources to support delivery of the program in a proactive fashion. Program strengths were felt to be its flexibility in the types of projects it can support, including projects that would not be supported through other funding sources, and the focus on community and partnerships. Interestingly, flexibility may also be a weakness of the program as one respondent noted that there might be a tendency for some organizations to use the program for things that go beyond the program's intended objectives.
Evidence gathered through the formative evaluation suggests that EBSMs may have a number of impacts on the community. To begin, analysis of survey data shows that the Self-Employment program may lead to increased employment among participants and has contributed to the development of small business. Thus, businesses started through this program contribute to the economic health of the communities in which they are found. EBSMs may also benefit communities by reducing out-migration. A review of the administrative data revealed that a higher proportion of comparison group members than participants leave the Yukon, presumably in search of employment, suggesting that programs may contribute to clients staying in their communities (although it may also be that those most able to find employment elsewhere leave, while those with few options or skills remain behind and thus are available to participate in the EBSMs). Nonetheless, qualitative evidence suggests many programs and services are unavailable outside of Whitehorse and that clients may need to relocate in order to access these services. Finally, survey evidence suggests that Aboriginal and disabled clients account for a higher proportion of program participants than comparison group members (i.e., LMDA eligible clients). As such, EBSMs appear to be reaching all members of the community that require these programs and services.
A number of programs and characteristics of service delivery were perceived to contribute to clients assuming more responsibility for their employment situation. Programs were felt to be moving clients off income support but it was recognized that a number of other factors, such as the availability of jobs, must be present before a reduced reliance on income support can be achieved and that a number of barriers to self-reliance exist.
Over four-fifths of survey respondents indicated that the programs have had at least a moderate positive impact on all aspects of their employment situation about which they were asked. Participants were also more likely than comparison group members to rate the employment services they received as very important in helping them to obtain their current job. Further, the rates of employment among participants increased dramatically from the pre- to the post-intervention periods. It is important to bear in mind, however, that too little time has elapsed for an accurate assessment of the impact of Employment Benefits and Support Measures (EBSMs) on clients to be made. These incremental results will be pursued at the summative stage of the evaluation.
Communities may benefit from EBSMs in a number of ways. The reduction of out-migration, the provision of services to the full range of clientele that require employment services and the development of small businesses are all examples of how EBSMs can have positive effects on the community.
|
In this chapter we present qualitative evidence of the perceived efficiency and cost-effectiveness of Labour Market Development Agreement (LMDA) programs and services, as well as objective measures of program efficiency based on a review of survey and administrative data related to the costs and number of successes for each of the different Employment Benefits and Support Measures (EBSMs).
Respondents to the qualitative components of the evaluation had few suggestions of how the efficiency and cost-effectiveness of LMDA programs and services could be improved. One government respondent, however, felt that although most third-party deliverers are good, the delivery of some generic services by some third-party agents may not be the most cost-effective means of delivering services because so little competition exists among delivery agents.
The relative cost of the various EBSMs was explored by analyzing administrative and survey data in conjunction with overall budget information for each EBSM to derive the average cost per intervention. The average cost per intervention was calculated by dividing the total actual26 budget used for each EBSM in the 1998-99 fiscal year by the total number of interventions in the population of participants that occurred within this fiscal year.27
As shown in Table 6.1, the average cost per intervention was highest for Training Purchases clients ($5,676), a figure which is far above the average cost per intervention for the Enhanced Feepayers program that replaced Training Purchases ($1,491) and which also had the lowest cost per intervention. This finding is consistent with expectations, as the primary distinction between Training Purchases and Enhanced Feepayers is that the latter program provides "negotiated financial assistance" to clients which usually results in a larger financial contribution to the cost of training on the part of the client. Another finding of note was the relatively high cost per intervention of the Targeted Wage Subsidy (TWS)28 and Self-Employment (SE) programs compared to the Job Creation Partnerships (JCP) program ($4,906 and $4,534 versus $2,802, respectively). Once again, this finding is consistent with expectations based on the funding structure of the programs (TWS and SE programs are typically longer in duration).
Table 6.1 also shows the number of successes for each program on each of three measures of employment success: ever employed in the post-program period, employed 12 consecutive weeks in the post-program period, and employed at the time of the survey. These measures of success are presented in order from the least difficult to achieve (ever employed) to the most difficult (employed at time of survey); for most EBSMs (with the exception of TWS), the number of successes decreases with the difficulty of the employment outcome measure. Nonetheless, even with the most stringent measure of success, the majority of participants in all EBSMs were successful.
It should be noted that these analyses are restricted in so far as they provide only short-term information on the number of employment successes. The nature of some interventions (for example, Enhanced Feepayers, Training Purchases, etc.) is such that we would expect outcomes to manifest themselves over the longer term if the notion holds that participants are obtaining skills that will help their careers over time. In the case of Enhanced Feepayers, this program was already observed to be the least costly intervention and it is likely that the effectiveness of this intervention will be manifested over the longer-term. Thus, while these analyses provide a glimpse of program cost-effectiveness as it stands now, these results can be expected to change at the summative stage of the evaluation.
|
The Canada/Yukon Labour Market Development Agreement (LMDA), like all provincial and territorial labour market development agreements, prescribes the use of primary measures to monitor short-term LMDA success in three primary measure areas: number of Employment Insurance (EI) clients served by LMDA interventions and the percentage who are active claimants, participants returning to work, and unpaid EI benefits of participants returning to work. The Agreement further states that results targets are to be set annually for these measures and adjusted to reflect target attainment and changing economic conditions during the previous fiscal year.
It would appear that these targets are in fact sensitive to changing conditions, realities and restraints in the Yukon. Between 1997/98 and 1998/99, the targets set for the number of EI client participants served by the LMDA and the proportion who are active EI claimants were reduced, while the targets for the number of returns to work and the amount of unpaid EI benefits were increased. The flexibility in the setting of targets is a positive development, adhering to a principle laid out in the Canada/Yukon LMDA.
Two concerns were raised with respect to the targets, however. First, they include Apprentices who are not "typical" EI clients returning to work and possibly generating unpaid EI benefits. Second, the targets are focused on Part I EI benefits, but ignore incidences where a participant may subsequently receive Part II benefits, which effectively reduces the estimated "savings" to the EI account. Exclusion of Apprentices in the targets and a consideration of both Parts I and II benefits are suggested.
As for target attainment, Table 7.1 shows that, based on the administrative data, there were 619 EI clients who participated in LMDA interventions in 1998/99, which is fairly close to the target for that year (759). As well, 57 percent of these participants were active EI claimants, which almost meets the target of 60 percent. Of those who participated, 283 returned to work, which again is close to the target (318). However, in 1998/99 there were only $1.31 million in unpaid EI benefits resulting from EI clients returning to work before the end of their claim. This is well short of the target of $2.35 million.
| Primary Measure | All EBSMs* | Five EBSMs** | ||
| Targets | Results | HRIB | "Alternative" Methods and Measures | |
| Number of EI clients served by LMDA | 759 | 619 | 520 | 802 |
| Percentage of LMDA participants who are active EI claimants | 60% | 57% | 65% | 47% |
| Number of EI clients returning to work (RTW) | 318 | 283 | 287 | 463-548 |
| Unpaid EI benefits from active EI claimants returning to work *** | $2,350,000 | $1,308,951 | $844,710 | $716,461 — $751,150 — |
| Notes: * Refers to targets and results for all active measures under the Canada/Yukon LMDA, including the EBSMs as well as Group Services and Apprenticeship. ** Covers a "results period" up to October 31, 1999. *** Figures are just for active EI claimants and exclude reachbacks and non-EI clients who are not being paid EI benefits and therefore can generate no unpaid benefits. |
||||
An important component of the evaluation work was the validation of the information systems and methods used to measure LMDA performance on the basis of the primary measures. The presence ofsound information systems is deemed important for accurate monitoring and measurement of program performance. This evaluation study uncovered a number of problems, consideration of which would contribute to the ongoing process of improving the data and the methods used to measure Employment Benefits and Support Measures (EBSM) performance.
One major concern raised was with regard to the participant count, the first primary measure (columns 3 and 4 of row 1 of Table 7.1). A major source of participant data is the third-party contractors' files. The analysis revealed potential data-entry problems, that almost one third of all EBSM participants were accounted for in these files and not in the Human Resources Investment Branch (HRIB) files. This would indicate a significant underestimation of the number of participants, which has implications for the ability of the information systems to measure performance. Moreover, accounting for contractor-only participants (row 2) drives down the active claimant share to a lower level (47 percent) than is apparent in the HRIB files (65 percent). At the same time our analysis also revealed that almost all those who were indicated as being participants in the files did self-identify as participants in the survey conducted for this evaluation.
Regarding the second primary measure, returns to work, two different investigations were conducted in this evaluation. First, consideration was given to the issue of the extent to which the performance picture presented annually to program managers based on the administrative data misrepresents actual performance of that year's interventions over the full course of the post-intervention period. Our work indicated that, assuming the administrative data were accurate, a manager using data available at the end of the year would obtain a fairly accurate picture of LMDA success for interventions started in that year. For example, in 1998/99 the number of actual returns to work for interventions started in that year would be under-estimated by only 10 percent using fiscal year-end data.
Second, concerns were raised in this analysis about return to work (RTW) capture and accuracy. Using self-reported employment results in the evaluation survey to corroborate the RTWs recorded in the administrative data indicated that a large proportion of actual returns to work were not in fact captured in the administrative data (row 3 of columns 3 and 4 of Table 7.1). The inaccuracy was particularly great for participants on the contractors' files who were not originally in HRIB's files. The fact that actual returns to work appear to be underestimated indicates that the return to work target was likely exceeded by an even greater amount than indicated above. More importantly, this finding raises concerns, once again, over the ability of the administrative data systems to measure LMDA performance. On the other hand, a large majority of the returns to work recorded on the file were accurate, according to the evaluation survey.
A number of potential causes were suggested for the underestimation of returns to work. Among them are the previously mentioned problem of not all participants being captured in the files, a failure to contact clients in a follow-up telephone call, reachbacks and active claimants who had not returned to work before the end of claim but who might have returned to work after their intervention, and a follow-up call question that may be subject to misinterpretation and inconsistency across the regions. As for active EI claimants, there may be measurement problems as well, such as participants who return to work but who fail to report doing so in order to "protect" remaining entitlement weeks, which leads to an under-estimation of returns to work.
Our analysis also revealed some overestimation of unpaid EI benefits (UB), the third primary measure. Using alternative measures explicitly accounting for EI benefits actually paid, it was shown in this analysis that unpaid benefits were overestimated by 10-15 percent (row 4 of columns 3 and 4 of Table 7.1).
Potential problems were identified in the way HRIB measures unpaid EI benefits, which could account for this overestimation. In its calculation of unpaid EI benefits, HRIB ignores benefits paid in the return to work period (12 consecutive weeks receiving 25 percent of entitled EI benefits), and considers lapsed EI benefits as unpaid. These factors could lead to an over-estimation of unpaid benefits by HRIB.
Finally, the total return to work performance of the LMDA was considered in this analysis. It was found that active EI claimant participants are more than twice as likely to return to work and the latter are four times as likely as non-EI clients (56 versus 27 versus six percent). The return to work rate of income assistance recipients (who are non- or reachback EI claimants) was only somewhat higher than the reachback rate. The return-to-work rate for Apprenticeship was higher than the five main EBSMs, though concerns were raised about the inclusion of Apprentices (who artificially raise the return to work count) in an exercise to measure RTW success for EI clients.
The purpose of this exercise was to point out potential shortcomings in HRIBs accountability process. In light of the possible sources of error in measuring participation and performance identified in this analysis and bearing in mind the need for HRIB to still deliver information in a timely and cost-effective fashion, it is suggested that there needs to be a re-assessment of the methods of gathering information on participants and the measurement of returns to work and unpaid EI benefits. In particular, there needs to be greater effort and scrutiny in the gathering of information on participants and greater stringency in the capture of returns to work in the follow-up procedure. Furthermore, to reduce possible errors in measuring unpaid EI benefits, it is suggested that consideration be given to more intensive examination of administrative data. Without good information systems and methods, accurate measures of LMDA success cannot be achieved.
|
Overall, the Employment Benefits and Support Measures (EBSMs) were perceived to be functioning quite well, although various aspects of the implementation and delivery of the EBSMs require improvement. In general, the areas for improvement concern communications, information sharing and resource issues. The main findings and conclusions from the formative evaluation are presented in the following sections.
LMDA programs and services were generally perceived to be consistent with Employment Insurance (EI) legislation and the priorities of the Yukon Territory and were felt to be meeting the needs of clients, employers and communities. In instances where programs and services were felt to be less relevant (for example, Human Resource Centre of Canada [HRCC] not responding to clients' emotional needs, program eligibility, timing of wage subsidies), these perceptions could often be traced to a lack of understanding of the role played by EBSMs in the range of employment programming in the territory (for example, increased use of third-party providers in service delivery, targeting Employment Insurance (EI) clients and need for programming in the off-seasons). Nonetheless, a number of factors were identified which may decrease the relevance of programs and services, including gaps in services provided and clients served, aspects of the design and delivery of programs (for example, emphasis on volume of clients served, definitions of success as simply employment), the need for coordination of program delivery; the need for more partnering (for example, with First Nations), communications problems (for example, information sharing between government, third parties and stakeholders), and resource issues (for example, need for training, supports for partnership).
Partnering efforts have shown improvement since the LMDA was first implemented. While partnering efforts among governments seem to be working at senior management levels, more attention to this may be required among government partners, as well as among senior and less senior levels in terms of communication and information sharing. Respondents to this evaluation feel that the greatest improvements in partnering would be realized through increased communication and consultation with partners, be it better reporting of the results of joint planning exercises and evaluations or more coordination of effort and consultation in the design, delivery and funding of programs and services. The development of common priorities among the three partner departments, as well as the creation of appropriate supports for partnering activities, may also improve this aspect of the LMDA.
A range of partnerships exists among government, employers and the community, including community consultations and informal networks among third-party delivery agents. Respondents generally felt that these relationships would also improve through increased communication and information sharing.
Overall, LMDA programs and services tend to be somewhat complementary to other non-LMDA programming delivered in the territory. Although no evidence of duplication was found, the primary area in which overlap was perceived to exist was in the types of services delivered through Starting Points and Job Finding Clubs (for example, resume writing, job search, etc.). Although some respondents felt similar client groups were targeted by different programs, these sources are not likely to represent major problems in that the duplication of programming was occurring with non-LMDA targeted client groups (similar programming for youth and First Nations) or, in other cases, there is some question about whether the example represented a true overlap in programming (i.e., similar client group targeted but different eligibility requirements). One form of duplication, the use of multiple interventions, was perceived to have positive consequences for participants.
From the perspective of clients, the delivery of services under the LMDA is thought to be good because of the relevant content of programming and the convenience of services (for example, no waits, staff well-prepared). Nonetheless, there is room for improvement of some aspects of service delivery, particularly the quality of services such as counselling and job search advice, and information on the types of programs and services available. Survey evidence also showed that a large majority of clients are accessing self-serve resources, and that this is typically occurring at an HRCC or through the Internet. The use of return-to-work action plans was only moderate, although there was a high rate of completion of action plan activities.
Respondents provided mixed feedback on the extent to which programs and services are sufficiently flexible to adapt to the needs of the community when they are used as designed. Although a number of third-party deliverers felt programs were sufficiently flexible, it was also clear that some lack a thorough understanding of the rules and guidelines for program administration.
The delivery of programs and services relies heavily on the use of third parties and they were perceived to be doing a good job. A primary obstacle to effective service delivery among third parties was the lack of information sharing to allow delivery agents to manage clients effectively. Positive changes to service delivery that have occurred under the LMDA include more autonomy among third parties, partnerships to support third-party service delivery and a more client-centered approach.
The lack of information sharing among governments and third parties has led to administrative inefficiencies by causing all parties to collect the same information for the same clients, delivery agents to provide similar assessment services, clients to complete similar forms each time they access services from different delivery agents, and employers to be approached by different service providers. Suggestions to reduce administrative inefficiencies were focussed on better communication and coordination among all groups.
Non-HRDC government respondents generally felt that case management was not working, although HRDC respondents were more positive concerning this aspect of service delivery. Part of the negative perception may be that some respondents, particularly non-HRDC respondents, may have misunderstood how and by whom case management was to be done (and therefore felt it was not being done). Nonetheless, all respondents identified weaknesses of the case management approach, including data systems problems (i.e., Contact IV) which make it difficult to properly manage clients and the labour intensity of the approach.
LMDA programs and services have not been well promoted in the community. Many clients felt ill-informed of the nature of the programs and program objectives prior to their participation and there was no clear understanding of the range of program options, both LMDA and non-LMDA, available to clients. Similarly, there was a perception that third-party delivery agents needed to be better informed of the range of programs and services available in the territory and that there was a need to promote programs among employers. Although some advertising is taking place, word of mouth was felt by some to be the most effective means of promoting EBSMs, especially since governments lack funds dedicated to the promotion of EBSMs. Information provided by third parties to clients accessing EBSMs was perceived by clients to be adequate and included information on other programs and services.
Among respondents to the qualitative components of the evaluation, mixed perceptions exist concerning the availability and quality of Labour Market Information (LMI). While some respondents reported a wealth of LMI, others felt that it is not readily accessible and that what information does exist is of limited usefulness because it is not specific to the Yukon and is never or rarely community specific. Overall the results suggest low awareness and/or access to the LMI rather than a lack of such information.
Current monitoring and data collection systems were found to be inadequate. The sharing of client information is generally perceived to be insufficient to support client service and program development. Further, a comparison of administrative files used in the development of the participant survey sample frame indicated that the existing tracking and monitoring systems might be failing to capture the full universe of EBSM program participants.
A number of programs and characteristics of service delivery were perceived to contribute to clients assuming more responsibility for their employment situation. Programs were felt to be moving clients off income support but it was recognized that a number of other factors, such as the availability of jobs, must be present before a reduced reliance on income support can be achieved and that a number of barriers to self-reliance exist.
Over four fifths of survey respondents indicated that the programs have had at least a moderate positive impact on all aspects of their employment situation about which they were asked. Participants were also more likely than comparison group members to rate the employment services they received as very important in helping them to obtain their current job. Further, the rates of employment among participants increased dramatically from the pre- to post-intervention periods. It is important to bear in mind, however, that too little time has elapsed for an accurate assessment of the impact of Employment Benefits and Support Measures (EBSMs) on clients to be made. These incremental results will be pursued at the summative stage of the evaluation.
EBSMs may lead to several positive impacts in the community. These include the reduction of out-migration, the provision of services to the full range of clientele that require employment services and the development of small businesses.
Changes to the accountability targets for the Canada/Yukon LMDA suggest that these targets are flexible and sensitive to the changing conditions, realities and restraints of the territory. The accountability targets have fallen just short of being met. The targets for the number of clients served, the proportion of participants who were active EI claimants, and the number of participants returning to work were all very close to being met. The target for unpaid benefits resulting from EI clients returning to work, however, was considerably short of being met.
A number of potential measurement problems were identified in the administrative data and methods used by HRDC to track and measure LMDA success. These included not all EBSMs of participants in the administrative dataset, the underestimation of returns to work and the overestimation of unpaid benefits. These problems cast doubt on existing measures of success based on these data and indicate the need for HRDC to re-assess its methods in this respect.
|
|
| Last Modified: 2002-11-29 | Important Notices |
![[To Top]](/web/20060126222803im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}