|
|
     |
||
 |
     |
 |
|
Formative Evaluation of the Final Report Evaluation and Data Development August 17, 1998 AH-SP-069E-08-98
I. Introduction The Employment Insurance (EI) Act, or Bill C-12, came into effect on July 1, 1996, replacing the Unemployment Insurance (UI) Act and the National Training Act. The EI legislation represents a restructuring of the old UI system. The new Act provides both income support and active measures designed to assist unemployed Canadians return to work as quickly as possible. Part I of the Act deals with changes to the income benefits and Part II outlines the employment benefits and support measures that are available to clients. A formative evaluation of the Employment Benefits and Support Measures (EBSM) was carried out in mid-1997. The broad objectives of this formative evaluation were the following:
The formative evaluation reviewed Targeted Wage Subsidies (TWS), Self-Employment (SE), Training Purchases and Skills Loans and Grants (SL&G;), and Job Creation Partnerships (JCP). Because the majority of regions were not using Skills Loans and Grants at the time of the evaluation, this benefit has only been examined to the extent that Negotiated Financial Assistance (NFA), a precursor to SL&G;, has been implemented. The support measures evaluated were Employment Assistance Services (EAS) and Local Labour Market Partnerships (LLMP). Interviewees were also asked to discuss issues related to the Transitional Jobs Fund (TJF) but could not provide much information on this component. The formative evaluation was conducted using the following methodologies:
It should be mentioned that with the exception of New Brunswick, the formative evaluation was taking place at a time when Labour Market Development Agreements (LMDAs) with the provinces/territories had yet to be implemented. Labour Market Development Agreements are agreements whereby provinces/territories could become responsible for the design and delivery of employment benefits and support measures. Following the signing of the LMDAs, Canada and the provinces will jointly develop an evaluation plan that will determine the need for and timing of formative and summative evaluations in each province. The summative evaluations will focus on the longer-term impacts of the interventions. For example, the summative evaluations may address the incremental impacts of EBSM and the achievement of targets in terms of employment and unpaid benefits. The summative evaluations will need to integrate data and information from provinces that will likely have different approaches to the delivery of active measures to EI clients. II. Findings The following findings address the evaluation issues developed by HRDC and refined by Price Waterhouse at the start of the evaluation. a) Implementation The findings related to implementation can be grouped into the following themes: local implementation and delivery; third-party service delivery; quality of services; communication; client selection and targeting and target setting process; cultural change; official languages; and systems. Local Implementation and Delivery
Third-Party Service Delivery
Quality of Services
Communication
Client Selection and Targeting and Target Setting Process
Cultural Change
Official Languages
Systems
b) Impacts and Effects on Individuals The findings related to the impacts and effects of the EBSM interventions on individuals have been grouped as follows: employability and employment; reduction of dependency on EI; and impacts on clients' attitudes. Employability and Employment
Reduction of Dependency on EI
Impacts on Clients' Attitudes
c) Primary Results Measurement One of the objectives of the formative evaluation was to examine the two primary results indicators to determine how close the operational estimates produced by the Human Resources Investment Branch (HRIB) were to actual client data found on individual Benefit and Overpayment (BNOP) files and through client survey responses. The intention is to provide information that will help HRIB to refine the methodology used to calculate the primary indicators and to improve data integrity. It is important to note that, in order to implement the operational indicators quickly and to provide data on a timely basis, the initial methodology designed by HRIB was relatively simple. As a result, some level of detail was lost and calculation rules did not take into account exceptions and individual anomalies. It is also important to note that the July 1996 to March 1997 period was the start-up of EBSM and, at that time, the accountability framework was under development and just in the process of being implemented. It is believed that several aspects of the methodology used to produce the HRIB calculations of the savings results are likely to lead to overestimates. Given the differences of the population covered by the formative evaluation and the universe covered by the HRIB populations, it is not possible to comment on the magnitude of the overestimate. The following factors are believed to have the potential to contribute to an overestimate in the savings results used by HRIB:
As explained above, it is not possible to quantify the magnitude of the overestimate given the inability to access some data and the differences in the populations studied. d) Impact on Communities These findings relate to the following: the extent to which local labour market and economic development plans were integrated during the implementation of EBSM; partnership issues; and whether EBSM met the needs of communities in areas other than just employment.
III. Conclusions This section provides a summary of the conclusions drawn and lessons learned from the formative evaluation of EBSM.
The purpose of this report is to present the findings of the research conducted for the formative evaluation of the Employment Benefits and Support Measures (EBSM). 1.1 Context and Background The Employment Insurance (EI) Act, or Bill C-12, came into effect on July 1, 1996, replacing the Unemployment Insurance (UI) Act and the National Training Act. The EI legislation represents a restructuring of the old UI system. The new Act provides both income support and active measures designed to assist unemployed Canadians return to work as quickly as possible. Part I of the Act deals with changes to the income benefits and Part II outlines the employment benefits and support measures that are available to clients. Access to personalized interventions is available to individuals who meet selection criteria identified by the local Human Resource Centres of Canada (HRCCs) and are prepared to participate. To be eligible for employment benefits and support measures, unemployed workers must be receiving EI benefits, have received UI benefits in the last three years, or have received maternity or parental benefits in the past five years. The benefits are targeted to those who need extra assistance and are prepared to make a personal commitment to a back-to-work action plan. 1.2 Main EBSM Evaluated The five main EBSM on which the formative evaluation focused are described below.
The evaluation also examined Local Labour Market Partnerships (LLMP) and the Transitional Jobs Fund (TJF), but not in as much detail as the EBSM interventions described above. LLMP is the vehicle through which Regions and local offices can experiment with approaches to improve the functioning of their labour markets and address local labour force priorities. The TJF provides financial support for activities that will promote sustainable economic activity and, in turn, the creation of sustainable employment. 1.3 Evaluation Objectives and Scope A formative evaluation of EBSM was carried out in mid-1997. A formative evaluation is typically conducted toward the beginning of a program to assess how it is doing and identify areas for modifications early on. The broad objectives of this formative evaluation were the following:
This formative evaluation focused on design, implementation and delivery issues as well as the early experiences of clients. The formative evaluation was conducted using the following methodologies:
A Methodology Report detailing our approach to the formative evaluation was submitted on May 30, 1997. It should be mentioned that the formative evaluation was taking place at a time when Labour Market Development Agreements (LMDAs) with the provinces/territories had yet to be implemented. Labour Market Development Agreements are agreements whereby provinces could become responsible for the design and delivery of employment benefits and support measures. Following the signing of the LMDAs, Canada and the provinces will jointly develop an evaluation plan that will determine the need for and timing of formative and summative evaluations in each province. Generally, the negotiation of these agreements did not have a major impact on the conduct of this formative evaluation, except in New Brunswick where federal staff had been transferred to the province. The consultants were asked to not conduct any interviews with local level staff or focus groups with clients in this province given the sensitivities that surrounded the recent changes. The summative evaluations on these programs will focus on the longer-term impacts of the interventions. For example, the summative evaluations may address the incremental impacts of EBSM and the achievement of targets in terms of employment and unpaid benefits. The summative evaluations will need to integrate data and information from provinces that will likely have different approaches to the delivery of active measures to EI clients. 1.4 Presentation of Report The remainder of this report has been organized around the following six chapters:
Each of the chapters devoted to findings presents the main findings related to the evaluation issues examined. The findings on each of the interventions evaluated are presented in appendices to this report, bound separately.
In order to better understand the differences that exist between various types of EI recipients, we have created socio-demographic profiles. These profiles describe the "average" EBSM participant and non-participant, and illustrate different characteristics among the various program participants. The primary source of data for these profiles is the survey of participants and non-participants. However, the following variables from HRDC administrative files, presented in order of importance, were used to "match" non-participants1 to participants for comparison purposes:
Matching procedures used these three variables because they were the only variables for which complete and reliable data existed for all records on the various HRDC databases. Without reliable data, matching and subsequent analysis are inaccurate. For a more detailed description of the matching methodology used for this survey, please refer to the EBSM Formative Evaluation Methodology Report. In this chapter, profiles illustrate the following:
We first describe the overall picture of the typical EBSM participant compared to non-participants to gain perspective on what characterizes the typical EI recipient in Canada. Regional profiles are presented in the EBSM Survey Report which has been submitted under separate cover. 2.1 Overall Profiles
Exhibits 2.1 to 2.9, on the following pages, present selected profile data for all EBSM participants, both active EI claimants and reachback participants, and for non-participants. While it is possible for EBSM participants to receive services under multiple programs, profile data in this chapter refer to survey respondents' most recent intervention only. Based on the socio-demographic profile, EBSM participants, on average, are more likely than non-participants to be male. A higher proportion of active EI claimants are male compared to reachback participants (see Exhibit 2.1). Gender by EBSM Participant Type and Non-Participants
Overall, EBSM participants appear to be better educated than non-participants. Over half (58%) of EBSM participants have completed at least some post-secondary education, compared to 44% among non-participants. One in six participants (16%) have not completed high school, compared to almost one in three (29%) non-participants (see Exhibit 2.2). Among participants, active EI claimants are more likely than reachback participants to have completed only some community college, technical school or CEGEP. Reachback participants are more likely to have received a university degree. Education by EBSM Participant Type and Non-Participants
On average, EBSM participants are 37 years old, while the typical non-participant is 39 years old. Reachback participants tend to be a few years older than active EI claimants (see Exhibit 2.3). Age by EBSM Participant Type and Non-Participants
Data in Exhibit 2.4 indicate that participants are more likely to be either married or single and never married. This pattern is consistent among non-participants, although more than half of the non-participants (62%) are married. Active EI claimant participants are almost twice as likely to be living common-law relative to reachback participants. Marital Status by EBSM Participant Type and Non-Participants
Over half of all participants, both active EI claimants and reachback, have no children under the age of 18 living at home. As might be expected due to the fact that non-participants are more likely to be married, a higher proportion have at least one child under age 18 living at home. In fact, about one-fifth (22%) have two children at home. One-quarter (25%) of active EI claimant participants have one child under age 18 living at home, compared to about one-fifth (20%) of reachback participants. Children Under Age 18 Living at Home, by EBSM Participant Type and Non-Participants
English is the most common mother tongue among participants and non-participants, although among participants, active EI claimants are more likely than reachback participants to indicate that English is their mother tongue. Mother Tongue by EBSM Participant Type and Non-Participants
Survey respondents were asked to indicate which one employment equity group they belong to. Data in Exhibit 2.7 illustrate that most survey respondents do not perceive themselves as belonging to any employment equity group. Proportions in each response category indicate that employment equity groups are well represented among EBSM participants. Employment Equity Group, by EBSM Paticipant Type and Non-Participants
Note: The figures reported in this table are minimum numbers because survey respondents were asked to restrict their responses to one category of employment equity groups. Prior to receiving EI services, EBSM participants were more likely to have been employed in other services, retail trade and transportation industries. Non-participants were generally employed in education and health, construction and other industries. Active EI claimant participants tended to be employed in retail trade, other services, transportation, and construction. Reachback participants were more likely to have been employed in the services sector, primarily in business and other services.  View Exhibit 2.8 On average, survey respondents were not likely to have received social assistance prior to receiving EI benefits. However, EBSM participants (13%) were about twice as likely as non-participants (6%) to do so. Among participants, reachback clients (22%) were twice as likely as active EI claimants (11%) to have received social assistance prior to receiving EI benefits. This is likely a result of reachback participants' benefits having expired. View Exhibit 2.9 2.2 EBSM Program Specific Profiles Exhibits 2.10 to 2.20, on the following pages, display socio-demographic profile information for active EI participants and reachback participants, by EBSM program. While differences exist among participants from each type of program, it appears that the most substantive differences are noted among participants from the Job Creation Partnership and Self-Employment programs. Employment Assistance Services (EAS) Active EI claimant and reachback EAS participants, on average, tend more than other participants to be female. Compared to the "average" EBSM participant, EAS participants are more likely to have received a university degree, especially active EI claimants. EAS participants also tend to be older than participants in other programs, on average, and reachback participants are slightly older than active EI claimants. Active EI claimant EAS participants are slightly more likely to be divorced compared to the average EBSM participant, while reachback EAS clients are more likely to be married. While active EI claimants were more likely to have been employed in education and health, retail trade and other services prior to program participation, reachback participants were more likely to have been employed in the retail trade, education and health, construction and accommodation, food and beverage sectors. The average EAS participant is more likely than the average EBSM participant to receive counselling services from a third-party, particularly so for reachback participants. Of those who have received counselling, the majority have completed at least some of their Action Plan activities, although reachback participants are more likely to have completed only some, while active EI claimants are more likely to have completed all of their activities. View Exhibit 2.10 - Exhibit 2.20 Job Creation Program (JCP) Active EI claimant JCP participants are more likely to have completed only some high school compared to the average active EI claimant participant. Both active EI claimant and reachback JCP participants are more likely to claim French as their mother tongue, and to be Aboriginal. Active EI claimant JCP participants are more likely to have been employed in the fishing and trapping and government services sectors. Reachback JCP participants are also more likely to have been employed in the fishing and trapping industry. JCP participants, both active EI claimant and reachback participants, are more likely to have used social assistance as income support prior to receiving EI. In fact, reachback participants display the highest "dependency" rate of any other program participants. Both active EI claimant and reachback JCP participants are less likely to receive counselling services from an HRCC. They are also about twice as likely to have completed none or only some of their Action Plan activities, which is likely related to the fact that most were still in the JCP job at the time of the survey. Training (TRN) Training participants are more likely to emulate the average or typical EBSM participant, both those who are active EI claimants and those who are reachback clients. Reachback training participants are more likely than the average reachback participant to have been employed in business services prior to program participation. Participants in both of these groups are the most likely to receive counselling services from an HRCC as opposed to a third-party. Among those who had received counselling, Training participants have the highest rate of action plan completion. Self-Employment (SE) Compared to the average active EI claimant and reachback participant, Self-Employment respondents who are active EI claimants and reachback participants are more likely to be older, university educated and married. They are also most likely to be single, never married, and more likely to have children under age 18 living at home. SE respondents are also the least likely to have used social assistance prior to participating in an EBSM program. SE respondents are the least likely to have received counselling services from an HRCC, particularly those who are reachback clients. However, participants in SE tend to have completed most of their action plan activities. Targeted Wage Subsidy (TWS) Participants in TWS who are active EI claimants resemble the "typical" EBSM active EI claimant participant very closely. There are, however, a few notable differences among TWS reachback participants. First, TWS participants of this type tend to be female, unlike the average reachback participant. Second, TWS reachback participants are more likely to have completed community college, technical school or CEGEP. Third, these participants tend to be somewhat younger and, fourth, are much more likely to have been employed in the manufacturing sector prior to participating in EBSM. While there are no differences among TWS participants with respect to who provided counselling services, reachback clients in TWS tend to have completed fewer of their Action Plan activities, on average. Exhibits 2.21 and 2.22 present a financial profile of EBSM components. The total budget for EBSM (Part I and Part II) for July 1, 1996 to June 30, 1997 was $877,772,000. Part I benefits included the income benefits paid to claimants while participating in Part II EBSMs. Part II provided support for clients to participate in EBSMs. 2.3 Predicting EBSM Program Participation This analysis expands the socio-demographic profiles, using statistical modelling to illustrate how program participation is impacted by behavioural and demographic characteristics. Explanatory variables included in the final statistical model of EBSM participation were selected primarily by a computer algorithm. However, variables used in this analysis were chosen based on their importance in related research. Modelling results are explained using the odds ratio, a measure which quantifies the impact of an explanatory variable on the likelihood that a respondent participated in an EBSM intervention. For example, the statement "being female doubles the odds that a person participated in an EBSM program" expresses an odds ratio of two. View Exhibit 2.21-Exhibit 2.22 All Participants versus Non-Participants Fourteen explanatory variables were statistically significant in predicting participation in an EBSM intervention. These effects are independent, controlling for the effects of all other variables. The most statistically significant variables in predicting participation are as follows:
"Creaming" is present if we are able to predict if a person is a participant or non-participant based on variable such as education, age, income, etc. If higher education helps us to predict, in a model, that the person is likely to be a participant, then there is evidence of creaming. Active EI Claimant and Reachback Participants versus Non-Participants Two notable exceptions exist in the model for active EI claimant participants. First, active EI participants tend to have similar personal incomes prior to unemployment compared with non-participants. Second, gender is not statistically significant in distinguishing between active EI participants and non-participants. Strong evidence exists to indicate that reachback participants tend to have lower personal income prior to unemployment. Having a personal income of less than $10,000 prior to unemployment increases the odds that a respondent is a participant by 2.5 times. Reachback participants also tend to be male. Being male increases the odds that a respondent is a participant by 1.5 times. EBSM Program Participants versus Non-Participants EAS participants are similar to non-participants with respect to personal income prior to unemployment, equity group membership and use of social assistance. These two groups also tend to be similar in their agreement with the statements "unemployment is one of the worst things" and "I would like to go to school". The odds of being an EAS participant are nearly doubled if the respondent is single as opposed to married. Being female increases the odds of being an EAS participant by 1.5 times. Attitudes differentiate JCP participants the most from non-participants. For example, compared with a respondent who strongly disagrees with the statement "unemployment is one of the worst things", a respondent who strongly agrees is six times more likely to be a JCP participant. Similarly, the odds that a respondent was a JCP participant are increased 2.2 times if they strongly agree with the statement "I can do little to change important things in my life". Interestingly, JCP participants are the only EBSM intervention group to respond differently to the statement "I would like to go to school". Disagreement with this statement was more common among JCP participants than other participants. Training participants are the only group who are similar to non-participants with respect to education. For all other interventions, participants are more educated than non-participants. Strong evidence exists to suggest that training participants believe that they have less control over situations in their life compared to non-participants. The odds of being a training participant are doubled if the respondent strongly agrees with the statement "I have little control over things in my life". Compared with a respondent with no education, a respondent with university education is at least six times more likely to be a SE participant than a non-participant. For SE participants, being married increases the odds of being a participant by 1.5 times. With the exception of the statement "I can do little to change important things in my life", TWS participants are similar in their agreement with other statements such as "unemployment is one of the worst things" and "there are no jobs". This general agreement with such statements suggests that TWS participants have similar attitudes to non-participants. Being single increases the odds of TWS participation by 1.6 times, while equity group membership more than doubles the odds of TWS participation.
This chapter presents the findings related to the following main themes:
Each of these headings is discussed in detail below. Readers interested in obtaining more information on the systems issue can refer to the revised Information Systems Report dated September 30, 1997. 3.1 Local Implementation and Delivery Findings related to the following topics are discussed in this section on local implementation and delivery:
The major findings related to each issue are presented in boxes, followed by explanatory text.
Despite the potential for duplication of programs at some levels, duplication was reported to be minimal. It was believed that the Labour Market Development Agreements would help address the issue of duplication of federal and provincial programs in the provinces where those agreements have been or will be implemented. Because some SARs would be eligible to participate in an EBSM intervention, they would have access to both federal and provincial programs, some of which might be very similar. The potential for duplication exists for the Targeted Wage Subsidies and Self-Employment programs, in particular. A Review of Programs for Integrating Social Assistance Recipients Into the Workforce identified at least 7 such provincial programs (3 of which were joint provincial-federal efforts, and 3 of which were available from the same province).3 Interviewees also reported that Industry Canada, the Atlantic Canada Opportunities Agency (ACOA) and provincial ministries all have Self-Employment programs in place in the Atlantic. It was also reported that some Employment Assistance Services (e.g., resume preparation, job finding clubs) were duplicated within communities. While the same client would not have access to programs at both the federal and provincial levels, the assumption of interviewees was that administrative costs would be reduced if one type of program was available from a single level of government.
The advantages of local flexibility for the HRCCs include the following:
Based on the feedback from interviewees, the benefits of local flexibility (i.e., local decisions to invest in clients for results) appear to outweigh any drawbacks. However, flexibility was reported to be difficult to accept by some staff who felt more comfortable using a "rule book" or who have difficulty accepting that clients might have access to different services based on where they live or even on their case manager. Some staff, especially those who now combine roles, also felt that they lacked the training and tools to operate within the new system. While Regions generally support local level flexibility, they also appear to have had more difficulty adapting to it than HRCCs have. The difficulty for them comes from determining how to provide advice to HRCCS without infringing on their flexibility or telling them what to do. Difficulty also results from uncertainty as to whether to intervene when HRCCs take action which Regions feel, will result in failure (e.g., approving more projects than their budget allows). Local level flexibility can lead to confusion among community partners and clients. This confusion among clients and community partners stems from anticipated inconsistencies in the type and range of services available between and within provinces/territories. Local level flexibility also poses challenges to, and leads to frustrations for, community organizations that serve an area encompassing numerous HRCCs with different priorities and ways of doing business (e.g., different monitoring requirements). These organizations have to be aware of these different priorities and requirements when preparing proposals in order to adapt them to the HRCC to which it is being submitted, even if the proposed project covers the entire area they serve.
Case management can be described at a very broad level by its two main components. One of these components is similar to a conceptual counselling function, whereby assistance is tailored to each individual's needs until an employment result is achieved. The other is a tracking and co-ordination function whereby client progress is documented on the Client File and tracked until the employment result is registered. Case management starts on the Client File with commitment to a return-to-work action plan and ends with the employment result. At the end of the assisted services (with completion of plan), the Client File is kept open for the results data collection (automatically determined for clients returning to work before the end of the benefits entitlement or through a follow-up 12 weeks after the intervention, as co-ordinated by the case manager). Except for its co-ordination and tracking feature, case management was generally not viewed as a new function for HRCCs, especially for staff who had been fulfilling a counselling function prior to the implementation of EI. Two of the major issues related to case management were that it takes time of which the remaining staff have very little and that the systems do not support it efficiently (entering data is also a time-consuming process). Results-based accountability tracking is therefore under-reported. Lack of resources was the most frequently mentioned reason for the lack of follow-up of clients who do not return to work before the end of their entitlement period. The follow-up appears to be very inconsistent across the HRCCs interviewed (some do extensive follow-up while others do very little). Unless resources are specifically assigned to the tracking and inputting of data, this task will likely not get done. Some HRCCs have dealt with the issue of case management being time-consuming by having third-parties conduct this function (including conducting the follow-up and, in some cases, achieving results).
According to the majority of interviewees, most EBSM participants would have an action plan. On the other hand, the majority of the clients who participated in the focus groups reported not having one. One explanation for this inconsistency is that clients are either unaware of having an action plan or cannot remember having developed one. For example, a client might have agreed to complete a course. While the HRCCs might view this as having an action plan - and entered related information in NESS - the client may not be aware of this. All respondents to the survey of participants were sampled on the basis that they were case managed (i.e., had an action plan in NESS). Survey respondents who had participated in a counselling session were asked whether they had prepared an action plan. Survey results indicate that while only a minority of respondents have attended a counselling session (29.5%), the majority of those who did have prepared an action plan (70.5%). The majority of those who reported having prepared an action plan felt that this plan was important4 (34.6%) or very important5 (43.6%) in helping them get back to work. Findings indicate that it would be possible to follow up and capture results for clients who participate in an EBSM intervention and have an action plan entered in NESS. However, it also appears that clients do not necessarily know what an action plan is or that they have one, even though NESS may indicate that they do. While clients may be aware of the need to take a given course of action, and commit to it, "action plan" is a term with which they are not necessarily familiar. "To be considered for case management, a client must commit to a return-to-work action plan."6 Findings indicate that from a counselling point of view, it may be beneficial for clients' understanding of their responsibilities under case management for action plans to be more clearly defined.
Over four-fifths (85.9%) of respondents who received their counselling services from a third-party organization said they had prepared an action plan, against two-thirds (67.9%) of respondents who had received these services from an HRCC. The majority of the survey respondents who reported having prepared an action plan said that they had either designed the plan themselves (45.5%) or discussed and agreed to the action plan's steps with their counsellor (44.9%). As shown in Exhibit 3.1, those who had received their services from a third-party were less likely to report that they had designed the plan themselves than those who received these services from an HRCC (37.85% versus 46.1%). On the other hand, those who had received the services from a third-party were more likely to say that they had discussed and agreed to the steps with their counsellor (54.3% against 43.9% for those who had received their services from an HRCC). The implication is that third-party organizations probably spend more time assisting clients in the development of their action plan than HRCC staff does.  Overall, almost two-thirds (63.2%) of respondents who had an action plan reported that someone had followed up on the progress of this plan. Almost three-quarters (72.7%) of participants who had received counselling services from a third-party reported that someone had followed up with them on their action plan, compared to 59.6% of respondents who had received the services from an HRCC. Few focus group participants reported having an HRCC counsellor or case manager who followed up on their progress.
The majority of, but not all, HRCCs interviewed for the formative evaluation reported that NFA had been implemented and was currently being used by staff. Those who had only recently started using NFA did not have much experience to report. A few reported that they were not currently negotiating (although they had in the past) because they had very little or no Part II money left.  The majority of training focus group participants and survey respondents reported that they had not negotiated a contribution to the cost of their training. As shown in Exhibit 3.2, only one-quarter (27.9%) of these survey respondents reported that they had negotiated a contribution. Many of the HRCCs reported using tools (e.g., booklets and pamphlets, balance sheet) to help both their staff and their clients negotiate financial assistance. Many HRCCs reported having guidelines, while some even had policies, on their approach to NFA. A number of HRCCs started out with a flat rate and eventually moved to full negotiations. Where HRCCs do not use any guidelines, staff identified the potential for inconsistencies between the approaches used by individual staff as a concern. Interviewees explained that, typically, the clients' current income (e.g., spousal income, child support, pension) and expenses (incremental cost of participating in training only or all expenses) were examined during the NFA process in order to assess the contribution that they can make to the cost of their training. They also reported that NFA was generally accepted as a positive process. However, some staff still feel uncomfortable negotiating assistance, even with training. Uneasiness about using NFA was felt to be particularly strong among staff with a counselling background since negotiating is not part of their previous skill set. Survey participants who negotiated their participation to the cost of their training were asked how satisfied they were with the process. Almost two-thirds reported satisfaction, with one-fifth (21.5%) being satisfied7 and two-fifths (40.5%) being very satisfied.8 Generally, participants in the focus groups with community partners and third-party deliverers also reacted positively (except for the Ottawa participants) to the concept of NFA. They felt that this process would contribute to limiting abuses and making clients more committed to their training.
In extreme situations where clients are unable to contribute anything to the cost of their training, HRCC interviewees explained that the negotiated contribution could be fairly minimal. Interviewees explained that contributing even a token amount, is important for clients to get a sense of ownership over the outcomes of the training and be more committed to it. Only in a few cases was it reported that NFA limited access to training for clients without the means to contribute since training was then denied. As a matter of fact, of the survey respondents who reported that they were no longer participating in training (84.2%, or over four-fifths), only 0.9% reported that it was because they could not afford to pay. Respondents to the survey of non-participants were asked why they had not participated in an employment program or used employment services. Only 2% reported that it was because they could not afford the fee. Most (40.9%) said the reason was that they had found employment. 3.2 Third-Party Service Delivery This section discusses the findings related to the use of third-parties to deliver services.
A number of HRCCs reported using third-party service deliverers, very often in order to deal with the lack of resources that have resulted from downsizing. Employment Assistance Services, in particular, are being delivered by third-parties in order to cope with an expanded pool of clients who require services and HRCCs' decreasing capacity to serve them. In some cases, HRCCs are also using third-parties to deliver services which staff feel uncomfortable delivering (e.g., Negotiated Financial Assistance). The use of third-parties was reported as being limited in some Regions because of the upcoming Labour Market Development Agreements and uncertainties regarding the approach that would be used to deliver employment services in the respective provinces. The majority of third-party deliverers who have been asked to case manage clients did not anticipate major difficulties doing this, as long as this responsibility was accompanied by funding from HRDC that would allow them to allocate resources to this activity. It was reported that some HRCC staff and union representatives had difficulty accepting the use of third-party service deliverers because of the seeming contradiction (cutting staff and then using third-parties to deliver services that HRCCs can no longer offer as a result of a lack of sufficient staff). On the other hand, other staff saw using third-parties to deliver services as a way to deal with the extra workload that cannot be picked up by their HRCC.
Each HRCC has its own contracting process for third-party service delivery. The process (e.g., programs available, where to obtain information or submit a proposal, the criteria or requirements) can be particularly confusing for third-parties who do not have an established relationship with one or more HRCCs. The various requirements used by different HRCCs make the process even more confusing to these organizations. Aside from the issue of unclear contracting process, third-party deliverers also mentioned the following related difficulties:
While third-parties had a number of concerns related to the contracting process, they realized and acknowledged that HRCC staff usually did the best they could to work within a system that, despite recent changes, can be relatively inflexible.
The majority of third-party deliverers described frequent monitoring and follow-up activities, but reported various requirements for reporting results to the HRCCs with which they had contracts. One consistent requirement appears to be the need for organizations involved in the delivery of training programs to report absenteeism to HRCCs. Contact IV, the case management software, was being used or was expected to be used by many of the third-party deliverers consulted in the conduct of this evaluation. In mid-August, National Headquarters reported that Contact IV was being used by approximately 700 third-parties, with plans to expand usage to 1,100 third-parties by the end of September. According to the NHQ representatives interviewed at that time, implementation was well under way in British Columbia and most of the Eastern Regions. Implementation in Ontario was said to be lagging. New Brunswick, where the LMDA has been implemented, has apparently asked HRDC permission to use Contact IV for case management. Two Regions, Alberta and Newfoundland and Labrador, are using their own monitoring systems (Convertor in Alberta and Client Adjustment Tracking System (CATS) in Newfoundland and Labrador). Alberta will reportedly be using Contact IV once the new version becomes available. CATS had been implemented during the early days of The Atlantic Groundfish Strategy (TAGS) and is therefore well established among third-parties. It was reported that work was being done to find a way to upload data from CATS to Contact IV and then to NESS. 3.3 Quality of Services This section of the report presents the findings related to the perceived quality of services delivered by both HRCCs and third-parties, and satisfaction of clients with the services delivered.
Based on the findings from the qualitative research conducted for the formative evaluation, client satisfaction with services appears to be linked to two main factors: ease of access to services and service deliverers, and quality of the services provided once they were accessed. Respondents were asked to rate their satisfaction using a 5 point scale, where 1 is not at all satisfied and 5 is very satisfied Once individuals have accessed the services provided by HRCCs, especially personal services such as counselling, the quality of the services provided was perceived to be high. Clients, both individual clients and community organizations, who had developed a close relationship with one staff member at the HRCC tended to be particularly satisfied with the quality of the services delivered. Comments about HRCC staff included that they provided positive service in a friendly environment, and that they took time to understand their clients and encourage them. However, numerous individuals consulted (clients, community organizations and employers) commented on the difficulties they had experienced getting in touch with someone at their local HRCC. The most frequent complaint was that it was nearly impossible to reach someone directly, that messages had to be left on voice mail and that it often took a long time for messages to be returned. A few community partners commented that they were now bypassing the telephone entirely when trying to contact someone at the HRCC. They found that they obtained responses more quickly when sending requests by fax. The lack of staff appeared to contribute to clients having difficulty obtaining accurate and consistent information, and to their overall impression that staff was ill-informed and that the system was disorganized. Inconsistencies in information were also a concern stated by community partners who dealt with many HRCCs. Community partners were also concerned that clients would suffer because of the increased reliance on technology (e.g., voice mail, job banks), especially if there is no staff available to help clients use this technology. The fact that the quality of services was generally perceived to be high but that access to HRCC services was a major area of dissatisfaction could explain the moderate overall satisfaction levels of survey respondents. Indeed, only slightly over half of survey respondents (56.4% of participants and 53.3% of non-participants) expressed satisfaction with HRCC services. Gorey et al. used a panel study to evaluate the City of Windsor's Special Services and Children's Services Branch from 1990 to 1994. Their background research into studies of client satisfaction with human service programs indicated that "consistently high satisfaction ratings have been observed across all of these service domains, ranging from 65% to 99% 'satisfied' or 'very satisfied'."9 Using this range as a benchmark indication of client satisfaction with human service programs would therefore indicate that survey respondents were only moderately satisfied with the level of service received from their local HRCC. It should be noted that the majority of people who made the comments noted above also acknowledged that HRCC staff was under a great deal of pressure to do more with less as a result of downsizing. Many of the individuals consulted stressed that while they understood that HRCC staff was doing the best that they could under the circumstances, they did not like the situation. Some community partners felt that the cutbacks being made to the HRCCs meant they had to pick up the slack. For their part, HRCC staff reported that their major impediment to providing quality service, in addition to downsizing, was the new roles they have had to combine with their existing role. For example, many staff now combine the roles of both counsellor and project officer. However, many staff consulted felt ill-prepared to take on a new role and believed this was having an impact on the quality of the services they could deliver to clients. For example, some staff who used to provide counselling services only felt uncomfortable negotiating contracts with third-parties. Of all respondents to the survey of participants, 56.4% rated their level of satisfaction a 4 or 5, a moderate level of satisfaction. As demonstrated in Exhibit 3.3, overall satisfaction levels were highest among participants in the Atlantic provinces and Québec and lowest in three of the four Western Regions (British Columbia, Alberta and Manitoba).  Exhibit 3.3 also shows that satisfaction with the employment services received was greater among participants of all provinces, except for Manitoba, than it was among non-participants. A likely reason would be that participants' active involvement in an intervention that might lead to employment contributes to satisfaction.
As shown in Exhibit 3.4, participants in the Self-Employment program reported the greatest over-all levels of satisfaction, with three-quarters who reported satisfaction with services (33.9% reported being satisfied and 40.9% reported being very satisfied).  These high levels of satisfaction among SE participants might be due to the sense of control that Self-Employment focus group participants reported from operating their own business. High levels of satisfaction among this group might also be related to the fact that SE participants typically receive much assistance and support, often one-on-one, during the developmental stages of their business. Three-quarters (75%) of SE survey respondents reported satisfaction with the assistance they received through their participation in the program (23.6% rated their satisfaction a 4 and 52.9% rated it a 5). These findings were echoed in the focus groups with SE participants. One-on-one and personalized assistance was frequently mentioned by focus group participants and interviewees as the basis on which other interventions should be built. This assistance was believed to be particularly needed for individuals who might not be as resourceful as others. There was an expressed concern that the usefulness of self-serve tools, for example, is limited for these individuals and that what they need is one-on-one assistance to help and guide them through the steps of finding employment.
Respondents to the survey of participants were asked to rate their satisfaction with the income or financial assistance received. Findings are illustrated in Exhibit 3.5. TWS respondents rated their level of satisfaction a 4 or 5 in a proportion of 53.5%, compared to 65% for JCP participants. This could be due to the fact that while TWS participants receive a wage paid by the employer and partially subsidized by HRDC, JCP participants continue to receive their Income Benefits up to the end of their Benefit period (which can be topped up at the discretion of the employer and/or the HRCC). Before entering into a contract, TWS participants are thus likely to have expectations for a salary that might or might not be met in this contract. JCP participants, on the other hand, are aware of the amount of their benefits and probably have few expectations concerning the income they will receive during the period of their intervention.  Self-Employment and Training participants were also asked to rate their level of satisfaction with the income from their business and the financial assistance received, respectively. Half (51.6%) of SE respondents reported being satisfied with the income that they received from their business (24.2% rated their satisfaction a 4 and 27.4% rated it a 5). Two-thirds of training survey respondents reported satisfaction with the financial assistance received while on training (19.4% rated their satisfaction a 4 and almost half, 46.7%, rated it a 5). Compared to other respondents, training respondents more frequently expressed high levels of satisfaction (i.e. rated their satisfaction a 5) than any other group of survey respondents.
Comments on satisfaction with third-party services were provided most frequently by training and SE participants. SE participants, as was discussed above, were generally very satisfied with the assistance they obtained from third-party deliverers. Comments about their high levels of satisfaction were made both in the focus groups and survey. Training survey respondents also reported high levels of satisfaction with the training in which they participated and services they received from the training organization. In fact, levels of satisfaction on these two components were higher than on any other aspect measured. Eighty-five percent of training survey respondents reported that they were satisfied or very satisfied with the training they received (23.6% rated their satisfaction a 4 and 62.2% rated it a 5), and in a proportion of 83.5% for the services provided by the training organizations (26.5% rated their satisfaction a 4 and 57% rated it a 5). Satisfaction with these two aspects was high across all Regions. While a majority of focus group participants reported having gained something from the training in which they participated, a number of focus group participants reported that the usefulness of the training had been limited for them. A number of participants reported feeling out of place in their class. These participants felt that greater attention in selecting training participants would decrease the likelihood of such situations occurring. Some focus group participants also commented on the inadequate content of their courses and on the competence of some teachers when explaining moderate satisfaction with their training. 3.4 Communication This section discusses how the availability of EBSM interventions and changes from the UI to the EI system were communicated.
Members of the Employment Implementation Co-ordination Committee (EICC) reported that the committee had been very hands-on in the early stages of implementation (e.g., held weekly conference calls). Issues for discussion were requested from HRCCs and the Regions would report back to them on the outcome of the discussions. The EICC also identified staff training needs and created working groups to deal with this issue. Based on our involvement in reviewing TAGS, it would appear that the early and close involvement of the EICC in EBSM implementation, as well as representation from the Regional committee, have made this implementation much smoother than would have been the case otherwise. Where problems or difficulties were experienced, it was believed that it was due to a break in or lack of continuous flow of communication from the EICC to Regions, and from Regions to HRCCs.
HRCCs frequently reported communicating changes related to EI and EBSM through information sessions with clients. Over half (58.1%) of survey respondents reported having received information from an HRCC about the services they most recently used. In Ontario, a number of HRCCs are delivering Group Information Session (GIS) which can make attending an information session mandatory subsequent to filing a claim for Employment Insurance. Other means of information dissemination included radio, newspaper or television advertisement as well as the provision of information by HRCC staff (e.g., during counselling sessions). Employers can also be a source of information on interventions such as Targeted Wage Subsidies and Job Creation Partnerships. Despite this variety of communication means, focus group participants frequently reported a lack of information regarding the interventions available that could help them get back to work. One of the most frequent complaints was that clients had to ask about the interventions available as opposed to being proactively informed about them. Focus group participants often mentioned finding out about interventions through word-of-mouth. Community partners echoed the view that their clients were not well informed about available interventions. One concern expressed by community organizations was that it was only the most resourceful clients (i.e., those who might be the least in need of an EBSM intervention) who find the information. In addition to feeling ill-informed about interventions in general, some focus group participants also appeared to lack information on the specifics of the interventions in which they were participating. A number of JCP participants, in particular, were not aware that their employer was supposed to give them some time to look for employment towards the end of the JCP project.
One of the most frequent means of informing community organizations about the changes related to the implementation of EI was also information sessions. One-on-one communication between organizations who have had a long-lasting relationship with HRCCs (by telephone or personal visits) was also frequently identified as an additional means of information. In addition, community organizations often belong to networks through which information about available or new programs is rapidly disseminated. Obtaining information was apparently more difficult for organizations who did not have established relationships with HRCCs. Even organizations to whom information about the changes had been disseminated reported getting conflicting information depending on whom they talked with within HRCCs. A number of them felt that this was due to HRCCs not fully understanding themselves the changes associated with EI and the programs available under it. 3.5 Client Selection and Targeting This section presents the findings related to the selection and targeting of clients for EBSM interventions.
Individuals eligible for participation in an intervention are reachback clients and those for whom an EI claim period has been established. The clients and community organizations consulted often stated that these criteria were too restrictive and that individuals whose weak attachments to the labour force (such as youth, persons with disabilities10 and immigrants) would not make them EI-eligible were being ignored under the new Legislation. It was felt that, generally, access to programming for these groups was increasingly limited, at both the federal and provincial levels. It should be noted that interviewees mentioned the existence of a new Opportunities Fund that would be specifically dedicated to persons with disabilities. However, they did not know much about it at the time of the interviews and provided few comments on the Fund. In addition, there are a number of labour market transition programs available to youth under the Youth Employment and Learning Strategy that are designed to improve their future employability.11 The extension of EI eligibility to reachback clients was viewed very positively by the great majority of HRCCs, clients and partners consulted, since it expands the pool of clients eligible to receive employment benefits and support measures.
A number of Regions reported that it had been difficult for HRCCs to build a balanced client portfolio (i.e., a balance of types of clients (e.g., little versus much education) and of clients for whom results can be achieved both in the short term, and medium to long term) in a context where results were emphasized. An issue of concern was that HRCCs would "cream" by targeting clients whose chances of finding employment were greatest, therefore increasing their chances of achieving results in the short term. Since unemployment rates are highest among individuals with less than a high school diploma,12 one would suspect that if creaming was occurring, unemployed individuals with relatively high levels of education would be targeted by the HRCCs. A comparison of survey results for participants and non-participants do show that there might be "creaming" done by HRCCs. Exhibit 3.6, shows the odds that an individual with a university education would be an EBSM participant. However, it would make sense that having a university education would increase the odds of being an EAS user since EAS users have to be self-sufficient enough to find their way in a self-serve environment. It is also known to HRDC from previous evaluation studies that SE participants tend to be more highly educated than other participants. In addition, the large number of participants in training interventions would limit the opportunities for major differences in education to be observed. The two areas where survey results would support the theory that there is some "creaming" being done at the local level are JCP and TWS.
Although the importance of having a balanced portfolio was integrated into communications early in the life of EBSM, the new focus on results was emphasized so much that the former message was not received as clearly as the latter. As a result, some Regions had to review their communications and stress that short term results should not be pursued to the detriment of assisting clients whose results would come in the longer term. It was felt by these Regions that HRCCs had now incorporated this message and were applying its principles.
While HRCCs generally did not report having targeted specific groups to receive EBSM interventions, a number stated that they had (e.g., youth, persons with disabilities, displaced and older workers, women, single parents, immigrants and Aboriginals). Of all these, youth tended to be the group most often identified. While some HRCCs reported that they had targeted specific groups for EBSM interventions, the results of the survey of participants indicate that this practice is probably not wide spread. Indeed, less than 5% of survey respondents self-identified as an Aboriginal person (4.3%), visible minority (3.9%) or person with a disability (1.6%).13 However, youth does appear to make up a significant proportion of EBSM users: almost one-third (31.0%) of survey respondents were between the ages of 16 and 30, and one-half (49.1%) were between the ages of 31 and 45. It should also be added that HRCCs appear to be targeting repeat EI users for EBSM participation. Indeed, three-quarters of participants who responded to the survey reported that their last EI claim had not been their first, compared to half of non-participants. This issue is discussed further in Section 4.2, Reduction of Dependency on EI. A few HRCCs have developed client targeting plans based on community needs strategically identified during the consultations with their community and the subsequent development of their business plan. For others, the approach to targeting clients has been less strategic. It was felt that historical data was lacking to facilitate the prioritization by HRCCs of specific client groups. In addition, some HRCCs reported focusing on specific groups of clients (i.e., active EI claimants only) simply because they had run out or were running out of money to deliver EBSM interventions for the remainder of the fiscal year. 3.6 Cultural Change This section discusses the extent to which EBSM has resulted in a cultural change in HRCCs' way of doing business, including the use of case management and NFA, and the extent to which a management structure based on results has been implemented and accepted.
The extent to which interviewees reported a cultural change in HRCCs' way of doing business varied. A number believed that other factors working in conjunction with EBSM have brought about changes (e.g., maintaining/increasing quality of service with fewer resources). However, local level flexibility was a positive change attributed to EBSM, specifically the opportunity for local level offices to increasingly make their own decisions, which they have embraced wholeheartedly. While an increased level of effort for developing and maintaining partnerships, especially with employers, was identified as a need in some areas, an increased commitment to partnerships (e.g., in the form of increased consultation) has also been identified as positive.
Accountability for results is something that is reportedly well accepted at the HRCC level, even though there is still reticence on the part of some staff to only work with EI-eligible and reachback clients and lose "the entitlement mentality" (i.e., the concept that clients who work the required number of weeks to be EI eligible are entitled to receiving benefits as opposed to having a responsibility for looking for another job). Community partners reported that they had noticed a greater focus on results on the part of HRCCs. Some organizations have adapted or changed the clientele they served in order to obtain results. Consequently, community partners believed that some clients were "falling through the cracks." Interviewees also frequently expressed concerns about the possibility of inaccurate data being inputted into HRDC's systems that will be reflected in the results. The lack of follow-up and the strain placed on limited staff to enter data in the systems were identified. Other systems-related issues are expanded upon in Section 3.8. There appears to be quite a bit of confusion at the local level concerning how results are calculated and, very importantly from management's point of view, what they mean and how to use them once they have been reported.
Given that achieving results was an entirely new process, targets for employment and unpaid benefits to the EI account were set using a top-down approach by establishing benchmarks based on 1994 and 1995 data. These data came from the following two sources:
These benchmarks were adjusted to budgets, including re-investments, and given as minimum targets to be set through a bottom-up process beginning in 1998-99 and informed by results. The establishment of targets was, in part, driven by the need to have set targets before entering into LMDA negotiations with the provinces, by the Shaping the Future exercise (Shaping the Future refers to $105M in incremental unpaid benefits for 1996-97) and by the need to report to Management Board on result-based achievements. The Regions used various approaches to set targets for HRCCs and consulted the local level to varying degrees. A minority of Regions set targets in consultation with local managers, taking into consideration local economic factors. Regions generally set targets for HRCCs using an allocation or resource-base model (e.g., if one HRCC receives 25% of budget, it is expected to achieve 25% of the Regional target). Some Regions explained to HRCCs how targets had been set and a number asked the local level for feedback. While generally not consulted in the setting of targets for the local level, HRCCs generally felt that the targets set for them were realistic and that they would be able to achieve them. The top-down approach used by the Regions to set targets for HRCCs for 1997-98 did not seem to be an issue for them since the HRCCs lacked the experience they felt they needed to establish targets for the first year of EBSM. While it could be argued that Regions and HRCCs should have been consulted during the target setting exercise, the little experience they had, especially at the local level, might have made it difficult for them to set their own targets. However, after over a year of working under an accountability framework that focuses on results, the Regions and HRCCs should theoretically possess the information that would allow them to set local targets that factor in local economic conditions for the year 1998-99, as planned. 3.7 Official Languages This chapter presents the findings related to official languages. The main themes covered in this chapter include the following:
Despite downsizing, clients continue to be able to access services in the language of their choice from HRCCs. The majority of survey respondents reported that they received information about services and services in the language of their choice. Overall, 92.1% of survey respondents in general (including participants and non-participants) indicated that they had received information about employment services in their language of choice. EBSM participants were more likely than non-participants to have received information about employment services in the official language of their choice (97.3% versus 86.9%). We also looked at the findings for EBSM participants from an official language minority community (prefer French in all provinces except Québec and prefer English in Québec). Please note that the sample size becomes very small in many of these provinces (e.g., less than 30 respondents in each of the following provinces: B.C., Alberta, Saskatchewan, Manitoba, Ontario, Newfoundland, Nova Scotia, and P.E.I.). Of those participants in an official language minority situation, 97.8% received employment services information in their preferred official language. This is very similar to those participants not in an official language minority situation, where 98.6% of participants received service in the language of their choice. Offices reported being able to provide bilingual services in those offices required to do so, and some non-designated bilingual offices also demonstrated a capacity to serve clients in a minority official language. The more bilingual the community, the more likely clients were to be able to access services in the language of their choice. However, their ability to receive services and interventions from third-parties in the language of their choice may be limited due to the fact that the delivery of services and interventions in the language of choice of clients is not a consistent contractual requirement. In instances where services are available, this is more likely to be the result of a decision made by the organization rather than a requirement placed upon it by the HRCC. It was suggested that services in a minority official language were easier to obtain in larger centres than smaller ones because larger centres were more likely to be staffed by bilingual personnel. However, some smaller communities with a significant number of minority language members reported that they were better served than the larger centres because the HRCC and community organizations were staffed by individuals who reflected the language composition of the community, and because there were close linkages between the HRCC and the community. The availability of interventions in both official languages was more of an issue of concern than the availability of primary services. In some minority official language communities, it proved difficult to identify focus group participants whose mother tongue was the minority language and who had participated in an EBSM intervention. This indicated that members of the minority official language community are probably not participating in the interventions to the extent that they could. HRCCs and third-party deliverers indicated that, in some instances, attempts had been made to offer an intervention, particularly training or a training component, but that there had been insufficient demand to warrant offering the training course. Even members of minority official language populations themselves, such as Anglophones in Sherbrooke, indicated that one reason they did not access interventions was because they were not being proactive enough in requesting them. A number of HRCCs serving communities with language populations other than English or French indicated that they had little demand for French services, but more so for other languages. Demand was also thought to be influenced by the knowledge level of potential participants. Minority language organizations indicated that their constituents were less likely to pursue information from an organization which did not provide primary service in their own language. For instance, if information on TWS or SE is not being provided by a Francophone delivery agent, then Francophones might be less likely to know about these interventions. Others stated that there is a limited comfort level in making enquiries in one's second language, causing people to be less proactive in requesting information. Minority language community members supported this hypothesis by agreeing that individuals are more likely to go to institutions or organizations to obtain information if the information is given in their primary language. Lack of minority official language employers in the community is also a limiting factor in the use of interventions. It is difficult for Anglophones and Francophones living in a minority situation to use TWS, JCP and even make the most of some EAS, such as job finding clubs, if there is limited employment available in their language of choice, or if they cannot communicate with the employer in their language to secure employment. For example, providing services in French to a Francophone interested in participating in a TWS placement in an Anglophone community might not be an issue, but finding a placement that will allow this person to work in French might be. Of the interventions being delivered by third-parties, the one most accessible to members of official language minority communities is EAS. Obtaining EAS services in a minority language was not a problem for clients. In order to build better information and access links between providers and users, HRCCs are beginning to partner with minority language institutions and organizations to ensure that services are provided to minority language community members. LLMPs, for instance, are being used as a partnering tool to help address the service and labour market needs of minority language populations.
Under the changes associated with EBSM, HRCCs appear to have expanded their consultation with community players who represent the needs of official language minority communities. Evidence of this was found in speaking with HRCCs that serve substantial minority language communities, such as those which exist in Québec and Ontario, that are taking steps to deliver EBSM through partnerships with community organizations that serve these communities. The new approach under EBSM has given HRCCs the flexibility to broaden their consultation with these communities and expand the services, which they deliver through third-parties, who are closer to the needs of the minority language community which they represent. However, where official language minority communities exist but are smaller in size, and where the HRCC possesses a bilingual service requirement, there did not appear to be a particular focus by the HRCCs on involving the minority language community in planning for the ongoing needs of this group.
Survey respondents reported being generally satisfied with the services that they received in the language of their choice. Overall, 92.1% indicated that they had received information about employment services in their language of choice. Except in New Brunswick and Québec, where 37.2% and 91.4% of respondents identified French as their preferred language in which to receive information, over 90% of respondents across Canada reported preferring English. Three-quarters (74.4%) reported being satisfied or very satisfied with the information that they had received. Satisfaction levels by Region are presented in Exhibit 3.7. Exhibit 3.8 shows the proportion of people who received services in their language of choice.
On occasion, there might arise isolated complaints. Just 5% of survey respondents who did not receive services in their preferred language indicated that they had lodged a complaint when they did not receive employment services in their preferred language. Reasons for complaints might include: difficulties communicating with a client in French or English; difficulties obtaining application forms in the desired language; having to deal with a process to get into a program which takes longer for a minority language applicant because a bilingual person has to review the application or because the intervention is not immediately available; or having an intervention delivered by an individual who is not of the same language group and who may not be able to empathize with the needs of minority language participants.
Although the vast majority of participants reported receiving training in the official language of their choice, minority official language participants, especially in rural and remote areas, reported that the lack of availability of training in their language of choice, including language training, was a major gap in service delivery. Many participants who were bilingual accessed the training in the majority language. Otherwise, training may be offered to minority language community clients but they may have to travel greater distances than majority language clients in order to obtain it. If there is no labour market demand for Francophone or Anglophone employees with a particular skill, that training is not likely to be available. The perception that HRDC does not pay for language courses was also identified as a factor that limited their improved employability. Without the language skills of the majority population, members of minority language communities indicated that their chances of finding employment were greatly inhibited. The inability to access training was also found to have an impact on participants' access to other interventions such as SE, where there is often a course or training component given in support of the client's goal of establishing their own business.
The majority of HRCCs reported receiving limited demand for services in a minority official language. At least 90% of participants, with the exception of New Brunswick respondents, reported preferring to receive services in the language of the majority. In New Brunswick, over one-third of respondents reported preferring to receive services in French. We also looked at the results according to which language the participants speak most often at home. Within Québec, of those participants who speak English at home, 97% preferred to receive information about employment services in English. However, outside Québec, 47.1% of participants who speak French most often at home prefer to receive information about employment services in French. Low demand can be attributed to a number of factors:
One community organization stated the views of others succinctly when it noted that in order to communicate information to a minority language population and ensure its awareness of programming, it would not be enough to "have the information sitting on a shelf. You have to go out into the community."
It was reported that Francophones in Ontario and Anglophones in Québec need better access to labour market information to understand how they can meet employers' needs and fit into the labour market. In the interviews conducted, no HRCC, community partner or third-party deliverer indicated that they had addressed the issue of integrating members of minority official language communities into their local labour force. 3.8 Systems A systems assessment was conducted during the course of this evaluation, using the following four specific HRDC systems:
A systems report was prepared based on experiences using these databases during the months of June and July 1997. The conclusions presented in the systems report are summarized here. This section also presents some of the systems-related findings of the qualitative portion of this evaluation.
The following high-level conclusions have been reached from the review of systems in relation to the adequacy of systems for summative evaluation:
As a result of downsizing, a number of HRCCs stated that they did not have the resources necessary to enter information in the systems as it became available. Given this situation, interviewees questioned whether results could be measured at the national level. Some interviewees reported that, nationally, it had been stated that only 50% of case managed clients had been entered in the systems. It follows that accompanying results would not show up in systems. This is compounded by the fact that some information would likely not have been captured during the roll-out of Contact IV to third-parties. Another quoted figure was that 50,000 more clients were identified as having participated in an intervention, nationally, than the number who were reported as having an action plan. This information stresses the need to put a priority on assigning resources to the tracking and inputting of data in order for the information on results to be accurate.
Another issue that needs to be dealt with in view of the monitoring of results is follow-up of clients to record the employment results of all participants, except those who have returned to work before the end of their benefit entitlement. Some HRCCs reportedly conduct systematic follow-ups while others do very little. The key reason cited for limited follow-up, was a lack of resources. While some HRCCs have dealt with this issue by using third-parties to deliver the case management functions, others simply are not following up with clients. The majority of third-party service deliverers reported collecting information on clients in the context of monitoring and follow-up activities. However, the information that they report back to their HRCC varies based on individual HRCCs' requirements. Many third-party service deliverers reported collecting more information than was requested by their HRCC. Some of them reported that they had taken it upon themselves to provide their HRCC with client data because very little was required of them in this respect. This begs the question of how data will be available to HRDC at the time of the summative evaluations if there is not a common base of information that is forwarded from third-parties to HRCCs.
This chapter presents the findings related to the impacts and effects of the EBSM interventions on individuals in terms of the following:
It should be noted that additional findings on employment can be found in Section 5.0, on Primary Results Measurement. 4.1 Impact on Employability and Employment The various EBSM interventions play a role of different importance in improving clients' employability and helping them find employment. It must be recognized that there are other factors at play, such as the state of the local economy and the willingness of clients to move to find a job, that also have an impact on the probability of individuals finding employment (which can be mitigated by helping clients become "job ready" for available employment opportunities).
The majority of participants in the focus groups with clients indicated that they acquired new skills and competencies, and gained useful work experience from their participation in an EBSM intervention. In a number of cases, participants reported that the job-specific or more general skills acquired through their participation in these interventions were transferable to other jobs. Even focus group participants who had been disappointed with other aspects of their participation in an intervention often reported feeling satisfied with the skills and/or experience they had gained. Interviewees also reported that the job experience and softer skills acquired (e.g., independence, time management, and responsibility) generally make participants who do not find a job with their TWS or JCP employer more attractive to other employers. They also reported that even SE participants who did not end up starting their business or maintaining it in operations acquired the generic skills and work experience that would help them in a future job search. The expanded set of skills and competencies, as well as increased self-confidence, which clients gain through their participation in a number of EBSM interventions were seen to improve their situation when looking for employment. These qualitative findings are generally supported by the survey findings. Survey respondents were asked to rate the extent to which they agreed with the statement "The skills that I have learned through the (name of intervention) will help me obtain full-time work in the future."16 As Exhibit 4.1, shows, agreement with this statement was strongest among training and SE participants, with 71.9% and 70.4%, respectively, agreeing or agreeing strongly with it, followed by TWS participants (61.5%), EAS users (52.5%) and JCP participants (51.4%).  The findings related to participants' satisfaction with the skills and job experience gained during their intervention indicate that, for them, more important than acquiring skills is gaining job experience. Indeed, three-quarters of TWS (76.4%) and JCP (74.3%) survey respondents reported that they were satisfied17 or very satisfied18 with the job experience they gained through their participation in these programs, even though 61.5% and 51.4%, respectively, felt that the skills acquired would help them find employment.
A number of focus group participants expressed the need for a "buffer" period at the end of their intervention to provide them with the opportunity to find work. One suggestion was to assist participants with "next steps". Focus group participants identified the lack of a placement component as a gap in many training programs. It was felt that placements would improve participants' chances of finding work at the end of the training program (less than half of training respondents (44.8%) reported finding work as a result of their training). A report synthesizing the programs available to SARs to integrate them into the workforce stated that on-the-job training programs had "mixed results in terms of employability, although [those] delivered through private or not-for-profit agencies were consistently found to have positive impacts on the employment of clients and helped trainees find and/or retain employment in positions for which they were trained."19 Slightly more than one-third (36%) of EAS respondents reported that they had found work as a result of using EAS. It should be noted that EAS users include individuals who may have used a variety of services, from consulting LMI to job finding clubs, and that EAS could be the springboard for participation in other interventions, which themselves may lead to employment.
Many participants felt that individual personalized support should be at the root of every intervention. However, such personalized intervention is not widely available from HRCCs anymore. Issues such as downsizing (resulting in overworked staff and departure of experienced staff) and lack of proper staff training and system support have had a negative impact on HRCCs' ability to help clients return to work. While a number of third-party organizations reported having the capacity to provide individual assistance to clients, they also felt an increased pressure to place clients in interventions quickly so as to get short term results. These participants believed that spending more time up front examining client needs would lead to improved results in the long term. 4.2 Reduction of Dependency on EI
The information that the formative evaluation can provide on the extent to which participation in an EBSM intervention has an impact on reducing client's dependency on EI is limited. Although 72% of respondents had completed their intervention, the short time frame following the completion of their EBSM means that not enough time has passed to determine the impact of the interventions on EI dependency. The findings from the formative evaluation will actually be used as benchmarks for measuring reduction in EI dependency at the time of the summative evaluations. Nevertheless, some preliminary findings in this area can be assessed using information provided by survey respondents on whether their last EI claim was their first and whether they have received income from Employment Insurance since the end of the their intervention.
Overall, three-quarters of respondents in the survey of participants (74.9%) reported that their last EI claim had not been their first (compared to half (48.8%) of non-participants). Of these respondents, three-quarters (76.3%) reported that they had not received income from Employment Insurance since the end of their intervention, indicating that short term dependency on EI may be reduced as a result of participating in an EBSM intervention. As shown in Exhibit 4.2, the greatest impact appears to have been among JCP participants. Indeed, the greatest proportion of respondents who stated that their last EI claim had not been their first was found among JCP respondents (82%). These respondents also stated in the second greatest proportion (preceded by SE participants) that they had not received income from EI since the end of their intervention.  4.3 Impacts on Clients' Attitudes This section describes the extent to which EBSM appear to have had an impact on clients' attitudes towards finding and keeping employment. Again, the reader should keep in mind that clients' attitudes are only one factor that has an impact on their ability to find employment. Participants at the local level raised the issue of the economy and the availability of employment in local industries as key factors that have an impact on clients' employment and the development of a positive attitude.
"Boosting self-confidence" was the most common term used by focus group participants to describe the attitude they developed as a result of participating in an EBSM intervention. Working (through a TWS, JCP or SE program) or participating in an intervention that allows one to develop skills (e.g., training) or make connections (e.g., job finding clubs under EAS) play a major role in improving an unemployed individual's shattered self-confidence and in contributing to a positive attitude. Another way in which all EBSM interventions assisted the majority of clients to develop a positive attitude was by providing them with a supportive environment. Support, as well as the opportunity to network, is provided through group sessions such as job finding clubs, SE workshops and training. In many cases, just having something to do contributes to participants maintaining or regaining a positive attitude toward employment.
Overall, interviewees felt that the implementation of EBSM contributed to clients assuming more responsibilities by requiring clients to do much more of the "legwork", such as conducting research on employment opportunities before being approved for training or SE. A minority of participants also mentioned that case management and the development of action plans assisted clients in taking responsibility and following through with their formal commitments. While requesting clients to conduct research for the development of their business plan seems to be a standard practice across the country, requiring them to conduct research that can justify participation in training is not. The financial contribution that some clients are asked to make through the process of NFA to the start-up and operations of their business or to the cost associated with their training was also identified by interviewees as one aspect of EBSM that made clients more responsible. However, the majority of training focus group participants and survey respondents reported not having negotiated a contribution to the cost of their training.
The two primary results measures for the EBSM components are as follows:20
One of the objectives of the formative evaluation was to examine the two primary results indicators to determine how close the operational estimates produced by the HRIB were to actual client data found on individual BNOP files and through client survey responses. The intention is to provide information that will help HRIB to refine the methodology used to calculate the primary indicators and to improve data integrity. For the purposes of the formative evaluation, the following proxy of the first primary result measure was adopted: the percentage of participants who became employed as a result of their EBSM intervention. The formative evaluation estimates are based on the survey of a sample of program participants as well as administrative data. As a result, the proportion or percentage of participants who became employed was used rather than the total number. The second indicator remained the same: the gross EI unpaid benefits for clients who have returned to work (program administration costs have not been taken into account). Participants were defined as claimants or reachback clients who had an action plan and who had participated in one or more of the six EBSM under evaluation after July 1, 1996. As a result of differences in the availability of data, employment and unpaid benefits have been calculated differently by the formative evaluation and by HRIB. For the formative evaluation, the employment definition is based on the survey responses. All information used to calculate EI unpaid benefits for surveyed EBSM participants is based on data obtained from HRDC information systems. The HRIB definitions of the two primary results indicators are based solely on information contained in the HRDC systems. It is important to note that, in order to implement the operational indicators quickly and to provide data on a timely basis, the initial methodology designed by HRIB was relatively simple. As a result, some level of detail was lost and calculation rules did not take into account exceptions and individual anomalies. It is also important to note that the July 96 to March 97 period was the start-up of EBSM and, at that time, the accountability framework was under development and just in the process of being implemented. Exhibit 5.1 outlines the definitions used by the formative evaluation and by HRIB.
The purpose of the following section is twofold. First, the primary results calculations for the formative evaluation are presented and discussed. Second, some issues with respect to the methodology used by HRIB to calculate results are discussed. 5.1 Formative Evaluation Calculation of Employment and Unpaid Benefits This section of the chapter provides a summary of the results calculated for the EBSM formative evaluation sample. Under the EBSM formative evaluation methodology, the estimates of primary measures were based upon survey responses of 3,000 individuals who participated in EBSM programs between July 1996 and March 1997. In addition, benefit data from HRDC's Status Vector file were used. 5.1.1 Employment For the purposes of this formative evaluation, a participant was defined as being employed if they had worked for at least three consecutive months after the end of their intervention. This definition of employment was chosen for the following reasons:
Exhibits 5.2 and 5.3 show the proportion of participants employed, broken down by program and by region, based on the survey. The total proportion employed according to the methodology is 55.2%. This number includes active claimants and reachback participants employed after their entitlement. Of the 55.2% of participants (active and reachback) considered to be employed by the survey, half had gained employment before their entitlement ended.
Focusing only on those participants considered employed after their entitlement ended, the survey shows 27.6% employment after their entitlement. It is important to note that while HRIB methodology included the calculation of employment for participants who are employed after their entitlement ended, due to earlier implementation constraints, the information was often not available and could not be calculated. HRIB's policy is to capture employment results for participants employed after the end of the benefit period or for participants without a benefit period. Results should be captured at action plan closure or by a 12 week follow-up. However, in practice, employment for participants who are considered employed after their entitlement ended is rarely captured. In most cases, employment is not calculated due to methodology implementation challenges that require telephone calls to determine participant employment status. 5.1.2 Unpaid Benefits Conceptually, unpaid benefits to the EI account are calculated based on a method similar to that employed by HRIB. However, there are important differences in the actual algorithm used to calculate these EI unpaid benefits. The approach of the formative evaluation differs in that, for each week of a participant's claim, we focus on the observed differences between EI entitlement and actual EI payout. In other words, we examine on a week-by-week basis whether a participant has drawn less than the EI amount to which they are entitled. EI unpaid benefits are calculated only for the most recent intervention of survey respondents who satisfy the definition of employment as described above. For each employed participant surveyed, EI unpaid benefits are calculated as follows:
Average gross EI Part I unpaid benefits per person is obtained by calculating the average value of unpaid benefits for individuals who were considered employed according to the EBSM formative evaluation definition. Total unpaid benefits is calculated as the product of three quantities: the average gross EI unpaid benefits per person employed, the proportion of people considered employed and the total number of people who have participated in EBSM interventions in Canada at the time of this evaluation. Exhibits 5.4 and 5.5 show the average gross EI Part I unpaid benefits per person employed by program and by region. It is important to note that the average unpaid benefits for the formative evaluation is calculated only for those participants considered to be employed according to the formative evaluation definition. The average unpaid benefits for participants is the weighted average of the unpaid benefits for those participants employed before their entitlement ended and those employed after their entitlement ended. Again, unpaid benefits can be attributed to an active claimant deemed employed after their entitlement period since it is possible that such individuals could have started working before satisfying the EBSM formative evaluation definition of employment.
Reachback participants are not included as they are not entitled to Part I EI and, as a result, unpaid benefits can not be calculated for them. As indicated earlier, the HRIB methodology for the calculation of unpaid benefits was developed to provide operational information on a timely basis so regions could quickly determine if they were moving in the right direction. In order to implement quickly, the methodology was designed to be relatively simple and did not take into account the full range of anomalies in calculating unpaid benefits. As a result, the estimates were higher than the actual when individual files are examined. Exhibits 5.6 and 5.7 show total gross EI Part I unpaid benefits as estimated by the formative evaluation. Note that these estimates are for the 261,207 individuals who participated in an EBSM intervention between July 1st, 1996 and June 30th, 1997 as reported in the Monitoring and Assessment Report.22
To illustrate the method used to calculate total unpaid benefits for the survey, consider the total unpaid benefits of participants employed before entitlement ending ($264,414,346). To obtain this figure, the proportion of all participants employed before entitlement ending (27.6%) multiplied the total number of EBSM participants (261,207). This estimated total number of participants employed was multiplied by the average unpaid benefits per person employed before entitlement ending ($3,319) to obtain the total unpaid benefits estimate. The same procedure was followed for each program and region for those employed before entitlement ending and those employed after entitlement ending. Total unpaid benefits are the sum of the unpaid benefits for those employed before entitlement ending and those employed after entitlement ending. 5.2 Discussion of Results Calculations One objective of this formative report was to examine the estimates of the two primary results indicators (e.g., how many clients are working and the unpaid benefits for EI claimants returning to work). It was, however, not possible to conduct a direct comparison of the estimates produced by HRIB and the formative evaluation estimates. As discussed earlier, the methodologies used by HRIB and by the formative evaluation differed. The HRIB databases were, at the time of the fieldwork for the formative evaluation, undergoing significant revisions. A later version was used for the Monitoring and Assessment Report. Given that this database has been continuously under development, to date, HRIB has been unable to verify the datafile used. As a result, the evaluation team has not been able to quantify the differences between the estimates from the formative evaluation and the estimates from the Monitoring and Assessment Report. However, it is believed that several aspects of the methodology used to produce the HRIB calculations of the savings results are likely to lead to overestimates. Given the differences of the population covered by the formative evaluation and the universe covered by the HRIB populations, it is not possible to comment on the magnitude of the overestimate. Exhibit 5.8 summarizes some of the differences between the two methodologies.
This analysis is based on information and the datafile received in October 1997. It does not take into account any subsequent changes in the HRIB methodology. The following factors are believed to have the potential to contribute to an overestimate in the savings results used by HRIB:
As explained above, it is not possible to quantify the magnitude of the overestimate given the inability to access some data and the differences in the populations studied. It should be highlighted that HRIB has an operational framework that emphasizes punctual reporting of results. At the time of the evaluation, July 1996 to March 1997, the operational indicators were being implemented and emphasis was placed on providing data on a timely basis. As a result, the initial methodology was simple and some level of detail and correction for individual cases and anomalies was lost. The intention of the indicators was to provide operational information to the regions to allow them to determine progress.
This chapter discusses findings related to the following:
A majority of community partners reported having been consulted on the needs of their community during the implementation of EBSM. Consultations were both formal and informal, with informal consultations taking place on an ongoing basis with community partners that have a close relationship with local HRCCs. Many HRCCs also reported working with local or regional economic development bodies in planning the delivery of programs. Community organizations that did not have close working relationships with the HRCCs generally reported that they had not been consulted on the implementation of EBSM but had been informed about the changes to the EI legislation. In some areas, HRCCs reported that they chose not to involve community partners in EBSM planning given the coming implementation of the LMDAs because they did not want to create unrealistic expectations on the part of the community organizations. While the majority of community partners reported having been consulted on the local economic and labour market plans, not as many indicated having been consulted on the mix of employment benefits and support measures that should be used to integrate these plans. The great majority of HRCCs consulted reported that the five main measures and benefits examined in the course of this evaluation (TWS, SE, Training, JCP and EAS) were all available in their community. However, the link between availability of intervention and the communities' economic development needs was not always apparent. HRCCs felt that they would be in a better position to adjust their next business plans to the economic needs of their communities since they would have a year's worth of experience working under EBSM.
The extent to which partnerships exist at the local level between HRCCs and other relevant organizations in the community appears to be linked to two main factors: the length of the relationship between HRCCs and community partners, and the personality of individuals (both management and staff) within HRCCs. The organizations which have had an ongoing relationship with their local HRCCs for a number of years were generally satisfied with the relationship, although the partnerships might not be as two-sided as they would like. Community partners often reported that the extent to which partnerships existed between the HRCCs and others in the community also depended on individual managers' and staff's commitment or comfort with a partnership approach. These two factors have apparently had an impact on the extent to which community partners felt consulted prior to the implementation of the various EBSM in their community. Despite relatively high levels of satisfaction with current partnerships, a number of community partners felt that there was a need to increase the efforts put into developing partnerships. Indeed, some organizations that had been identified as community partners by HRCCs reported that they did not feel like partners at all. In their view, being partners meant more than communicating regarding funding. In many communities, the use of LLMPs does reflect a true partnership approach. Rather than driving the agenda of a project and providing all of the funding, HRDC was only one of a number of partners at the table under many LLMPs. Other partners contributed in-kind resources and funding, and were active partners in developing the project. LLMPs were seen to be an effective mechanism to draw partners together to address specific labour market issues. A few interviewees also mentioned the need to obtain more input on the part of individual employers (as opposed to umbrella groups such as the Chamber of Commerce) into the identification of local labour market needs. With downsizing, the interviewees at the HRCC level felt they had been less preoccupied with the needs of employers than the needs of unemployed individuals. In some areas, it was felt that HRCCs needed to either take more time to find out about community needs, or play a bigger role in bringing the community and industry together to identify needs. The lack of HRCC staff available to develop partnerships and the lack of resources on the part of community partners were cited as examples of impediments to developing partnerships. HRCCs explained that their community partners still looked to HRDC to provide the funding rather than coming to the table as a partner willing and able to contribute financial or in-kind resources. In addition, the uncertainty about the future delivery of EBSM created by the signing of the Labour Market Development Agreements made it difficult for some HRCCs to continue to develop partnerships.
At this stage, the impact of EBSM on the development of long term employment is anecdotal as many of the activities are in the early stages. Indeed, a majority of interviewees mentioned that it was too early to evaluate the potential of the EBSM interventions to contribute to the development or creation of long term employment in communities. However, based on survey results, potential for long term employment looks good. Interviewees also felt that LLMPs would also play an important role in the development of long term employment. Furthermore, the phasing-out of block purchase training and increasing use of individual purchases were said to contribute to better meeting the needs of local labour markets.
So far, the impacts of various EBSM interventions on communities have not only been felt in the area of employment but also in the area of social development. Interviewees said that JCP contributes to the social development of communities since the program is used primarily by not-for-profit organizations. JCP allows these organizations to have individuals working for them whose wages they might not be able to afford otherwise. Some interviewees mentioned that training institutions and organizations operating in the voluntary sector had known for a while that HRDC would gradually decrease its funding to them. A few interviewees believed that these cuts had negatively affected these organizations (e.g., some training institutions are struggling to maintain their operations and some are left without core or project-specific funding).26 On the other hand, others believed that cuts in funding had forced training institutions to respond to labour market needs as opposed to offering blanket training to anyone eligible.
This section provides a summary of the conclusions drawn and lessons learned from the formative evaluation of EBSM (the key conclusions are highlighted in italics). The relatively new Employment Insurance regime was introduced at a time when many other changes, most notably downsizing, were affecting the department of Human Resources Development Canada. EI was also introduced together with a new philosophy which was hoped would result in a major culture change in the way the department does business. Two of the most important components of this new philosophy are the focus on results (i.e. employment and reduced dependency on EI) and the flexibility which has been given to the local level to make the majority of their decisions without having to obtain approval from the Regions. One of the main conclusions of this formative evaluation of EBSM is that while management and staff at all levels seem to have accepted and adapted relatively well to the emphasis placed on results, the accuracy of the data currently being collected to measure results is seriously put in doubt The introduction of EBSM, combined with other change factors such as downsizing, has meant that staff at the local level have often seen their responsibilities increase without the necessary resources to deal with them. Tasks such as case management, negotiating financial contributions and developing partnerships take time, of which the remaining staff have very little. As a result, tasks such as following up on the action plans developed with clients or entering data into systems have been put on the back burner because of a lack of time or resources. The implication is that unless resources are assigned to tracking and entering data, these tasks will not get done. Regions have tried to emphasize the need to achieve results while helping a mix of clients and soften the earlier message that focused on the importance of achieving results. However, some HRCCs are still struggling with the concept of having a balanced portfolio, that is, assisting a mix of client types and of clients who will achieve results in both the short term and medium to long term Indeed, a comparison of survey results for EBSM participants and non-participants indicate that there might be "creaming" done by the HRCCs, especially for JCP and TWS. This formative evaluation has shown that, while HRCCs have been asked to have a balanced portfolio, the emphasis on short term results has dominated the implementation of EBSM. Together with a lack of experience and good local labour market information, the emphasis on results has made it difficult for a number of HRCCs to develop strategic client targeting plans. Negotiated Financial Agreements are another change to which many staff have had to adapt. Overall, NFA, which will be an integral part of Skills Loans and Grants, is well accepted by both staff and clients The groundwork for SL&G; has therefore been laid and the philosophy that asks clients to contribute to the cost of their training has been endorsed by both clients and staff. Generally, local level flexibility has also been very well accepted and integrated by staff, and is viewed as a very positive aspect of EBSM. However, one of the key concerns regarding local flexibility is that it may lead to uneven service delivery (especially in large centres such as Vancouver, Toronto and Montreal), which staff found difficult to justify, even in a context where flexibility is emphasized. This would indicate that more communication work linking flexibility and results-based management is needed. With downsizing, HRCCs have increasingly had to deal with issues related to their capacity to deliver services with fewer staff. One way that the local level has done this is to use third-party organizations to deliver services. The increased use of third-parties to deliver services has had impacts on staff since many now have to incorporate tasks, such as negotiating contracts and monitoring the work of third-parties, into their work without necessarily having been trained to accomplish these tasks. Furthermore, the way contracts are handled by HRCCs may have an impact on the efficiency with which services are delivered by third-parties and, consequently, on the use that is made of HRDC's funding. The short notice given to third-parties and the relatively short time frame for contracts were identified as major sore points that limited third-parties' ability to efficiently deliver services. Survey respondents only indicated moderate levels of satisfaction with the employment services received or used. Two of the most frequent complaints expressed by focus group participants were the lack of awareness of the programs available to clients and the difficulties experienced in accessing staff within HRCCs. It would appear that lack of knowledge about EBSM interventions is one of the main barriers to accessing them. HRCCs and their third-party partners appear to struggle with the concept of partnerships, and what this concept implies in terms of respective roles and responsibilities, beyond giving money for one party and receiving it for the other. While the issue of partnerships was identified as an area that needs to be developed, the caveat is that developing partnerships takes time and resources, which are increasingly limited for a number of HRCCs. Organizations which do have long-standing relationships with HRCCs or individuals within HRCCs did not feel that they had been well informed about the EBSM-related changes and, for a number of them, consulted prior to implementation. One group that reported feeling somewhat in the dark concerning available programs under EBSM was community organizations who do not have close links to local HRCCs. These organizations not only felt confused concerning the interventions available, but also about how to go about contracting with HRCCs. The focus of the new EI has been placed on results in terms of employment and unpaid benefits to the EI account. It is still very early to make firm conclusions regarding the impact of EBSM on the creation and development of employment. More information on this issue will be obtained during the summative evaluations to be conducted in conjunction with the provinces/territories under the LMDAs. Furthermore, EBSM are only one of the factors that can have an impact on the employability and employment of clients. External factors over which staff and clients at the local level have little control, such as the state of the local economy, also have an impact on the potential for clients in a given community to find employment (which can be mitigated by helping clients become "job ready" for available employment opportunities). Nevertheless, it can be confirmed at this point that two of the major benefits of EBSM are providing clients with job experience through interventions such as TWS and JCP, and improving their self-confidence. Self-confidence was said to play a major role in helping clients remain optimistic about finding employment, and limit the potential for them to become discouraged and give up their job search. Official Languages One of the sub-objectives of this formative evaluation of EBSM was to assess the extent to which the services delivered by HRCCs and third-parties are available to clients in their official language of choice, as well as the quality of the services provided in one or the other of the official language While access to information is generally not an issue of concern, access to interventions in the official language of choice of participants can be a problem in official language minority communities. One reported reason is that demand for interventions in the minority official language is relatively low. However, demand might be low because the official language minority is not aware of the interventions available under EBSM. It was felt that third-party organizations that represent this community are in the best position to deal with minority official language issues. A number of HRCCs have already started to use these organizations to deliver services and it can be anticipated that, if this trend is maintained, access to interventions for the minority official language communities should improve. However, it must be recognized that there are factors, such as the availability of interventions like TWS and JCP in which clients can participate in their language of choice, that are labour-market related issues that cannot be controlled by HRDC. One thing over which the department does have some control is making language training available to clients. Without such training, it can be expected that employment opportunities for clients of the minority official language, and even those of the majority official language who live in bilingual communities, will be limited.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Table of Contents]](/web/20060126230221im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[To Top]](/web/20060126230221im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}