Formative Evaluation of the Employment Benefits and Support Measures (EBSM) - August 1998

Formative Evaluation of the Employment Benefits and Support Measures (EBSM)

View Whole Report

• Purpose
• Introduction and Background
• Data and Evaluation Methodology
• Key Findings

Since the introduction of the new Employment Insurance (EI) system, Human Resources Development Canada (HRDC) has been monitoring and assessing the impact of the EI reforms on individuals, employers and communities. As part of the monitoring and assessment process, a formative evaluation of the Employment Benefits and Support Measures (EBSM) was conducted in mid 1997. The broad objectives of this formative evaluation were to compare estimates of the two primary results indicators (how many clients are working and the unpaid benefits for EI Claimants) as calculated from the information within HRDC's systems and that available from the evaluation survey; provide feedback to managers and policy makers on design. delivery and client experiences; estimate preliminary impacts attributable to participation in EBSM; produce reliable information on "what lessons can be drawn"; and assess HRDC's administrative systems in terms of their ability to meet evaluation data requirements.

The Employment Insurance (EI) Act, or Bill C-12, came into effect on July 1, 1996, which provides both income support and active measures designed to assist unemployed Canadians return to work. Part I of the Act deals with changes to the income benefits and Part II outlines the employment benefits and support measures that are available to clients.

The formative evaluation reviewed Targeted Wage Subsidies (TWS), Self-Employment (SE), Training Purchases and Negotiated Financial Assistance (a precursor to Skills Loans and Grants), and Job Creation Partnerships (JCP). The support measures evaluated included Employment Assistance Services (EAS), Local Labour Market Partnerships (LLMP) and the Transitional Jobs Fund (TJF).

• Review of documents and administrative data. Administrative data were reviewed to assess baseline data requirements. determine the validity of the estimates of the key outcome measures and provide profile information on clients. Documents were also reviewed to provide a context for the evaluation;

• Interviews with key informants. A total of 75 interviews were conducted with HRDC key informants at the national (including Employment Implementation Co-ordination Committee members), regional and local levels as well as with third-party deliverers and community partners.

• Focus groups. A total of 49 focus groups (including dyad and triad interviews) were conducted in 12 communities across Canada, plus two minority official language communities. Focus groups were conducted with clients who had participated in the interventions evaluated as well as with employers who had participated in the TWS intervention;

• Case studies. A total of 10 case studies were researched. plus two case studies that specifically examined minority official languages. The main purpose of the case studies, which included an examination of interventions evaluated, was to identify lessons learned and best practices; and

• Surveys of participants and non-participants. Two telephone surveys were conducted, one with a total of 3,101 participants in EBSM interventions and another with a total of 4,010 non-participants. Non-participants were defined as clients who had not participated, or were not participating, in an EBSM intervention.

Impact and Effects on Individuals. It was generally believed that EBSM, especially SE, TWS and training improved participants' employability by providing them with the skills that would likely help them find or maintain employment. However, in participants' view, the job experience gained was more important than the skills gained through the EBSM interventions. It was also felt that individualized support maximized the impact of the EBSM interventions. Many participants felt that individual personalized support should be at the root of every intervention, that without individual counselling, many clients would not be able to fully benefit from the other interventions. Key HRDC informants believed it was too soon to assess the extent of the impact of EBSM on helping clients find employment however, the majority believed the interventions were having a positive impact on clients' attitudes by improving their self-confidence and providing them with opportunities to be in a supportive environment.

Impact on Communities. The EBSM interventions were having an impact on communities in areas other than just employment, such as social development and operations of not-for- profit organizations. There was consultation conducted at the local level concerning the economic and labour market needs of communities, and how EBSM in general could address these needs. While the satisfaction of existing partners with their relationship with local HRCCs was generally high, the development of new partnerships and enhancement of existing ones appeared to be required for HRCCs to operate within the spirit of EBSM. However, the use of LLMPs generally reflected a true partnership approach.

Official Languages. Primary services (e.g., provision of information) and EAS were generally available to members of official language minority communities. However, access to or use of interventions may have been limited due to factors such as low demand for the interventions, limited information on the available intervention, lack of minority official language employers in the community, or the inability of HRCCs or third parties to provide interventions in the language of choice of participants. HRCCs appeared to be expanding their consultation with community organizations who represented the needs of official language minority communities. Clients were generally satisfied with the services that they received in a minority official language. Limited access by minority official language communities to training in one's language of choice, including language training, was a factor limiting clients' reintegration into the labour market. Official language minority clients felt that they required labour market information on the opportunities available to them in their community.

Local Implementation and Delivery. There existed some duplication of EBSM interventions with other programs. It was believed, however, that existing duplication would be minimized following implementation of the development agreements with the provinces and territories. Local level flexibility was well implemented and generally accepted by HRCC management and staff. Regional offices appeared to have more difficulty adapting to this flexibility. Except for the requirement to follow-up with clients, case management was not viewed as much different than the traditional approach to serving clients. While clients might have an action plan entered for them in NESS, they were not necessarily aware of having one because they did not know what an action plan was. However, respondents who received their counselling services from a third-party were more likely to report that they had prepared an action plan than those who received these services from an HRCC had. Negotiated Financial Assistance (NFA) was reportedly implemented in the majority of HRCCs but experience in using it was limited. NFA was believed to have an impact on clients' commitment to their training, but not on their ability to access training.

Third-Party Service Delivery. HRCCs reported increasingly using third-parties to deliver services, in great part to deal with capacity issues. Third-parties were also increasingly being asked to case manage clients. The contracting process could be confusing to community organizations and timeline issues (e.g., short notice approval of funding, short term contracts) were identified as ongoing concerns. The extent to which the results of third-parties were monitored varied significantly from HRCC to HRCC.

Quality of Services. The quality of the services delivered by the HRCCs was generally perceived to be moderate. It was generally recognized that, while accessing services at the HRCCs was often difficult, HRCC staff were doing the best they could in a context of downsizing and cut backs. While focus group participants expressed some dissatisfaction with the training interventions provided by third-party organizations, survey respondents expressed very high levels of satisfaction with both the services provided by the training organizations and the training that they received. Satisfaction levels with the employment services received were highest among self-employment participants.

Communication. The establishment of the Employment Implementation Co-ordination Committee appeared to have facilitated the implementation of EBSM. Community organizations who had established relationships with HRCCs generally felt well informed about interventions. Those who did not have such relationships often felt confused as to what interventions were available for which groups of clients. Clients frequently reported feelings of frustration regarding their lack of knowledge of interventions available under the new EI.

Client Selection and Targeting and Target Setting Process. There was a strong sense among the clients and community partners consulted that some groups of individuals who might benefit from HRDC assistance were not receiving it because of the new eligibility criteria. On the other hand, the extension of EI eligibility to reachback clients was viewed very positively. HRCCs had struggled to integrate the concept of a balanced portfolio (i.e.; a balance of clients for whom results can be achieved both in the short and medium to long term). The HRCCs interviewed generally did not have specific client targeting plans. Targets for results were set using a top-down approach, which was not an issue for the majority of HRCCs since they felt they lacked the experience needed to set their own targets.

Cultural Change. EBSM worked in conjunction with other factors (e.g., downsizing, changes in the Service Delivery Network) to bring about changes in the way HRCCs conducted business. However, local level flexibility had been a positive change directly attributable to EBSM. Accountability for results was generally well accepted. However, new processes associated with EBSM had put a strain on staff that was not always recognized. Concerns were also expressed about the monitoring of results.

Systems. There were a number of systems-related issues that needed to be resolved to ensure that the data required for a summative evaluation were collected. The accuracy of the results calculated using the information contained in HRDC systems was questioned, primarily because of doubt over the thoroughness of information entered. Consistent follow-up and monitoring were needed to ensure that the proper information was collected and entered into HRDC information systems.

Primary Results Measurement. Upon comparison of the two primary results indicators calculated from HRDC's information systems and those provided from the evaluation survey, it was believed that several aspects of the methodology used to produce HRDC calculations of the savings results were likely to lead to over-estimates. The factors that had the potential to contribute to an over-estimate included: HRDC's method for calculating unpaid benefits during the 12 Week-25% Rule period which had the potential to overestimate the EI Part I unpaid benefits; HRDC's inclusion of unpaid benefits, which had neither been observed nor realized in the period to which the unpaid benefits were attributed: and HRDC's definition of employment (as per the 12 Week-25% Rule) which considered a reduction of unpaid benefits due to reasons other than employment, as due to employment.