Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

• Introduction
• Project Description
• Evaluation Approach
• Key Findings
• Relevance
• Design and Delivery
• Partnerships
• Post Program Activities
• Preliminary Impacts
• Conclusion
• Recommendations

Introduction

This brief summarizes the findings from the formative evaluation of the Employment Benefits and Support Measures delivered under the Canada-Nova Scotia Labour Market Development Agreement (LMDA).

Project Description

The Canada/Nova Scotia Agreement on a Framework for Strategic Partnerships was signed on April 24, 1997 to undertake joint activities in the area of labour market programs and services in Nova Scotia. While agreeing to work closely together in identifying areas of common interest, Canada and Nova Scotia continue to maintain separate responsibility for the delivery of their own labour market programs and services within the existing authorities legislated by their governments.

Employment Benefits and Support Measures (EBSM) are programs and services designed to assist current and former Employment Insurance (EI) clients prepare for, find, and keep jobs. The formative evaluation reviewed four employment benefits: Targeted Wage Subsidies, Job Creation Partnerships, Purchase of Training, and Self-Employment. The support measures evaluated included Employment Assistance Services, and Local Labour Market Partnerships.

Evaluation Approach

The objectives of the formative evaluation were to:

1. assess the tracking and monitoring system put in place to collect information on participants;
2. estimate preliminary impacts attributable to EBSMs;
3. provide feedback on design, delivery and client experiences for the purpose of program improvement; and
4. produce reliable information on what works best and what lessons can be drawn.

Six sources of information were used to evaluate EBSM: a review and analysis of administrative data; interviews with program officials; a survey of participants and a matched sample of non-participants; focus groups with Human Resources Centres of Canada (HRCC) staff; case studies at three HRCCs; and an econometric analysis to determine preliminary program impact.

Key Findings

In keeping with a formative evaluation, most of the key findings concern program implementation and are intended to provide information on what improvements are required to better meet program objectives. Program impacts were also explored, however these should be considered preliminary only.

Relevance

Most informants were of the opinion that joint cooperation with the Province had minimized unnecessary overlap or duplication between governments, especially for Targeted Wage Subsidies and Self-Employment.

Decision-making at the local level has provided good potential for the EBSM to be relevant to the needs of the community. Community agency representatives interviewed were pleased with the program and its new emphasis on community capacity building.

EBSM is consistent with the EI legislation and with the priorities of Human Resources Development Canada (HRDC). However, program guidelines such as targeting, selection, and case management are not always closely adhered to. At some offices, staff sometimes move away from targeting and selection criteria in their efforts to help all clients.

There are also many cases where an action plan is developed for high-need clients only — clients in need of upgrading and/or skills training — but none is developed for low and moderate need clients. There was no typical use of case management procedures, some offices are dedicated to it while others give it a low priority.

Design and Delivery

In general, implementation of EBSM went smoothly. This was partly because of the flexibility of the EBSM which allowed implementation decisions to be made at the local level. Many informants also felt most component programs of EBSM were continuations of pre-existing HRDC labour market programs making the transition easier. Flexibility of the EBSM's was viewed as the major strength of the organizational structure.

This flexibility, however, raises some concerns regarding consistency among HRCCs in terms of application of EBSM guidelines. In addition, training issues that hampered the flexibility were raised by a number of informants who believed that staff training - a key element of EBSM implementation - fell short of what was required. Most concurred that the training on EBSM programs was too general.

As well as difficulties with training, concerns were expressed related to the limitations of the management information system, especially at the HRCC level. Current administrative systems were said to be inadequate for providing timely information to help manage EBSM programs and demonstrate accountability.

In relation to service delivery, qualitative evidence suggests that the programs are meeting client needs. Most EBSM participants (78 percent) thought their program was good or excellent. Few (4 percent) gave their program a failing grade or a below average grade (5 percent). Discontinuation was not a major issue; according to the administrative data, 93 percent of EBSM clients successfully completed their intervention.

Partnerships

The Canada/Nova Scotia LMDA has been credited with re-establishing positive partnerships between the province and the federal government. There are, however, some challenges - although this is to be expected at the early stages of any partnership. Concerns raised by key informants include a cumbersome committee structure, committee members being on multiple committees, and communication difficulties. LMDA partners are addressing these concerns by streamlining the committee structure.

Post Program Activities

A survey of program participants indicated that about 70 percent were employed for at least 12 consecutive weeks following participation in EBSM. Self-Employment and Targeted Wage Subsidies clients were the most successful in this regard.

Over three-quarters of Self-Employment participants were self-employed after their participation while about 58 percent of Targeted Wage Subsidies and 42 percent of Job Creation Partnership clients continued working for their placement employers.

The survey showed that 40 percent of Purchase of Training respondents contributed to the cost of their training and 95 percent of Self-Employment clients made an initial capital investment in their new business. Nearly all informants held that clients are much more committed to their action plan when it is not being entirely financed by HRDC. They have found clients to be more motivated to succeed if clients pay a share.

Preliminary Impacts

Econometric analysis was conducted to determine preliminary program impacts. These were determined by comparing the outcomes of participants who completed the program in 1997 to those of a comparison group who did not participate in EBSM. It should be noted that these are short-term impacts and should be considered preliminary only.

Participation in EBSM appears to have slightly increased time spent working — Targeted Wage Subsidies was most effective showing an increase of approximately 1-2 months in time spent working. However, no significant impact on earnings was found. Participation in EBSM does not appear to have had an impact on the time spent in school.

Participants who contributed to the cost of their intervention seemed to do better in terms of post-program earnings than those who did not contribute.

There is some evidence that EBSM modestly reduced EI use, measured in terms of both weeks of benefits and total benefits received. However, participation in EBSM did not reduce reliance on social assistance.

Conclusion

Results of the formative evaluation of EBSM under the Canada/Nova Scotia LMDA indicate that implementation went smoothly — a finding mostly attributed to the high degree of flexibility of EBSM. This flexibility provided for decision making at the local level and thus good potential for the EBSM to be relevant to the needs of the community. However, lack of EBSM specific training raised some concerns in regard to consistency among HRCCs in terms of application of guidelines.

The LMDA has been credited for re-establishing positive partnerships that had been forged between the province and the federal government. There were also many examples of successful partnerships with employers and the community undertaken through the EBSM interventions.

Most EBSM participants viewed their program favourably; 78 percent thought their program was good or excellent. About 70 percent were employed for 12 consecutive weeks following participation.

Preliminary impact results also appear favourable suggesting increased employability for participants and reduced use of Employment Insurance.

Concerns were identified that have an impact on EBSM delivery: the availability of timely client results data to assist management and delivery staff in fine-tuning programs and services under EBSM; the lack of a coordinated and consistent approach to delivery throughout the province; and the uncertainty about rules for the delivery of some program components.

Recommendations

• Policy

  Many of those interviewed for the evaluation indicated that the flexibility of EBSM and devolution of power to local staff present challenges to new and experienced workers. HRDC should examine the degree of flexibility it has granted the delivery network in the application of EBSM. A brief policy manual laying out how and when to use each EBSM component and specifying rules that cannot be broken should be developed.

• Administration

  Central administrators and local managers should ensure that delivery staff adhere to the guidelines established for the program. To the extent such guidelines as targeting and selection, case management, and action plans are deemed important for effective program delivery, HRDC should require their use at the local level (which may entail further resources and training).

• Training

  Many local program staff are attempting to carry out their duties under considerable uncertainty regarding the rules around EBSM delivery. Training for newly implemented programs must be more specific, especially when the program has a high degree of front-line discretion. Training on the appropriate application of Job Creation Partnerships and Labour Market Partnerships especially are needed.

• Program Monitoring

  Program monitoring needs to be strengthened. Given the importance of sound results measures to policy formulation, HRDC should provide the resources needed to collect relevant data and ensure managers and staff have access to the information. HRDC needs to reiterate the importance of collecting and using results information to line managers.

HRCCs must also ensure that third parties take monitoring seriously and follow through to ensure the third parties are collecting required data.

In support of the monitoring function, HRDC should come to an agreement as to what types of data are required for appropriate monitoring and design one data system which is simple to use for data input as well as the extraction of results data.