Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The Canada/BC LMDA Evaluation Committee is pleased to deliver this Overview Report from the Formative Evaluation of Employment Benefits and Support Measures under the terms of the Canada/British Columbia Labour Market Development Agreement. This Overview Report is complete with detailed findings. The Joint Evaluation Committee has additional technical reports providing more detailed information.

This study was conducted by Keith Vodden of ARC Applied Research Consultants, and by Moira Silcox, Diana Tindall, Rosemary Wallbank, Brad Belhouse, Echo Hsieh of Canadian Facts.

This evaluation could not have been completed without assistance from many staff members in all three organizations. Their cooperation and forthright communication are much appreciated.

We trust you will find the report informative.

The governments of British Columbia and Canada entered into a joint Labour Market Development Agreement (LMDA) on April 25, 1997 to facilitate the co-management of labour market development programs in B.C. Each has agreed to work together in the design and management of Employment Benefits and Support Measures (EBSMs) and in the operation of the National Employment Services (NES). Delivery, in British Columbia, remains the responsibility of Human Resources Development Canada (HRDC). Most delivery is achieved through contractual arrangements between HRDC and third party service providers.

Eligibility under the LMDA is restricted to unemployed individuals pursuant to the Employment Insurance Act. Clients must be either a) active employment insurance (EI) claimants; b) former EI claimants whose benefit period has been established or ended within the past three years; or, c) those who have established a claim for maternity or paternity benefits within the past five years and who are returning to the labour force for the first time since leaving work to care for a newborn or newly adopted child. The latter two groups are eligible through reach-back. A subset of clients are also current or eligible clients of provincial income assistance (IA). These clients are classified as mutual clients.

The LMDA incorporates EBSMs as the mechanisms to achieve the LMDA objectives. As of 1997-98 there were four employment benefits-Job Creation Partnerships (JCP), Targeted Wage Subsidies (TWS), Self-Employment (SE) and Training Purchases (TP)-and two support measures-Employment Assistance Services (EAS) and Local Labour Market Partnerships (LLMP). The objective of these EBSMs is to assist individuals to prepare for, obtain and maintain employment resulting in savings to the EI Account. In so doing, a return on investment will be achieved through a reduction in dependency on EI and IA and additional tax revenues through increased employment.

Targets established for fiscal year 1997-98 were 68,885 EI clients served, of whom 26,000 are mutual clients; with employment returns of 25,008 and savings to the EI Account (unpaid EI) of $117.4 million. (It should be noted that a considerable but unspecified portion of the employment returns and unpaid EI targets reflect the activities of clients who pre-date the LMDA.)

Under the Accountability Framework included in the Agreement, a two-phase evaluation (formative and summative) is to be conducted. This report presents findings from the formative evaluation. Other reports relating to this evaluation include Methodology Report, Technical Report, Evaluation Brief, Recommendations, Lessons Learned and What Worked Well.

The formative evaluation uses multiple lines of evidence (both qualitative and quantitative) to explore seven types of evaluation issues-relevance, design/delivery/ implementation, impact on participants/employers, impact on community, assessment of short-term success indicators, overall impact/cost-effectiveness and EBSM-specific issues.

Qualitative data were collected in the summer of 1998 (approximately 15 months after the LMDA was signed through focus groups, key informant interviews and literature and document reviews. They focus on how co-management of the LMDA is proceeding throughout B.C. as well as more specific information on the operation of the Agreement in three local areas.

Quantitative data were collected in the early fall of 1998 (approximately 17 months after the LMDA was signed through a telephone survey of 1,200 EBSM participants across the province and a separate, comparative survey of 400 individuals who were matched to participants on key variables, but who did not participate in EBSMs. Survey responses were analyzed using univariate, bi-variate and regression techniques.

Key findings for the period of the evaluation follow.

A. Relevance

The relevance of EBSMs were assessed against five key priority areas:

• Consistency With Intent Of EI Legislation—All sources contacted generally considered the EBSMs to be consistent with the intent of the EI legislation.

• The six EBSMs are harmonized to ensure no unnecessary overlap/ duplication. No direct duplication with provincial programs was identified.

• The EBSMs generally promote reduced dependency on EI. However, in the short term, EAS may increase dependence by increasing awareness of other EBSMs. Also, because TWS provides supported employment and insurable hours of work, it may act as a mechanism to get reach-back clients back on EI, thus increasing or sustaining dependency in the short term.

• Considerable effort has been expended on joint activities related to the co-management of the LMDA by HRDC, the Ministry of Advanced Education, Training and Technology (MAETT) and the Ministry of Human Resources (MHR).

• The EBSMs are broad and sufficiently flexible to allow significant decisions at the local level to reflect local circumstances and needs.

• In many respects, the LMDA has been implemented within a framework for success. An Accountability Framework exists for the measurement of success within B.C.; however, there have been serious issues with the systems used to report on success. Ongoing management/performance information is inadequate, unreliable and not timely.

• Consistency With Federal And Provincial Priorities-EBSMs appear consistent with Federal and Provincial priorities. However, the traditional client base of the federal and provincial governments are markedly different. Federal clients tend to have more work experience and a stronger attachment to the labour market. They tend to have recently lost their job and be eligible for EI income benefits compared to IA clients who are either not or are no longer eligible for EI. Provincial clients tend to have more barriers to employment. These tend to be different barriers than the job skill barriers of the typical federal client. Friction exists related to who should be the clients of the LMDA. This friction may be aided by the low number of mutual client participants relative to targets.

• Bilingualism/Language Of Delivery—The demand for services in French is very low, estimated as less than 2% of clients, even in some designated bilingual communities. Service in French is available through a variety of arrangements in most areas of the province. However, some clients may not be served in their area of residence.

• There are some contracts with third party service providers who deal with clients whose mother tongue is neither English nor French and who require English as a Second Language (ESL) assistance.

• Illiteracy, which is estimated by some third party service providers to be as high as 20% among their clients, has implications for access to EBSMs not only in an official language, but for potential access in alternate languages.

• Clients—All groups contacted feel that most eligible active EI clients are being reached or are being made aware of the EBSM programs and services. Only 9% of all participants surveyed report any difficulty accessing EBSMs. (The methodology may have excluded others who for reasons such as poor literacy and poor skills in an official language were unable to access EBSMs.)

• However, evidence suggests that the LMDA may have under-achieved its participant targets. Reported participants appear to be less than 80% of the LMDA's share of the 1997-98 the targets. The number of mutual clients served is less than one-third of the anticipated target. However, other evidence suggest that the official count of participants is under-reported.

• Reasons for the under-achievement of targets are complex. Some of them (e.g., client flow) are being addressed. Others, such as poor public transportation which limits participation in EBSMs and access to jobs and literacy, are less readily resolved.

• Many interviewees are concerned that changes, which pre-date the LMDA, in EI legislation and eligibility criteria (1996), as well as in provincial programming and eligibility criteria (implementation of BC Benefits welfare reform initiative (1996)), result in a substantial group of unemployed persons who are not eligible for existing services.

• Under-Represented Groups—The survey and administrative document review found that approximately 50% of participants are female, 4% to 6% are of Aboriginal descent, about 4% have a disability and somewhere between 2% and 9% report themselves to be members of a visible minority. Because of the need to self-identify, these figures may understate the situation (other than for gender).

B. Delivery

HRDC, MAETT and MHR have established a variety of mechanisms to assist with joint planning, decision-making and program delivery. Findings related to LMDA delivery are:

• Joint Planning—An enormous amount of effort and time has gone into joint planning, especially at the local level. A good deal of co-operation also exists at upper levels of the organization, but below the executive level. The Agreement requires the Management Committee to prepare an annual plan. No separate annual plan was prepared for 1997-98 as the Agreement itself was felt to be adequate. To date, a province-wide annual plan for 1998-99 has not been signed because of disagreement over IA savings. While this has not prevented those in the field from moving ahead on their own plans, it did result in their planning being conducted in a vacuum. Further, it provided an unfortunate signal to many trying to work co-operatively on their own planning exercise.

• Fourteen sub-provincial plans were developed locally and reviewed centrally by Secretariat and field staff. There is a significant level of variation among the sub-provincial plans we reviewed. Deficiencies observed include lack of results information, lack of detail on how non-mutual clients are to be served, use of mixed units for reporting purposes (making comparisons difficult), lack of estimates by specific EBSM and a general lack of detail.

• Community Input—The level of community input into LMDA business planning varies enormously from area to area. However, in all cases it seems that only a limited group in the community, primarily third party service providers, was involved. The community consultation framework prepared in April 1998 by the Communications Working Group had not been fully implemented at the time of the evaluation.

• Local Decision-Making—Almost all interviewees report that there is sufficient flexibility at the local level and that the flexibility has been used effectively to respond to local needs. However, a high level of local autonomy does have some drawbacks and may result in inefficiencies.

• Target Setting—Targets for the number of participants, proportion who are active versus reach-back, number of all participants returning to work and unpaid benefits by current claimants who return to work are set by National Headquarters (NHQ) of HRDC. Targets for mutual clients are negotiated within British Columbia. Mutual client targets exist at both the Human Resource Centres of Canada (HRCC) and the sub-HRCC level (MHR regions). While no targets are set for each EBSM, HRCCs base current year planning on previous year's numbers. In general, targets are not internally consistent. There is no link between the types of clients targeted and the demand for and supply of EBSMs.

• Client Flow—Client flow for mutual clients is another area where local autonomy has been used successfully. However, even in the best situations there have been problems. Each area, and in some situations sub-area, has developed its own process. Interviewees noted a need to refine the process but most also felt that client flow has improved recently. System problems inhibit the distribution of data on mutual clients and some tracking of mutual clients is still done manually.

• Contracting Process—Contracting is one of the most frequently cited areas of discontent. There have been no joint contracts due to financial and legal difficulties. In some situations, the two governments have contracted with the same organization, but they use two separate or "parallel" contracts. Such contracts typically share common start and end dates but vary in terms of value and possibly services contracted for.

• Data Collection, Monitoring, Tracking—There is an Information Sharing Agreement in place between the two governments. It came somewhat late in the process of LMDA implementation and its absence in the early months caused problems. There appear to be some differences in the way this Information Sharing Agreement is interpreted. The lack of an interface between the computer systems of the two governments/three departments causes additional problems in sharing information.

• Information gathering has improved over time. Increased familiarity with Contact IV has resulted in improved information gathering. Nevertheless, there are still some manual systems in place to overcome some system connectivity problems.

• There is concern regarding the completeness, reliability and accuracy of some data. For example, there is no agreement or consistency in terms of the unit of measurement to be used, nor are there definitions for specific units. Reporting remains a problem. The software in use does not produce all the reports required for day-to-day management. Other reports are produced by NHQ. They are not timely and their accuracy is questioned by some.

• HRDC's ongoing management and performance data do not allow it to identify how many/which clients are receiving which EBSM. Thus, without a major effort such as a survey, it is not possible to evaluate the effectiveness of these measures as determined by outcomes.

• Partnering—Processes to support partnerships vary from area to area, as does their effectiveness. Co-operation with community groups is happening to some extent. However, it varies from area to area. Most of the community groups involved in consultation and partnerships to date appear to be either the same as, or closely allied with, third party service providers. Co-operation and partnerships with employers at the local level are not common. Employers are not automatically brought into the consultation and decision-making loop. As a consequence, the impact of most EBSMs is limited to the supply side of the labour market. Their effectiveness may be further constrained if the supply side impact does not match a demand side requirement.

• Third Party Service Providers—In the three field sites, virtually all programs are delivered through third party service providers.

• Duplication/Overlap—The potential for duplication exists as federal and provincial governments continue to offer employment-related programs. However, our evaluation identified no duplication as the programs target different clients or the same clients at different stages in their progression toward a return to work.

• Efficiency And Effectiveness—Some interviewees noted efficiencies through the LMDA, primarily in the area of parallel contracting. However, some inefficiencies were identified. These include having too many third party service providers offering the same service in a geographic area, contractors having to compete with each other through advertising to maintain their share of a finite client base, and short contract lengths that have forced contractors into shorter and more expensive leases and equipment rental agreements. Contractors reported that uncertainty in the contracting process creates uncertainty for their staff, many of whom leave in search of more secure positions.

  Inefficiencies also result from the lack of congruent boundaries for local delivery areas of the three co-management partners. Excessive time is spent co-ordinating the co-management function.

• Clients—The EBSMs collectively offer a broad range of RTW activities. In addition, there is significant flexibility at the local level regarding how EBSMs are applied to meet local needs. The qualitative data suggest that the EBSMs are highly relevant to the employment needs of clients. While many clients receive multiple interventions, these are sequential, not concurrent. Survey results indicate participants are satisfied with their EBSMs and give them positive ratings in terms of their usefulness.

• Employers—Employers' main involvement with the LMDA is through TWS. They are not involved in community consultation and those in our focus group sessions knew little about the LMDA and co-management. It appears that lack of awareness, not the characteristics of TWS and LLMP, is the factor limiting greater participation by employers.

• Information And Communications—The unemployed have a variety of labour market and employment resources available to them. Eighty-two per cent of participants in our survey had made use of resources on a self-serve basis.

• There are significant differences between participants and comparison group members in terms of their use of labour market information and employment resources. Participants were more likely to use the Internet, printed career and work search information and technology provided for their use. Survey respondents rated the information they had received highly.

• There has been limited communication with community groups, including employers.

• Case Management—Case management is widely used, primarily by third party service providers. Not all clients are case-managed, particularly those going through short interventions. Usually, however, those going through longer, more complex or multiple interventions that involve an action plan are case-managed.

• Negotiated Financial Assistance—Clients are expected to contribute to their Return To Work Action Plan (RTWAP) if they are able. Negotiated Financial Assistance (NFA) is the process used to determine ability to contribute and level of contribution. The extent to which NFA is used varies by area, HRCC, third party service provider and even within each specific contract. The amount of knowledge of NFA also varies. Many, but not all, third party service providers have received training or assistance on NFA from their HRCC staff. There is some inconsistency in the application of NFA.

• NFA appears to result in a sharing of training costs for some—5% of surveyed participants said they paid all of the costs of their training, while 46% said they paid part of the costs. This compares to 42% and 8%, respectively, for comparison group members.

C. Impacts/Outcomes on Clients

EBSMs are expected to produce a number of impacts and outcomes for participants. Short-term impacts and outcomes identified by the formative evaluation are:

• Assuming More Responsibility-The EBSMs are intended to encourage clients to assume more responsibility for identifying employment needs, sharing in the cost of assistance, if appropriate, increasing employment and reducing their dependency on EI/IA.

• Views regarding achievement of this goal varied widely from area to area and from agency to agency within an area. Influencing factors include the extent to which a client is job-ready, the prevailing economic conditions and the availability of jobs, especially entry-level or lower-paying jobs. Where jobs are available, EBSMs are seen as reducing dependency. Where there are few long-term jobs, EBSMs are seen as being less successful in securing employment. However, they are still seen as making clients more marketable and better able to market themselves. Action plans are seen as assisting or even forcing clients to take more responsibility for their employment.

• Discontinuation—Almost one-half of participants surveyed who left their EBSM before completion, discontinued because they had found employment. In 24% of cases, discontinuation was a negative result of the program-the withdrawal of funds or dissatisfaction.

• Satisfaction—A majority of the participants whose EBSM involvement was finished at the time of the survey gave their EBSMs a positive satisfaction rating. Participants rated TWS as the most useful EBSM in obtaining full-time employment, followed by SE and TP.

• Participants also gave their EBSMs positive ratings in terms of their usefulness in building their confidence, focusing their career goals and learning how to do a certain job. EBSMs were also considered to be useful, though to a lesser extent on average, at helping participants develop job skills such as managing time and organizational skills, obtain full-time employment and improve or reach their earning potential.

• Attitudes To Work—Most of the participants who had finished their EBSM report that their attitudes toward work and the future have become more positive or stayed the same since their EBSM ended.

• Further Training—Twenty-three per cent of participants have taken further training or courses since the EBSM ended. Almost one-half of these participants took this training as a result of their involvement in their EBSM.

• Employment—Seventy-five per cent of participants no longer involved with their EBSM have been employed subsequently. For members of the comparison group, 69% had worked after the date when the participant they were chosen to represent was scheduled to complete their intervention.

• Excluding SE participants, 64% of whom were in business for themselves, 92% of jobs in the post-program period for participants were with employers (91% for comparison group members).

• Although participants are more likely to work than comparison group members, they tend to have lower average weekly earnings.

• Dependence On EI And IA In The Short-Term—In the short post-EBSM period, 40% of participants (including those on EI at the end of their EBSM) and 37% of comparison group members had received EI, while 9% of participants and 5% of comparison group members had received IA.

• The percentage on EI at the time of the survey was lower for participants (10%) than comparison group members (12%), possibly indicating a reduction in dependence over time. (This will be a topic for the summative evaluation.) Months on EI and benefit rates for those on EI were comparable across the two groups. The only outlier is the current rate on EI for TWS participants which stands at 25% compared to 9% for all other EBSMs. A possible explanation is that time spent on a TWS increases EI eligibility. The percentage on IA at the time of the survey remained higher for participants.

• Impact Of Negotiated Financial Assistance—Those who paid at least part of the costs of their training placed more importance on being independent from EI and IA. However, in the short period reviewed under the formative evaluation, paying for training had not reduced the dependence on EI and IA. Those paying for training were also much more likely to turn down a good paying job if they had to move to get it. They were also slightly less likely to be employed in the period after training.

D. Impacts/Outcomes on Communities

• Impacts On Communities—The following impacts were observed on communities:

  • EBSMs appear to have had some employment impacts, in particular, those SE participants who have started their own business and have hired others to work with them.

  • To date there is little evidence to show that social infrastructure in communities is increased through EBSMs. Nor are city councils and other community organizations involved much with EBSMs.

  • Participation in EBSMs reduces the willingness of residents to move-potentially, participation is viewed as a substitute for moving from the community to obtain employment.

  • Greater volunteerism by participants may result from participation in an EBSM. However, there could be other unobservable factors which influence both participation and volunteerism.

• Impacts On Partners—We were unable to identify partners other than third party service providers. Staff report a lack of time to develop partnerships within their communities.

E. Assessment of Short-Term Success Indicators

This section examines the achievement of short-term targets (the number of participants served, returns to work and unpaid EI benefits) and the appropriateness of success indicators.

• Participants Served—According to Human Resources Investment Branch (HRIB) data on EBSM interventions which started between April 25, 1997 and March 31, 1998, there were 48,427 participants. 97% of the active claimant, 20% of the reach-back and 27% of the mutual client target for fiscal year 1997-98. There were between 2,845 and 3,197 apprentices represented, depending on the method used to identify apprentices.

• There are discrepancies between various HRIB data on participants. Evidence from other HRDC systems suggest that actual numbers of participants may be 35% higher overall and 130% higher for apprentices.

• Returns To Work—According to the HRIB results file, 9,745 of participants under the LMDA had returned to work by March 31, 1998. This is less than the 23,656 returns to work reported by HRIB in 1997-98-the difference being carry-over clients who participated before the LMDA started. Our evaluation focused on non-apprentice participants under long-term interventions (EBSMs). The evaluation found 9,442 to 11,743 of this group had returned to work by September 1998. Accepting HRIB RTW numbers for apprentices and for short-term interventions there may be as many as 16,111 RTW for participants under the LMDA.

• Unpaid EI Benefits—HRIB data suggest unpaid benefits of $42.3 million for participants as of March 31, 1998. This is less than the $93.8 million reported by HRIB for 1997-98 again because of carry-overs. Our estimate, using a longer time frame and non-apprentices under EBSMs, identifies from $21.1 million to $23.2 million. Our estimate will exclude the unpaid EI of apprentices who, after taking the in-school training, return to their employer. The HRIB estimate includes $12.5 million unpaid EI attributable to individuals who may be apprentices. The evaluation method also does not include the unpaid EI for the 2,814 Group Services participants who return to work according to HRIB.

The evaluation found additional problems with the measures and measurement of short-term success.

The evaluation explored alternative short-term measures that may be used in the place of current HRIB measures. These proposed measures are system-derived. The alternative RTW measure uses earnings or the lack of reporting while on claim to identify work. The alternative unpaid EI measure focuses on the part of the claim period where an attributable impact may occur. These measures are presented and compared to the HRIB measures in the report.

F. Cost-Efficiency/Cost-Effectiveness

• Program Cost-Effectiveness-Views regarding efficiency and effectiveness vary from organization to organization and interviewee to interviewee. Most interviewees agreed it is always possible to become more cost-effective. Management inefficiencies result from the lack of congruence in the geographic coverage of local delivery areas within the three organizations. Co-ordination is time consuming as a result. This time could be reduced through re-alignment of the boundaries.

• Many felt that delivery was already fairly cost-effective. Fewer contracts would reduce the contract administration costs. Longer contracts would enable contractors to negotiate better deals (e.g., for leases) and thus realize savings that could be passed on either to clients or to the EI Account.

• Costs Per Success-Interviewees generally could not identify the costs per success.

G. Conclusions

Principal conclusions are summarized below:

• The LMDA delivered EBSM that were highly relevant to the unemployment needs of clients. While multiple interventions can be received they are sequential not concurrent.

• EBSMs are accessible to clients. Those who ask for programs and services are being served, and in their official language of choice. While demand for services in French is low, such services are available through a variety of arrangements.

• Participants are satisfied with their EBSMs. TWS is rated as most useful in obtaining full-time employment, followed by SE and TP. Although more participants than individuals who compare to them obtain work, after the EBSM, wages tend to be lower for participants.

• An enormous amount of effort and time has gone into joint planning under the LMDA. Planning is highlighted as a success of the LMDA. However, there have been disagreements which have lead to no province-wide annual plan for 1998-99.

• The federal and provincial governments' approach and philosophy to contracting vary. Attempts to merge them have proved problematic. "Parallel" contracts offer some efficiencies and appears to work

• There is little evidence to date of the social infrastructure of communities increasing as a result of the EBSMs.

• There are significant concerns about the completeness, reliability and accuracy of some data from information systems. There are opportunities to enhance the connectivity between systems, the use of consistent definitions, and the timeliness and accuracy of reporting.

• Short-term success appears to be less than established targets. However, annual targets and official estimates of success do not reflect well the activities of participants under the LMDA. Participant counts are likely under-reported. The current RTW and unpaid EI measures may be significantly improved.

• Problems related to the measurement of the number of participants, RTW and unpaid EI limit accountability.

Canada and British Columbia have worked in concert to design and manage Canada's employment benefits and support measures since the start of the Labour Market Development Agreement (LMDA) on April 25, 1997. Recognizing the importance of evaluating results, they agreed to co-operate in the development of a formative evaluation following the first fiscal year of implementation. Recently, Human Resources Development Canada jointly with the Ministry of Advanced Education, Training and Technology and Ministry of Human Resources released a report entitled, Formative Evaluation of Employment Benefits and Support Measures Under the Terms of the Canada/British Columbia Labour Market Development Agreement. The formative evaluation report assesses the effectiveness of EBSMs in helping the unemployed of British Columbia to prepare for, find and keep employment.

The report identifies recommendations that focus on the priorities and objectives of the LMDA in the first year of implementation. Since the completion of the research, some significant changes have taken place in regard to the co-management activities that supported implementation of the Agreement. In year two of the Agreement, important learnings from the first year of implementation were applied in the production of sub-provincial plans and services to clients. Value has been added through experience. A joint management response by the three organizations to each of the formative evaluation recommendations is provided below.

Partnerships

Evaluation Recommendation I
Resolve the impasse at the senior management level around federal/provincial priorities and objectives and agree on a united province-wide plan for the coming year.

Management Response

• Canada and British Columbia acknowledge that different goals and perspectives on labour market development issues present an ongoing challenge.

• Progress is being made to clarify objectives and priorities. For example, agreement was reached in establishing 1999/00 priorities and issuing payment guidelines on benefits to mutual clients to field offices.

• All elements of the 1999/00 LMDA Plan have been published. The elements of planning released this year include determining budget and targets allocations for 99/2000, an Environmental Scan, Planning Priorities-99/2000 and a Local Planning Template for the LMDA sub-provincial plans. There is a shared intent to get elements of the plan out to the field sooner to support the 2000/01 planning process.

• The Management Committee is holding co-management meetings more regularly, most recently in March and in June to approve key LMDA activities.

Evaluation Recommendation II
Given HRDC's method of delivering its services since July 1996, integrate third party service providers into local decision-making and planning. Give them a role that is meaningful and respected.

Management Response

• Service providers are one of the groups involved in the community consultation process. Others may include clients, local governments, other ministries and governments, other community agencies and employers.

• Many LMDA planning consultations included client information from third party service providers as evidenced by some 99/00 sub-provincial plans.

• The Planning Working Group has requested field input into the approach and resources required to increase the focus on community consultation. Each sub-provincial area, as part of the LMDA planning process, decides how to incorporate community consultation and would determine how to engage service providers in a meaningful way. While it is important to recognize that service providers add value to consultation, it is also critical to distinguish between funding decisions and enhancement of client services.

Clients

Evaluation Recommendation III
Find a way to enable staff to deal with the needs of clients that may prevent them from looking for work or holding a job once employment is found.

Management Response

• There is a shared concern about clients who have multiple barriers to employment.

• The needs of multi-barriered clients have been identified in many sub-provincial areas and it is hoped that best practices to address some of these needs may be evident in some of the 17 LMDA pilot projects which are currently being reviewed. Field offices will receive the pilots review report and follow-up to the pilots review is planned.

• The Management Committee continues to encourage creativity in sub-provincial areas and sub-provincial areas continue to create local solutions. Creative approaches, such as incorporating literacy into programs so that a greater number of clients benefit from programming, are being tried and will be encouraged.

• Clients with multiple barriers are among the1999/2000 LMDA Priorities that address client service and investment strategy.

Implementation

Evaluation Recommendation IV
Review requirements for sub-provincial plans to ensure that they are sufficiently comprehensive for accountability purposes. Plans should include all clients (not just mutual clients) and present target information in units that will permit comparisons with targets and comparisons year over year.

Management Response

• During the first year of co-management, there was an initial focus on partnership development which has been successfully implemented in local offices to the extent that it now provides a building block to investment strategies for EI active and mutual clients based on community consultation. Sub-provincial plans for this year notably reflect more of a client/community focus.

• The 1999/00 Planning Template was revised to provide clearer guidance to improve the quality of sub-provincial plans. A recent review of all sub-provincial plans indicates that plans are more comprehensive and have significantly improved from the previous year.

• The Secretariat now provides results information centrally and has begun to do so on a more consistent basis as reports are produced.

• The Planning Working Group is revising the format of the templates for 2000/01 to include information that will link priorities and investments strategies to outcomes.

Management Response

• The LMDA specifies that results will be measured according to the number of active EI clients and reachback clients, including mutual clients, who access employment benefits and measures. Primary indicators are the numbers of clients who become employed and the resultant unpaid EI benefits. The Management Committee recognizes the need for improved data capture.

• HRDC's regional accountability group communicates with national headquarters to explain "made in BC" accountability issues and works with local offices to resolve data capture issues. An improved version of Contact 4 software is being implemented.

• The LMDA Budget and Targets committee is looking into the detail provided in the sub-provincial plans on investment strategy and service mixes and will provide recommendations to improve the explicit links between LMDA results and investment strategies.

Accountability

• Agree upon definitions to be used and ensure all staff and third party service providers are aware of them.
• Ensure all HRCCs have a common understanding of data requirements to ensure consistency and thus comparability among the data collected from the various sources.

Management Response

• The Management Committee recognizes that data integrity issues remain to be resolved although, since the time period considered under this formative evaluation, significant improvements have been made and continue to be developed in the area of systems, data integrity and clarifying definitions.

• HRDC's national headquarters (Human Resources Investment Branch) continues to work on the methodology defining "clients served" although timelines for completion of this definition have not been provided as yet.

• Beginning 1999/00, results will be distributed by the Secretariat to field representatives of all three organizations on a regular basis. These include:

  • Part II budgets/expenditure/ commitments report;
  • key performance indicators report (targeted employed, total employed, targeted unpaid benefits and total unpaid benefits);
  • mutual clients served produced jointly by MHR and HRDC (mutual clients targets per HRCC with a monthly breakdown of mutual clients served); and
  • mutual client activity report (mutual client participation in Employment Benefits and Employment Assistance services by MHR region).

• Within BC, definitional issues are being looked at, especially in the area of mutual clients and potential mutual clients, with the intent of communicating clearer definitions of how data is counted for better understanding of mutual client reports produced.

• The LMDA Budget and Targets committee is preparing a template for field offices to provide an additional report following P15/16 on their analysis of how 1998/99 results compare to their planned investment targets.

Evaluation Recommendation VII
Work with front-line managers to identify their needs with respect to data reports, and then ensure those needs are built into the systems.
Use the same framework for outcome reporting as is used for setting targets.

Management Response

• Little systems development has been undertaken to respond/ reflect co-management arrangements, for two reasons:

  • Y2K priorities; and

  • uncertainty around negotiations for provincial delivery.

• 1998/99 results have been provided by HRDC on the primary indicators of clients employed and unpaid benefits.

• MHR is implementing a local and provincial level tracking system.

• Secretariat is reviewing local approaches for gathering results to determine effective practices and is developing a process to share best practices.

Success

Evaluation Recommendation VIII
Determine why participant totals are low. Fix.

Management Response

• The Management Committee acknowledges that there are data capture issues for HRDC to address. The Human Resources Investment Branch (HRIB), is currently working on improving methodology for more accurate data capture and reporting.

• It should be noted that the formative evaluation was conducted early in co-management agreement and that referral processes and data capture are improving. Currently, methodology for medium term savings to the EI account is being developed.

Evaluation Recommendation IX
Review the methods being used to measure success. Make changes which focus on attributable success, make better use of the information available and provide more consistency in measuring success.

Management Response

• Issues relating to HRDC data capture methodology have been brought forward to Human Resources Investment Branch (HRIB) at HRDC's national headquarters by the HRDC regional headquarters accountability group.

Evaluation Recommendation X
At a minimum, present the measures of success separately for apprentices and non-apprentices. Preferably, remove apprentices from targets and the measurement of success.

Management Response

• HRDC's Human Resources Investment Branch has refined data capture for apprentices but continues to include apprentices in the measurement of success. In British Columbia, apprentices are separated in the primary indicators results reports for employed and unpaid benefits by HRCC area.

Management Response to other Evaluation Issues

Section IV: Impacts/Outcomes on Clients
The Management Committee acknowledges that the Ministry of Human Resources does not agree with the section entitled Impacts/Outcomes on Clients in the Overview Report for reasons of dissatisfaction with the methodology used by the evaluator and of considering this information on outcomes to be appropriate for a summative evaluation, not a formative evaluation.

The Management Committee also acknowledges that the Achieving Results Working Group recognized the pivotal role of the PRO's in the LMDA process and, consequently the Ministry of Human Resources, does not consider the importance of the PRO role to have been recognized sufficiently in the report.

This report presents findings from the formative evaluation of the Canada/British Columbia Labour Market Development Agreement (LMDA) in the form of a Final Overview Report. Other reports in this series of findings reports include Technical Report, Evaluation Brief, Lessons Learned , What Worked Well and Suggestions for Improvement. Detail on the methodology used in the formative evaluation may be found in the Methodology Report.

A. The Canada/British Columbia Labour market development agreement

The governments of British Columbia and Canada entered into a joint agreement on April 25, 1997 with regard to the co-management of labour market development programs in B.C. Each has agreed to work together in the design and management of EBSMs and in the operation of the NES.

Delivery is through staff of Human Resources Development Canada (HRDC) or third party service providers, using HRDC funds. Co-ordination of LMDA activities with provincial programs and activities of Ministry of Advanced Education, Training and Technology (MAETT) and the Ministry of Human Resource (MHR) involves a number of committees. These committees include:

• The Management Committee.

• The Secretariat.

• Four Working Groups on Planning/Operations, Systems and Achieving Results, Program Design Communications.

• Four Subgroups or Subcommittees on Information Sharing, Co-Location, Self-Employment, Targeted Wage Subsidy.

• A Budgets Forecasting and Targets Review Team.

The LMDA sets out roles and responsibilities for the federal and provincial governments. Key principles and objectives are:

• Enhancement of client service, program complementarity and program efficiency and effectiveness.

• Joint federal-provincial decision-making in the areas of planning, design and setting program and client priorities.

• No unnecessary overlap and duplication through co-ordination of employment initiatives.

• Reduction in dependency on public assistance.

• Acknowledgement of labour market partners' role in labour market development.

• Flexibility to allow local decision-making with regard to implementation.

• Equity principles with respect to members of under-represented groups will be respected in the design and delivery of programs and services.

• Commitment by persons receiving assistance.

• Sharing of information and data to support client service and program development.

• Implementation of the EBSMs within a framework for evaluating their success in assisting persons to obtain or keep employment.

EBSMs provide the mechanisms to achieve these objectives. As of 1997-98 there were four employment benefits:

• Job Creation Partnerships (JCP).

• Targeted Wage Subsidies (TWS).

• Self-Employment (SE).

• Training Purchases (TP) (sunsetting June 30, 1999) and expected to be replaced by a program of Skills Loans and Grants (SLG).

There were also two support measures under the Agreement:

• Employment Assistance Services (EAS).

• Local Labour Market Partnerships (LLMP).

Eligibility under the LMDA is restricted to unemployed individuals pursuant to the Employment Insurance Act (without employment and actively seeking employment). The Act is divided into two parts. Part I deals with entitlements to income support. Part II deals with discretionary funds. Legislative authority for ESBMs is found under Part II.

To be eligible for EBSMs under EI Part II, individuals must fall in one of the following groups:

• Current (active) EI claimants.

• Former EI claimants whose benefit period has been established or has ended within the past 3 years (eligible through reach-back).

• Those who have established a claim for maternity or paternity benefits within the past 5 years and who are returning to the labour force for the first time after having left work to care for a newborn or newly adopted child (eligible through reach-back).

The resulting definition of eligible recipients means that program participants may be current or eligible clients for IA, a provincially administered program delivered by MHR. Such individuals are classified as mutual clients. Targets for active claimants, reach-back clients and mutual clients have been established.

In addition, unemployed individuals who are not eligible for EI Part II may receive services under employment support measures (EAS and LLMP). These "uninsured individuals" are not included in the targets for the LMDA nor in the findings of the formative evaluation. However, note that apprentices who are current claimants are included in the targets for the LMDA regardless of whether they participated in an EBSM. We deal with the issue of apprentices in the assessment of short-term measures but exclude apprentices from other analyses in the formative evaluation.

The objectives of the EBSMs are to assist individuals to prepare for, obtain and maintain employment resulting in savings to the EI Account. In so doing, a return on investment will be achieved through a reduction in dependency on EI and IA and additional tax revenues through increased employment.

The Agreement requires assessment of the effectiveness, efficiency and value to Canadians of EBSMs. British Columbia and Canada have set up an Accountability Framework to measure expected results. Short-term results are based on performance measures. These track:

• The number of active EI claimants and other EI clients (reach-back), including mutual clients, who have been served under EBSMs.

• Return to employment of EI clients, including mutual clients.

• Savings to the EI Account as a result of re-employment of EI clients.

Targets were established for fiscal year 1997-98 based on these measures. These are outlined in Exhibit 1.

A number of factors which pre-date the implementation of the LMDA have influenced its development:

• EI changes in July 1996-reducing the duration of benefits and increasing the eligibility requirements.

• Downsizing of federal public service, as a debt reduction method, with the resulting increase in service delivery by contracted third parties.

• BC Benefits Welfare Reform Initiative including new provincial employability programming and revised eligibility requirements.

• HRDC's offer of provincial delivery of EI Part II programs and services to the province in a period of devolution.

• Other agreements in all provinces and territories, other than Ontario, with differing arrangements for co-management, devolution and strategic alliance.

• Current negotiations for a new LMDA for B.C. based on a devolved model.

B. Evaluation of the Canada/British Columbia Labour Market Development Agreement

As part of the Accountability Framework, British Columbia and Canada have agreed to co-operate in the development and implementation of a two-phase evaluation framework. These phases adhere to the normal evaluation practices related to a formative evaluation and summative evaluation. The formative evaluation will provide information to enhance the design and delivery of the LMDA and individual EBSMs. This report provides findings from this formative evaluation.

The formative evaluation uses multiple lines of evidence-both qualitative and quantitative-to explore seven types of evaluation issues:

• Relevance.

• Design/delivery/implementation.

• Impact on participants/employers.

• Impact on community.

• Assessment of short-term success indicators.

• Overall impact/cost-effectiveness.

• EBSM-specific issues.

Data were collected in the summer and early fall of 1998. Analyses and findings reflect results up to this period.

Qualitative data were collected on LMDA implementation at two levels:

• Province-wide (in B.C.).

• Area-specific (in three HRDC service delivery areas).

The evaluation gathered general information on how implementation of the LMDA is proceeding across B.C. As well, more specific information on the operation of the Agreement in the three local areas was collected.

Local areas were selected by the Joint Evaluation Committee (JEC) based on HRCC coverage. Sites were selected to be representative of HRCCs in three types of areas-major urban, minor urban and rural. The selection process also ensured that at least one site designated as bilingual was included.

Multiple lines of qualitative data collection were used. They were:

• Focus groups (area-specific).

• Key informant interviews (province-wide and area-specific).

• Literature/document review (province-wide and area-specific).

Eight focus groups were held with participants, employers and third party service providers. The focus groups allowed for in-depth discussion on a fairly narrow range of topics with a small group of people.

Sixty-eight key informant interviews were conducted with government personnel involved in co-managing the LMDA, staff in selected local offices (HRDC, MAETT and MHR) and selected partners, co-ordinators and community leaders in the three local areas. The interviews resulted in a breadth and depth of coverage on a wide range of topics.

As well, numerous references and files were reviewed regarding the issues of interest to the evaluation. In particular, references and files were reviewed in the three local areas. The literature/document and file reviews supplement the information gathered through the interviews and focus groups.

The data collected from these sources were analyzed and are presented primarily in the form of a series of case studies. (See Technical Report, Appendix 3-Case Studies, for details.)

The participant's survey collected quantitative data on the same list of evaluation issues. In total, 1,200 interviews were completed with EBSM participants across the province. The survey achieved a gross response rate of 35% and a net response rate of 42%.

Interviews were conducted with individuals who confirmed participating in specific EBSMs (see Exhibit 2). In addition, 119 interviews were completed with respondents who did not confirm participating in specific EBSMs.

These respondents included current clients (678) and reach-back clients (522). Targets were also set by region and for the three local sites of interest to the evaluation. Specific targets were not set for mutual clients. Instead mutual clients were selected randomly within the population of clients in the targeted groups.

A prenotification letter was sent to individuals in the sample file. Telephone contacts were then attempted with these individuals using a Computer Assisted Telephone Interview (CATI) system. All telephone interviews were conducted between September 9 and 27, 1998.

Survey responses were analyzed using univariate and bi-variate techniques. The key variables used in the bi-variate assessment relate to client status, their experience with the EBSMs and their demographic characteristics.

A separate survey was conducted of individuals who did not participate under an EBSMs (a comparison group). Individuals with characteristics, other than participation, similar to participants under EBSMs were selected statistically among the pool of individuals who were EI-eligible. A prenotification letter was sent to individuals in the sample file. Telephone contacts were then attempted with these individuals using a CATI system. The survey was conducted between October 29 and November 10, 1998.

The comparison group's survey collected quantitative data on the evaluation issues. In total, 400 interviews were completed with individuals similar to EBSM participants across the province. The survey achieved a gross response rate of 34% and a net response rate of 43%. It achieved good representation across areas of B.C. but fewer comparisons for reach-back clients (25%) than had been targeted (33%). Again we did not set specific targets for individuals to mutual clients.

C. This Report

This report has the following structure, organized to reflect the key issues of the evaluation:

• Chapter II—Relevance.

• Chapter III—Delivery.

• Chapter IV—Impacts/Outcomes On Clients.

• Chapter V—Impacts/Outcomes On Communities.

• Chapter VI—Assessment Of Short-Term Success Indicators.

• Chapter VII—Cost-Efficiency/Cost-Effectiveness.

• Chapter VIII—Conclusions.

This chapter assesses the relevance of EBSMs to a number of key priority areas. These areas include:

• The EBSMs' consistency with the intent of the EI legislation.

• Their consistency with both federal and provincial priorities.

• Whether they are available in French and other languages.

• Whether targeted clients are aware of and able to access the EBSMs.

• Whether under-represented groups are participating in EBSMs.

A. Consistency WITH INTENT OF EI LEGISLATION

The EBSMs were generally considered to be consistent with the intent of the EI legislation by all sources we contacted. For example, they are seen to be compatible with the following five LMDA guiding principles:

• Harmonization to ensure no unnecessary overlap/duplication. There does not appear to be much, if any, overlap among the six EBSMs covered under the B.C. LMDA. All appear to offer different forms of assistance to individuals to help them prepare for, obtain and maintain employment. No direct duplication of provincial programs was identified in the evaluation. Although there are some similar programs offered by the province, they are usually directed toward a slightly different clientele; or the same clientele but at different stages in a continuum of programs and services.

• Reduction in dependency on EI. EBSMs promote a reduction in the dependency on EI. They encourage clients to develop a RTWAP and to conduct activities in support of their plans. The objective is to prepare for, obtain and maintain employment-especially long-term employment. In the short term, EAS may increase dependence by increasing awareness of other EBSMs. Also, TWS provides supported employment and insurable hours of work. Because of this feature, it may act as a mechanism to get reach-back clients back on EI, thus increasing or sustaining dependency in the short term. Results for EAS and TWS clients may differ in the medium and long term. (This will be determined through the summative evaluation.)

• Co-operation and partnership with other government, employers and community-based organizations. The LMDA is co-managed by HRDC, MAETT and MHR. Considerable effort has been expended on joint activities related to the management of the LMDA. The effectiveness of co-management partnerships varies with better examples at the local level compared to the joint management level. Community partnerships with others, including business and community-based organizations, has been slow to start. Insufficient time by staff is the principal reason cited. Staff in both levels of government have focused primarily on service to clients and on co-management issues in the first year of implementation. Projects under LLMP, TWS and JCP do involve partners and plans are in place to have greater community, including business involvement in the future.

• Flexibility to allow significant decisions at the local level. EBSMs were generally seen as being very broad. There was also considerable flexibility in interpretation encouraged at the local level. This breadth and the high degree of flexibility enabled local management to make significant decisions related to the implementation of EBSMs to fit the circumstances of their community. The consensus of those contacted through the evaluation is that sufficient flexibility exists in the mix of EBSMs to respond to local needs and that for the most part, this flexibility has been used.

• Implemented within a framework for success. Targets for the number of clients, the number of mutual clients, the number of RTWs and unpaid EI have been set by NHQ. These targets are at the provincial level. Targets for total clients and mutual clients have been set for service delivery areas. However, targets for RTW and unpaid EI do not appear to exist at the local level. Local managers are more likely to use previous years' experience to develop their own targets for their area. Although an Accountability Framework exists for the measurement of success within B.C., to date there have been serious issues with the systems used to report on success. Ongoing management/performance information is inadequate and unreliable. As a case in point, the Mid-Year Review (October 1997) identified three possible interpretations of the data on clients served at 117%, 46% and 17% of the target. As of July 8, 1998, most key indicators were not yet available. HRIB released data about the end of calendar year 1998 on the 1997-98 results. These data are used in our report to comment on short-term success. However, while HRIB has the mandate to measure and report on short-term success indicators, local staff question HRIB information as the provincial totals vary dramatically from their own experience. Third party service providers typically track the success of their own clients and report better than the official success rates.

B. Consistency with Federal and Provincial Priorities

There are marked differences in the traditional client base of the federal and provincial governments. First, typical EI claimants are those who have recently lost jobs with an employer, while typical IA clients do not qualify for EI or have exhausted their EI eligibility. Federal clients tend to have more work experience and a stronger attachment to the labour market. Provincial clients tend to have more barriers to employment, more social problems (drugs and alcohol dependence, family or other problems) as compared to more labour market (job skills) barriers and to require more lengthy and/or multiple interventions to become job-ready.

Friction exists related to who should be the clients of the LMDA, with suggestions by those at the local level that HRDC is taking "the cream of the unemployed to make their projects and numbers look good." HRDC is said to be working with the easiest of the unemployed and to focus on quick fixes for these clients. Critics of the HRDC approach expect a lot of repeat clients-as "success" for clients will be short lived. This is a potential consequence of the short-term success indicators adopted for the LMDA. Measures, and therefore "reported" successes, are maximized through short interventions. The EBSMs appear to be fully consistent with HRDC's priorities as set out in the EI legislation discussed earlier. They target job-ready or near-to-job-ready clients and focus on getting these clients back to work as soon as possible.

The two Provincial Ministries have summarized their mandates as follows:

MAETT is responsible for labour market development, including active labour market attachment programs. MHR is responsible for income support, referrals and compliance. In this regard, a key objective of both ministries is to move BC Benefits clients to a permanent labour market attachment as early as possible.¹

The labour market focus of EBSMs appears to be fully consistent with this stated goal.

Based on results for 1997-98, the province is very concerned that its priorities are not being met through the EBSMs. Only 12% of clients in 1997-98 have been mutual clients, versus the 38% identified in the targets established for the LMDA. Data from our profile of clients indicate that the proportion of mutual clients has been increasing over time but remains below the targeted levels.

According to many we interviewed, provincial priorities are to get people back to work and thereby off the welfare roles. Savings to the IA account are not identified as a target of the LMDA. The province realizes savings to IA when its clients:

• Are employed or re-employed.

• Receive income support under EI Part II while on an EBSM.

• Move from IA to EI after an intervention.

HRDC has concerns regarding the perceived priority to get the unemployed off the IA system, particularly if doing so merely transfers the client to EI. There have been disagreements about the use of EI Part II for income support of mutual clients and the goal of transferring clients from IA to EI. Work on an annual plan for 1998-99 has not been completed as a result of the province's desire to have a joint target for IA and HRDC's position that it does not have a mandate for IA.

Until these differences in priorities are accommodated, there will continue to be tension related to the intended clients and goals.

C. Bilingualism/Language of Delivery

Nine HRCCs have been designated bilingual based on the Official Languages Act (Vancouver, Abbotsford, Chilliwack, Kamloops, Kelowna, Penticton, Prince George, Nanaimo and Victoria). Terrace and Nelson are not officially designated bilingual but have bilingual staff and provide bilingual service. Other offices that also provide bilingual service are Prince Rupert, Campbell River and Comox. If bilingual staff are away, temporary arrangements are made for service in French through other offices-typically Vancouver. A 1-800 number is available to assist clients.

Service in French is available through a variety of arrangements in most areas of the province. For example, in the Lower Mainland one contractor provides all French language services for all HRCCs on a cost recovery basis. Francophone clients tend to be spread out within the province. A flexible approach to delivery is considered to be more cost-effective. This flexibility is seen to be a main reason for the Lower Mainland program's success.

However, some of these arrangements mean that clients may not be served in their area of residence. For example, some Fraser Valley clients go to 10th Avenue in Vancouver for services in French.

Francophones living in the Interior and northern parts of the province are less likely to be able to receive services in French. Victoria recently initiated a pilot to provide French services to clients in the remaining areas of Vancouver Island. Other pilots are being held to determine need.

In the survey of 1,200 participants, 4% of respondents reported their mother tongue as French, but only 3 individuals (one-quarter of 1%) identified requesting an employment program or service in French. These data concur with reports from field staff where the demand for services in French appears to be very low-estimated as less than 2% of clients-even in a designated bilingual community such as observed in one of the case study sites. The main reason cited for the low demand for services in French is that clients expected to obtain jobs in English. Thus, they seek programs and services in the same language as the expected language of the jobs.

There are some contracts with third party service providers who deal with clients who have neither English nor French as their language. Service providers note that many of these individuals are highly-skilled immigrants who require English as a Second Language (ESL) assistance. Presently, some of these contractors try to insert employment-related ESL into their EAS sessions with this group of clients.

Service providers note that many immigrants are well-skilled; what they need is information on how things work in Canada-job finding, résumé writing and interview skills are all different from many other countries.

Another language issue relates to literacy. In the focus groups, third party service providers were asked what percentage of their clients were illiterate. Some answered as high as 20%. Thus, we assume that persons who are illiterate are receiving EBSMs. However, this has implications for access to EBSMs not only in an official language, but for potential access in alternate languages.

D. Clients

The LMDA targets current and former EI claimants through limited reach-back provisions. All groups contacted in the formative evaluation feel that most eligible active EI clients are being reached or are being made aware of the EBSM programs and services.

However, evidence from the profile of the characteristics of clients under EBSMs suggests that the LMDA has under-achieved its participant targets. Actual participants appear to be less than one-half of the targets. The number of mutual clients served appears to be less than one-third of the anticipated target. (More detail may be found in Chapter V-Impacts/Outcomes On Communities.)

Reasons for the under-achievement of targets are complex. Some of them are being addressed.

To overcome factors which may have limited client participation, staff in site visits identify that a major focus is now on refining, improving and enhancing the referral process (client flow) to reduce the number of eligible clients who are not being served. Just as there are claims that HRDC is failing to serve mutual clients, there are also claims that the province is not referring mutual clients for services.

However, some factors cannot be remedied with improved client flows:

• An issue expressed in two of our sites was the limitation imposed by poor public transportation. It limits participation as well as access to jobs. Individuals in rural areas cannot get to jobs. Still others cannot accept shift work or part-time work with irregular hours due to the limitations of car pooling.

• Another issue which may limit participation relates to literacy. As noted, some third party service providers estimate that as many as 20% of clients may be illiterate. This has implications for access to EBSMs not only in an official language, but for potential access in alternate languages.

We did not interview any clients who have poor English or French language skills in our qualitative data collection. However, indirect evidence comes from the participant and comparison group surveys. Nineteen per cent of participants do not have a grade 12 education. In the comparison group, 26% do not have grade 12.

Only 9% of all participants surveyed report any difficulty accessing EBSMs. In terms of frequency of reason cited for the difficulty, principal reasons were:

Note that our participant survey only includes those who were able to participate. Others may have been excluded from participation as a result of literacy issues. Also, note that the participant (and comparison group) survey was conducted in English only and, therefore, may have limited who we talked to-this could affect the literacy, bilingualism, visible minority and second language issues/findings.

One concern relating to clients was mentioned by a very large number of interviewees. In their opinion, changes in EI legislation and eligibility criteria (1996), and in provincial programming and eligibility criteria (transfer of provincial programs to MAETT), result in a substantial group of unemployed persons who are not eligible for existing services. These individuals need help if they are to return to work but are denied help as a result of a gap between eligibility for LMDA programming and provincial programming.

Those who appear more likely to fall outside the EI-eligible group are:

• Women who have been outside of the labour force for long periods to raise their families.

• Former resource workers who typically have up to a grade 8 education. They cannot re-educate without first upgrading and there are no funds for upgrading.

• Self-employed business people whose businesses have failed.

• Seasonal workers who have not participated in EI before and now no longer qualify for EI.

• Those with disabilities who may require additional supports to become job-ready.

It should be noted that uninsured participants can receive support under EAS and LLMP. Such individuals may benefit from the full range of EAS. Partnerships under LLMP can be formed to assist identified groups with their RTW activities, including those not eligible for other EBSMs. However, it is not known the extent to which those who are not EI-eligible are:

• Aware that these employment services are available to them.

• Avail themselves of these services.

• Captured by data systems when they do participate.

E. Under-represented Groups

The profile of participants identifies the following percentage of under-represented groups. Note that the data to support this analysis comes from administrative sources and that individuals need to self-identify to be considered as members of under-represented groups. As a result, actual participation by individuals in under-represented groups is expected to be higher than the figures indicated below:

• Forty-six per cent of participants are female. Though slightly higher proportions of female clients participated in EAS, males participated more in TP or JCP.

• About 6% of participants are of Aboriginal descent. They are more likely to participate in JCP and less likely to participate in SE. Participation by those of Aboriginal descent is much higher in Prince George and Terrace.

• About 4% of participants have a disability. Participants with disabilities are more likely to have received TP and less likely to have received either EAS or SE. They appear in programs in higher proportions in Burnaby, Surrey and Kelowna.

• Very few participants (2%) report themselves to be members of a visible minority. Participants who are visible minority members are in proportionally greater numbers in Burnaby, Prince George and Kelowna.

Our survey proportions were 54% female, 4% of Aboriginal descent, 4% disabled and 9% visible minority group. Again, respondents (except for females) need to self-identify. The actual proportion of clients who are from under-represented groups is likely higher as a result.

HRDC, MAETT and MHR have established a variety of mechanisms to assist with joint planning, decision-making and program delivery. These are discussed in this chapter.

A. Joint Planning

An enormous amount of effort and time has gone into joint planning, especially at the local level. A high degree of co-operation was observed to exist at upper levels (but below the executive level) of the organizations involved. Most government interviewees considered planning to be among the best elements of the LMDA activities to date.

The Agreement calls for:

An annual plan prepared by the Management Committee outlining the policy objectives to be attained, the employment benefits and support measures to be delivered, priority clients and programs and services to be offered by HRDC, notional funding allocations to each program and service, and mechanisms and strategies to be used in the implementation of the EBSMs and performance targets and measurement of results.

No separate annual plan was prepared for 1997-98. The Agreement, signed after the beginning of the fiscal year and with targets for 1997-98, was felt to adequately meet the requirement for an annual plan. Work on the 1998-99 annual plan began in the fall of 1997 and a first draft was presented to the Planning/Operations Working Group on November 28, 1997. However, to date, the province-wide annual plan has not been signed because of the disagreement over IA savings outlined in the previous chapter.

While this has not prevented those in the field from moving ahead on their own plans, it did result in their planning being conducted in a vacuum. Further, it provided an unfortunate signal to many trying to work co-operatively on their own planning exercise. Also, some groups obtained copies of the draft provincial plan-some did not. This caused problems at the local level, especially when those who saw the draft were from MHR (whose Deputy Minister declined to sign) and those who did not were from HRDC and MAETT (both of whose Management Committee members were willing to sign).

Fourteen sub-provincial plans were developed locally and reviewed centrally. The process to develop joint sub-provincial plans was assisted by a Co-management Planning Conference (November 13 and 14, 1997) attended by more than 150 delegates from HRDC, MAETT and MHR. A framework (template) for joint sub-provincial plans was prepared. Submitted plans were reviewed centrally and only 2 of 14 were judged to require revisions.

Despite this co-ordination, we observed a significant level of variation among the jointly-developed sub-provincial plans we reviewed. This may be an indication of the flexibility of the LMDA or a limitation of the framework or review process.

The three sub-provincial plans reviewed by the evaluation are deficient in a number of key areas:

• While targets are presented, there is little information regarding results.

• Plans focus on mutual clients; there is little, if any, detail on how non-mutual clients are to be served.

• Targets are sometimes, but not always, broken down by mutual clients. In some plans, the geographic units are mixed. This makes comparison from year to year and from one set of figures to another very awkward, if not impossible.

• Many statements are "motherhood"-they are extremely general, with little detail to show, for example how the principles are to be met and evaluated.

• Very little estimating by specific EBSM is indicated. EBSMs are intended to be client centred. However, estimates of the likely demand are not made.

B. Community Input

The level of community input into LMDA business planning varies enormously from area to area. However, in all cases it seems that only a limited group in the community was involved. To date, it appears that community input is limited to third party service providers. In interviews with staff of HRDC, MAETT and MHR, we were often told that there was some or a lot of community input. But when we asked for persons in the community (other than service providers) for interview or focus group purposes, very few, and in some cases none, could be identified. Third party service providers also commented on lack of community input at some sites. In addition, the community consultation framework prepared in April 1998 by the Communications Working Group has not been fully implemented in all areas.

Insufficient time is the prime reason cited for the lack of community input. In the first year of implementation, staff of both levels of government have focused primarily on service to clients and on co-management issues.

In one of the case study sites visited we were told that staff listen, seek input from the community, and their business plan records some community forums last year and plans for more in the coming year. Community input appears to have resulted in a smoother-running operation.

In another site visited, the community (primarily third party service providers) reported that typically consultation consisted of government telling them over two hours what was happening and then allowing 15 minutes for 17 service providers to make their comments. Service providers note that their input was ignored and their concerns were not addressed. Lack of community input has resulted in the service providers forming their own group and pressuring government to listen to them. This wastes time, energy and is a disservice to clients, who ultimately suffer from the friction between service providers and government staff.

Less formal community consultation is happening in some cases. Some individual project officers maintain regular and often frequent contact with third party service providers so that concerns can be addressed and programming changes can be made in a timely way.

C. Local Decision-Making

Almost all interviewees report that there is sufficient flexibility at the local level and that the flexibility has been used effectively to respond to local needs. We encountered numerous examples of local variation in interpretation to meet specific requirements. For example:

• One site has chosen to use Assessment and Referral Centres for referrals as staff there feel that it responds well to their local needs.

• Another site has used TWS as a mechanism for providing interim jobs to workers in its very seasonal/temporary and high unemployment labour market.

• French language services are delivered in the Lower Mainland on a cost-recovery basis.

A minority suggested that there may be too much local flexibility and autonomy. However, a high level of local autonomy does have some drawbacks. For example:

• Local staff in each area were required to develop procedures and ways of operating. This duplication would have been reduced through more central planning.

• Provincial government staff who have to deal with more than one HRCC (MAETT, MHR and HRCC boundaries are not aligned) have to learn and accommodate the different local arrangements.

• Clients in some areas are highly mobile; they too have to accommodate different local arrangements if they move from one HRCC catchment area to another.

• Communication requires significant attention. Sometimes the right hand does not know what the left hand is doing.

• There is also the potential for duplication of services or for services being provided at greater cost than necessary (e.g., in one site visited with its multiplicity of EAS contractors).

D. Target Setting

Targets for the number of participants, the proportions who are active versus reach-back, the number of all participants returning to work and unpaid benefits by current claimants who return to work are set by NHQ of HRDC. Targets for mutual clients are negotiated within B.C. Mutual client targets exist at the HRCC level and at the sub-HRCC level (MHR regions). This level of detail is not always contained in sub-provincial plans. The sub-provincial plans reviewed generally did not state how many clients are to receive which EBSMs, nor why. Local targets appear to be based on past experience. While no set targets are provided for each EBSM, HRCCs use last year's numbers as a rule of thumb for planning and setting targets for the current year.

In general, targets are not internally consistent. There is no link between the types of clients targeted (imposed by NHQ or negotiated), demand for EBSMs (determined by the mix of clients and their needs) and the supply of EBSMs (estimated using last year's numbers). As well, concerns were voiced in all three regions about the lack of follow-up regarding whether or not mutual clients actually receive services.

E. Client Flow

Client flow is another area where local autonomy has been used successfully. In one site, all unemployed (EI-eligible, uninsured participants and IA clients) are streamed through ARCs for unassisted (labour market information (LMI), access to equipment) or assisted (RTWAP, referrals to other service providers) services. This centralized help should facilitate client flow. Another example of good use of centralized services is the centre established for all French language services in the Lower Mainland.

Even in the best situations, client flow for mutual clients has been problematic. Each area, and in some situations sub-area, has developed its own process. Virtually all interviewees who addressed this question noted significant problems in client flows and a need to refine the process. However, most also felt that client flow has improved recently.

Mutual client flow starts with the Program Referral Officer (PRO), an MHR staff person. This is a new position created for the LMDA and staff had to learn their role on-the-job. A Mutual Client Plan Review Report by the Achieving Results Working Group (attached to a Management Committee memorandum (dated September 25, 1997) expressed concern about delays caused by the hiring process. At one site visited, the PRO was not in place until mid-June 1998. Such delays may have resulted in services to fewer than the targeted mutual clients

Only 12% of clients in 1997-98 have been mutual clients, versus the 38% identified in the targets established for the LMDA. Data from our profile of clients indicates that the proportion of mutual clients has been increasing. By quarter, the proportion of mutual clients is:

• April 25, 1997 to June 30, 1998 — 9.6%.

• July 1, 1998 to September 30, 1998 — 8.4%.

• October 1, 1998 to December 31, 1998 — 12.0%.

• January 1, 1999 to March 31, 1999 — 14.1%.

The proportion of mutual clients, although growing over time, remains significantly below the targeted percentage and is unlikely to reach its target level in the near term.

As well, data on mutual clients does not get distributed because of system problems. Service providers have client-level data, but it does not flow well to local offices nor to the provincial/federal level. One site visit site is still tracking mutual clients manually. To overcome system problems which limit the distribution of client information, a plan under consideration at the time of our field work would see 75 to 80 co-ordinators faxing their lists to the HRCC weekly.

F. Contracting Process

The contracting process is one of the most frequently cited areas of discontent. The two cultures of the federal and provincial governments are poles apart with respect to contracting, and merging them has proved a major stumbling block. Issues include the level of decentralization and local signing authority, the amounts of money available for funding services and, thus, ownership of programming, the different approaches toward accountability and different time frames. To add to the difficulties, some within MAETT, and particularly MHR (because it does not have direct program funding), feel they are not "equal partners" in the joint contracting process.

Most of the contractors that we spoke with have had contracts with HRDC for some years. HRCCs usually award contracts based on previous experience. This has been raised as an issue of concern. In one site, the direct award of one contract that included an agency having no prior experience created discontent in the community. Such practice is also seen as HRCC favouring one service provider over others by some.

HRDC is highly decentralized. HRCCs have authority to directly award contracts of a significant size. Staff of the provincial government have lower spending authorizations. Provincial contracts are awarded using a more formal competitive bid process.

The result is that there have been no joint contracts due to financial and legal difficulties. In some situations the federal and provincial government may contract with the same organization. However, they use two separate or "parallel" contracts. Each contract will typically share common start and end dates but will vary in terms of value and possibly services contracted for.

Some sites provide examples of HRDC and MAETT working together to provide a "one stop" service for both EI and IA clients. However, although there may be one contractor involved, there continues to be two separate contracts for the service with the federal and provincial governments due to impediments to joint-contracting.

Contracts generally contain a clear statement of performance expectations (e.g., targets, follow-up requirements). Few contractors felt that their roles and responsibilities were not clearly set out. However, at one site there was a problem regarding Contact IV (an HRDC software program used by third party service providers). At first it was not understood that use of Contact IV was mandatory.

G. Data Collection, Monitoring, Tracking

There is an Information Sharing Agreement in place between the two governments. It came somewhat late in the process of LMDA implementation and its absence in the early months caused problems. There still appears to be some differences in the way the Agreement is interpreted and some fear that mutual clients who decline to sign the information release form may be denied services. Our participant survey did not find any support for this concern. Only one mutual client reported that they did not qualify or were turned down for support under an EBSM. However, our comparison group survey did not ask for similar information. We are not able to assess whether eligible non-participants have been turned down.

The lack of an interface between the computer systems of the two governments/three departments causes additional problems in sharing information.

Information gathering has improved over time. It appears that all government staff and contractors are now aware that use of Contact IV by contractors is mandatory. At the onset, a major expense for contractors was new computers. Contractors have now upgraded their equipment to handle Contact IV so that most technology problems have reportedly been overcome, and contract staff have been or are being trained by HRCC staff in the use of Contact IV. Increased familiarity with Contact IV has resulted in improved information gathering. Nevertheless, there are still some manual systems in place to overcome some system connectivity problems.

There is still concern regarding the completeness, reliability and accuracy of some data. For example, there is no agreement or consistency in terms of the unit of measurement to be used (intervention/client), nor are there definitions for specific units (e.g., what constitutes a contact-telephone call/in-person contact).

There are also still some major problems with systems. These have frequently been highlighted by staff and contractors and are well known. For example, systems are being used to do things they were not designed for such as to "communicate" with each other. There are also insufficient fields and procedural problems with the sequencing of inputs (some things need to be done before others can be done).

Reporting is still a problem. The software in use does not produce the reports that some managers need for day-to-day management. Other reports are produced by NHQ but are not timely. Their accuracy is questioned by some.

HRDC's ongoing management and performance data do not allow it to identify how many/which clients are receiving which EBSM. Thus, without a major effort such as a survey, it is not possible to evaluate the effectiveness of these measures as determined by outcomes.

H. Program Delivery

Partnerships exist at various levels among the co-management partners-at the senior management level, through the joint structures and locally. Also at the local level, community partnerships may exist with employers and community groups.

At the senior level there are problems among all three co-management partners-not just federal/provincial but intra-provincially as well. The joint structures are variable in terms of the effectiveness of partnering. For example, the Achieving Results Working Group worked well, but others do not-for reasons which include inadequate resources and frequently changing memberships.

In terms of co-operation and partnerships among co-managers at the local level:

• In many instances, HRDC, MAETT and MHR staff already knew each other and worked well with each other. The level of contact pre-LMDA varied from area to area.

• The LMDA has required these persons to work more closely together.

• Many government interviewees have learned through this process how their counterparts do business. This has been helpful and educational, although the process was not always smooth initially.

• In at least one situation, some interviewees felt that the LMDA has formalized relationships that already existed. In doing so, the increased structure and formality brought about a deterioration in those working relationships.

• A lot more could be done to promote partnerships. Staff are aware of this and note it in their plans.

Co-operation with employers at the local level in the form of community partnerships are not common and seem to depend largely on relationships between individual staff persons and employers. Employers are not automatically brought into the consultation and decision-making loop.

Co-operation with community groups in the form of community partnerships is happening to some extent. However, it varies from area to area. Most of the community groups involved in consultation and partnerships to date appear to be either the same as, or closely allied with, third party service providers. Community groups, such as chambers of commerce and city staff, are not regularly incorporated in consultation and decision-making processes.

Processes to support community partnerships vary from area to area, as do their effectiveness. Processes include:

• Supportive government (primarily HRCC) staff. Such staff promote open consultation where views are sought, listened to, respected and incorporated in decision-making. Staff may also participate in community or corporate committees/activities.

• Flexible programming. Service providers design the type of programming that is most effective for their clients.

• Establishment of LLMPs.

LLMPs provide encouragement and financial support to establish community partnerships among local HRCCs and community groups. These partnerships may address issues such as community economic development and local labour market adjustment.

Partnerships under an LLMP are established in one of two ways:

• Organizations with an idea for a project will develop the idea and submit a written proposal to HRCC seeking partial support.

• Those co-ordinating the LMDA (HRDC/MAETT and MHR) will meet with local community groups to discuss the possibility of an LLMP. A proposal will emerge for partial support.

I. Third Party Service Providers

Program delivery is primarily by third party service providers. Some of these are for-profit, some not-for-profit. Issues expressed by third party service providers include:

• Some felt that their roles and responsibilities were clearly spelled out in their contracts; others felt that they were not and that they were evolving over time. A particular concern, in one area at least, is that the obligation to track using Contact IV was not clearly communicated at the onset so this requirement was viewed as a retroactive imposition for which they were not paid.

• They are responsible for finding clients (marketing), dealing with referrals, providing the EBSM contracted for, referring if necessary to another service provider, case management and tracking.

• Marketing is new to many service providers, especially to not-for-profit providers.

• Some believe that the provincial Request for Proposals (RFPs) model means a lot of money is being spent on RFPs and good proposal writers may win the contracts even if they are not necessarily good program providers.

In the three field sites, virtually all programs are delivered through third party service providers. HRCCs have downsized. The counselling units that used to exist have become responsible for contract management. Third party service providers have become the delivery arm.

Some EAS services are still delivered by internal HRDC staff. However, most are delivered by third party service providers. In our survey, 63% of participants said they accessed their EBSM through an HRCC, 31% identified a third party service provider and 6% did not know whether it was a government or third party service provider. By area, those from the North (69%), including Terrace (67%), were more likely to identify services being accessed from an HRCC. By EBSM, an HRCC was identified for 73% of TP, 65% for EAS, 52% for SE, 46% for TWS and 41% for JCP. This perception by clients does not align well with observations in our site visits. Potentially, clients are not able to perceive differences (nor do not care) between government and third party service providers. Another possibility is that clients go to HRCCs and are referred to services of third party service providers. As observed in one case study site , they may not realize that service providers are not part of the government.

Some contracts with third party service providers target services to specific groups including those with special needs. In some areas, service providers are required to provide services for all clients. When more than one service provider competes for the same client base, there is the potential that money will be spent on advertising to attract clients instead of to assist clients to get back to work.

Linkages have been established between service providers in some areas to better handle flows of clients among them. In one study site, third party service providers have formed a group to focus and address their concerns about the local co-management of the LMDA.

J. Duplication/Overlap

Overall, the level of duplication may be slightly less or at least the same as the situation prior to the LMDA.

There are three potential areas of duplication/overlap with the LMDA. The potential for duplication exists:

• With provincial programs. While there may not be direct duplication between the different federal and provincial programs available because eligibility criteria vary, there is still a great many programs serving similar clients, some of whom may be eligible for two or more of these programs. In our comparison group survey, 20% had taken one or more training programs since 1997. For 62% of the participants taking training, the cost of training was paid in whole or in part by some other organization, but not the LMDA.

• Through the federal and provincial governments using the same contractors. Attempting to reduce duplication through joint contracting has not been successful because of the different cultures and approaches to contracting. Parallel contracts have been used as a way to address these difficulties.

• Through having a large number of contractors providing the same service within a geographic area. Overlap and duplication of contractors appear to have been reduced in some areas with the introduction of the LMDA. In others, there has been no change; while in others it appears that there is more duplication than before. To illustrate the potential for greater overlap, in at least one area, all service providers had to bid on providing EAS services to all clients, rather than being allowed to concentrate on specific client groups. In one case study site there are 17 EAS service providers. At the time of the interviews, the HRCC was starting to address this problem and to streamline EAS contracts.

There has also been an increase in the number of LMDA-related meetings and co-ordinating/co-management activities that staff have to get involved in. This is particularly the case where boundaries between the various offices do not coincide. Thus, for instance, staff in one MHR office are obliged to deal with three HRCC offices. MAETT and MHR boundaries are also not aligned leading to further co-ordination effort. This lack of commonality across organizations creates a large amount of extra "non-productive" work on top of regular work loads.

K. Clients

The EBSMs collectively offer a broad range of RTW activities. In addition, there is a significant level of flexibility at the local level in interpreting how these EBSMs are applied to meet local needs. The qualitative data suggest that the EBSMs are highly relevant to the employment needs of clients.

From the profile, 15% of the 25,752 clients had received multiple interventions between April 25, 1997 and March 31, 1998. This includes 14% who participated in two interventions, 1% in three and less than 1% in four or five interventions. Combinations may include activities such as counselling and referral (EAS) to a training program, the training program itself, and then perhaps a return to EAS for a job finding club or résumé preparation.

We were advised that most multiple interventions are sequential, as described above. We did not find any instances of a client concurrently receiving, say, TWS and TP.

The only aspects of multiple interventions that may not be efficient are referral from one agency to another for services, and case management by an agency other than the one providing the service. (We were advised that some for-profit agencies get not-for-profit agencies to do the case management.) However, without having each agency provide all services (which would result in increased duplication), inefficiency will remain.

Some HRCCs, such as Coquitlam, have a case management centre that case manages all of the HRCC's clients. This may have benefits (e.g., it may be more efficient and less confusing for service providers and clients). However, some interviewees who work with Coquitlam and other Lower Mainland HRCCs suggested clients are less well served and are less likely to receive EBSMs other than EAS. Without reviewing this type of case, we are unable to comment on benefits, disadvantages and efficiency issues.

L. Employers

The lack of community consultation extends to employers in the community. We found that employers in our focus group sessions were not well informed about the LMDA or about co-management. Although we did not contact employers who were not involved in the LMDA, we suspect that lack of awareness and not the characteristics of EBSMs is the factor limiting participation most.

Those who participated in the focus groups generally felt that TWS met their needs. All employers knew that TWS implied an obligation for continuing work after the completion of the subsidy; however, some treated it as an opportunity to hire staff on a short-term, subsidized basis. Employers emphasized that the subsidy allowed them to train staff without having to pay the full salary during that period. A number of employers noted that in some industries it takes far longer to train a staff person than is implied by the average length of a TWS. However, TWS is intended to provide an incentive to train not to displace full training costs.

M. Information and Communications

1. Labour Market Information Used By Participants

The unemployed have a variety of employment resources available to them. In our survey we asked whether participants had made use of these resources on their own (i.e., on a self-serve basis). Eighty-two per cent had. The most frequent resources used by these participants were newspapers (66%), job banks at HRCCs (62%) and the Internet (42%) (see Exhibit 3).

Internet Resources

More detail was sought from the 42% of participants who identified using the Internet. A higher proportion of those using the Internet had the following characteristics-they had a disability (63%), were still involved with their program (56%) or were from the Lower Mainland (53%). The Internet resources most frequently accessed by those individuals who used the Internet were BC Work InfoNet (37%), job listings (other than Electronic Labour Exchange) or résumé posting services (31%) and the Electronic Labour Exchange (24%).

Those using printed or on-line (Canada WorkInfoNet or BC Work InfoNet) career and work search information were also asked for their opinions on the quality and ease of use of these sources. Ease of use was assessed on a scale of one to seven, where one was "not at all easy" and seven was "very easy." The average score was 5.3.

Printed Career And Work Search Information

More detail was also sought from the 14% who identified using printed career and work search information. Those using this printed material were more likely to have a post-secondary education (18%) and to have a household income between $20,000 to $40,000 (18%). Principal items searched by those using printed career and work search information were Work Futures/Job Futures (38%), The Job Guide (32%) and Career Paths (22%). In addition, 22% were not able to identify specific titles searched.

Those using printed career and work search information accessed it from a variety of sites. However, fully two-thirds (68%) report obtaining this information from an HRCC.

Technology Provided For Use

More detail was also sought from the 15% of participants who identified using technology provided for them by the HRCC/service provider. Those more likely to use the technology which was provided had the following characteristics-they were mutual clients (22%), 35 to 44 years of age (19%) and were of Aboriginal descent (18%). The technologies used most frequently were computers (90%), telephones (51%), fax machines (49%) and printers (42%).

Satisfaction with the quality of self-service tools was rated on a scale of one to seven, where one was "not at all satisfied" and seven was "very satisfied." The average score was 4.9.

Most (55%) had used these products within three months of the survey-73% had used them in the last six months. Users were also asked if they would use the tool again. Ninety-three per cent said they would.

2. Significant Differences in Labour Market Information Use by Participants

We asked a more limited set of questions on the employment resources used by comparison group members. We then assessed differences between participants and comparison group members in terms of their use of LMI through regression techniques. Regression models controlled for potential differences in the demographic and other characteristics between participants and comparison group individuals. Using regression techniques, we found a significant difference² in the use of:

We asked a more limited set of questions on the employment resources used by comparison group members. We then assessed differences between participants and comparison group members in terms of their use of LMI through regression techniques. Regression models controlled for potential differences in the demographic and other characteristics between participants and comparison group individuals. Using regression techniques, we found a significant difference in the use of:

• The Internet. Participants were 12% more likely (level of significance less than 0.005%) to have used the Internet as a result of having participated in an EBSM.

• Printed career and work search information. On average, participants were 5% more likely (level of significance 1.6%) to use this printed information than they would have without the EBSM.

• Provided technology. On average, participants were 4% more likely to use provided technology (level of significance 4.1%) as a result of having participated in an EBSM.

Participation may be responsible for these differences. Alternatively, some other unmeasured factor, such as motivation, may account for both higher use and participation in an EBSM.

3. Communication with Community, Employers, Clients

Section 13 of the LMDA notes that Canada and British Columbia agree to work jointly on preparing public information material and organizing related public announcements; as well as developing mechanisms to inform Canadians of the activities undertaken and the results achieved in the context of the Agreement.

As noted earlier, there has been limited communication with community groups.

In our focus groups, most employers knew about EBSMs through word-of-mouth. Other sources included potential employees who knew of the programs, advertisements in local newspapers, information at trade shows and from third party service providers.

Employers in focus groups say that some of the best ways to communicate with them are by advertisements and articles in local newspapers, information sessions held by HRCCs, and through unemployed individuals who visit them. Generally speaking, employers do not want telephone calls or television advertisements. Many do not belong to industry associations eliminating this as a useful avenue to approach non-associated employers.

To communicate with clients or prospective clients, a wide variety of communication tools are used, including posters, EI cheque inserts, advertisements in newspapers, posters in public places and on the Internet. From our survey of participants, a wide variety of sources of information were identified:

• HRCC offices were identified as the source for information on their EBSM by 45% of our participant survey respondents.

• A third party service provider was identified exclusively by 14% of participants and in combination with government by 6%.

• A provincial or municipal office was the source of information for 5% of all respondents and 12% of mutual clients. In our survey, 21% identified word-of-mouth as their source of information on the EBSM they participated in.

• Other mentions included newspaper (7%), union (2%) and all other mentions (5%).

Respondents rated the information they had received highly. Sixty-six per cent found it very useful and a further 25% somewhat useful. There was little difference in the positive responses given across location or client characteristics. Those who participated in TP or SE rated the information they had received slightly more positively. Those who received EAS or JCP rated the information received slightly less positively.

For clients in our focus groups, the majority claimed that they initially heard of EBSMs through word-of-mouth; others heard through personal investigation, school, another referring agency and the Internet. An HRCC was not mentioned as the initial source of information. Clients interviewed in the focus groups noted the following as the best means for communicating with them:

• Inserts with EI cheques. (However, they note that the insert must look different from regular information and be eye-catching.)

• Advertisements in local newspapers, the Free Employment Paper and on the Internet. They suggest that there should be one central information centre per community rather than many.

Most clients in the focus groups felt that they had enough of the necessary and appropriate information they needed. However, while we had a mix of active, reach-back and mutual client participants in the focus groups, we note that:

• All but one of the participants in the focus groups had English as their mother tongue; the one exception spoke English well.

• All had reasonable levels of education.

• All could read.

Thus, the focus groups did not provide information on clients who do not speak English, clients whose mother tongue is French, clients with low education levels and clients who cannot read. Interviewees suggested that some clients may not be able to access the information they need. However, efforts are made to accommodate such clients. For example, information sessions are held, where much of the information is in oral form, sometimes in a language other than English or French. French-speaking clients are referred in the Lower Mainland to 10th Avenue, which provides services in French for all Lower Mainland HRCCs. In some cases, clients from non-English/French speaking groups are referred to service providers who provide services in languages such as Mandarin, Cantonese or Spanish.

N. Case Management

There appear to be different definitions of what constitutes case management. For example, some see it as a means to ensure interventions are client-driven; others use it for follow-up and reporting purposes.

Despite the differences in interpretation, case management is widely used, primarily by third party service providers. Not all clients are case-managed-some, particularly those going through short interventions, are not. Usually, however, those going through longer, more complex, or multiple interventions that involve an action plan are case-managed. The agency that provides the intervention is not necessarily the agency that does the case management. This may result in some duplication. Some expressed concern that for-profit service providers were relying on not-for-profit groups to do their case management.

Clients in the SE program use their business plans as action plans. They are reviewed regularly and provide a basis for progress reporting and feedback.

O. Negotiated Financial Assistance

Clients are expected to contribute to their RTWAP if they are able. NFA is the process whereby staff or a third party service provider identify the client's resources and the cost of the RTWAP and then negotiate a sharing of these costs, if appropriate.

The extent to which NFA is used varies by area, HRCC, third party service provider and even within each specific contract. The amount of knowledge of NFA also varies.

Many, but not all, third party service providers have received training or assistance on NFA from their HRCC staff. The amount and depth of training varies significantly. At least one service provider interviewed had requested but had not received NFA training. At least one more had never heard of NFA.

At least one HRCC held a session with some third party service providers to ensure some consistency in the application of NFA. However, we note some remaining inconsistency in our qualitative interviews. What people include in NFA differs. For example, some say having a family member baby-sit would be counted; others suggest this would not be applicable. Some interviewees note that NFA does not apply to mutual clients. However, this interpretation is not universal. We think this lack of consistency may be in part as a result of the flexibility available to staff and delivery agents.

There is no recording of the savings realized through the application of NFA. In particular, neither third party service providers nor HRCCs capture information related to in-kind savings realized.

We asked specific questions about the sharing of costs of their intervention with the 251 TP participants in our survey. Seventeen per cent were not able to say who paid. A further 1% identified no costs. It is possible that NFA was not conducted with either group. The responses of this minority are excluded in the following discussion related to NFA.

Five per cent of the participants under TP said they paid all of the costs of their training. A further 46% said they paid part of the costs, while 49% identified paying nothing.

Those most likely to contribute to their training costs were those 45 years of age or older (76%), those with household incomes of more than $40,000 (74%) and those with a grade 12 education (68%). These groups are apparently better able to contribute to their RTWAP.

Ninety-three per cent of those who reported paying part or none of the costs of their training said they would have been unable to take the training without this support. The proportion was highest for those with less than a grade 12 education (100%), mutual clients (100%), members of an under-represented group other than females (100%), those under 35 years of age (97%), reach-back clients (97%) and those with household income under $20,000 (96%). The proportion was lowest for those 35 to 44 years of age (86%), those who had taken training more than one year ago (88%) and those with household income of $20,000 to $40,000 (88%).

The 78 comparison group members who had taken a training course since April 1997 were asked whether they had paid all, part or none of the costs of training. Eight per cent said there were no costs and were excluded from the following analysis. Forty-two per cent said they had paid all, 8% said part and 50% said they paid none of the cost of their training. Comparable numbers for participants were 5%, 44% and 51% respectively. Although those who reported paying none of the costs of training are almost identical across the two groups, participants appear to share the costs of their training in much higher numbers. NFA appears to result in a sharing of training costs.

Seventy-four per cent of those who received funding support for their training say they would have been unable to participate in the training without this support. Participants receiving partial or total support under NFA more often (94%) said they would have been unable to participate without support. This suggests that NFA has been successful at assessing ability to pay for training. However, it is also possible that participants were less inclined to admit they could have undertaken training without support in a survey to evaluate the LMDA.

P. Efficiency And Effectiveness

Some interviewees noted efficiencies through the LMDA. Parallel contracting, where HRDC and MAETT contract with the same contractor for complementary services over a similar time frame, should result in some efficiencies.

Some interviewees feel there are too many third party service providers available. Fewer contracts would reduce contract management activities, reduce overheads and may offer a better service at less cost.

Contractors say that they now have to compete with each other through advertising to maintain their share of a finite client base, rather than spending that money on clients. This tends to increase costs with no benefit to clients.

Short contract lengths, possibly as a result of uncertainties related to the future of the LMDA, have meant that contractors are forced into shorter term and more expensive leases and rental agreements for equipment. Another problem is the uncertainty it creates for staff-many of whom may leave in search of more secure positions.

Some contractors would prefer to focus on one specific group of clients; but HRCC staff have "forced" them to bid on providing services to all clients. This is inefficient as it forces contractors to deliver services at higher costs to a client group for which it does not have a comparative advantage. It appears most HRCCs do not make any special distinctions for under-represented groups. One case study site, for example, used to have a contract especially for persons with disabilities, but it was not renewed this year. It appears to be left to the third party service providers to accommodate the special needs and requirements of under-represented groups.

This chapter assesses short-term impacts and outcomes for participants. Recall that to be eligible, participants need to be unemployed and may be on EI Part I, if an active claimant, or have exhausted either a regular or maternity/paternity claim in the past.

A. Assuming More Responsibility

The EBSMs encourage clients to assume more responsibility for the following under the Employment Insurance Act:

• Identifying employment needs.

• Sharing in the cost of assistance, if appropriate.

• Increasing employment.

• Reducing their dependency on EI/IA.

As could be expected, views varied widely from area to area and from agency to agency within an area regarding achievement of this goal. Factors that affect views include the type of client dealt with (e.g., whether the agency deals primarily with clients who are not job-ready), the prevailing economic conditions and the availability of jobs, especially entry-level or lower-paying jobs.

Where there are jobs available, EBSMs are seen as reducing dependency. Where there are no long-term jobs, EBSMs are seen as being less successful in securing employment. However, they are still seen as making clients more marketable and better able to market themselves.

Action plans are seen as assisting or even forcing clients to take more responsibility for their employment. Those clients who are case-managed generally have a RTWAP. The plan may be prepared by themselves, alone or with the help of a counsellor, or prepared by the agency. In most cases, however prepared, the client and counsellor are required to sign off on the plan. In most cases, it appears that the plan is used to measure and track progress.

Some interviewees note that many clients, especially those receiving training or self-employment assistance, have contributed to the cost of their assistance; some have done so in a significant way and incurred large debts with no assurance of work in the future. Other interviewees note that clients either do not contribute (especially mutual clients) or contribute nominal amounts or make in-kind contributions only.

We asked questions in our participant and comparison group member surveys designed to identify if there was a difference between the level of responsibility of the two groups related to labour market issues. Differences between the two groups may be attributable to participation since the two groups were drawn to be statistically similar in observable characteristics other than participation. Note, however, that any differences may be due to other unobservable factors. For example, we cannot rule out the possibility that some hidden factor, such as motivation, is attributable both to participation and to any difference between participants' and comparison group members' responses to these questions. Findings are indicative but not definitive of an attributable impact of the EBSMs.

Overall, participants seem slightly more positive than comparison group members on questions related to the level of responsibility individuals have toward identifying their employment needs. They may take more responsibility for identifying these needs.

There is no difference between the two groups on the question asked about sharing in the cost of assistance. There are no significant differences between participants and comparison group members in terms of their willingness to pay for a training course. As we have seen earlier, both groups are also similar in terms of the proportion paying at least part of their costs of training. However, comparison group members are far more likely to pay the full cost of training.

Findings across the two groups suggest both share similar levels of responsibility for increasing employment. Questions common to the participant and comparison group questionnaires probed for factors which might be related to a heightened responsibility for increasing employment. Responses suggest participants' and comparison group members' views are similar across these statements.

Participants may be more responsible for reducing their dependence on EI/IA through their involvement with an EBSM. Participants' responses were slightly higher than comparison group members on a question asked to assess individuals' responsibility toward reducing their dependence on EI and IA. The regression analysis confirms that participation in an EBSM affects the importance placed on being independent of EI or IA. Again, however, we cannot rule out some unobserved factor, such as motivation, which may affect both participation and responsibility.

B. Discontinuation

Of the participants surveyed who were not still involved in their EBSMs, 16% report discontinuing their participation in the EBSM before it was completed. Almost one-half of the reasons for discontinuation were positive-to get a job. In 24% of cases, discontinuation is a negative result of the program-the withdrawal of funds or dissatisfaction. Of the participants who discontinued their EBSMs, almost one-half (48%) report doing so because they found employment. Other reasons why their programs were not completed include funding reductions (15%), maternity leaves (8%), lack of program satisfaction (7%), family responsibilities or medical reasons (6%), asked to leave (4%), difficulties with employer (4%), employer could not afford (2%) or other reasons (5%).

C. Satisfaction

A majority of the participants whose EBSM involvement was finished at the time of the survey give their EBSMs a positive satisfaction rating. More than three-quarters (79%) rate their EBSM as "5" or more on a seven-point scale where one means "very unsatisfied" and seven means "very satisfied." The average rating given to the EBSMs by these participants is 5.6. The average rating for participants taking TP and SE was 5.9 and 5.8, respectively.

Using a seven-point scale where one means "very unuseful" and seven means "very useful," participants rated the usefulness of their EBSM to them in obtaining full-time employment. The average was 3.9. TWS was highest rated at 4.9 and SE and TP were rated at 4.2 and 3.8, respectively.

Participants also gave their EBSMs positive ratings in terms of their usefulness in six areas. On average, these participants rated their EBSMs as being most useful in terms of building their confidence (5.0), focusing their career goals (4.9) and learning how to do a certain job (4.6). They were also considered to be useful, though to a lesser extent on average, at helping participants develop job skills such as managing time and organizational skills (4.0), obtain full-time employment (3.9) and improve or reach their earning potential (3.9).

D. Attitudes to Work

Most of the participants who had finished their EBSM report that their attitudes toward work and the future have become more positive (59%) or stayed the same (32%) since their EBSM ended.

E. Further Training

Twenty-three per cent of participants have taken further training or courses since their EBSM ended. Almost one-half of these participants (48%) mention taking this training as a result of their involvement in their EBSM.

F. Employment

Seventy-five per cent of participants who had completed or discontinued their EBSM or who were self-serve only clients have been employed subsequent to their involvement with the LMDA. For active claimants and reach-back clients, the percentage who have worked in their post-EBSM period was 75% and 77%, respectively. For members of the comparison group, 69% had worked after the date when the participant they were chosen to represent was scheduled to complete their intervention. For comparison group members chosen to represent active claimants, 70% had worked; and for those chosen to represent reach-back clients, 66% had worked.

Exhibit 4 compares the characteristics of jobs held by participants and comparison group members in the post-EBSM period. Most jobs held by participants (84%) or comparison group members (91%) were with employers. The exception was for participants of SE where 64% of jobs in the post-program period were in business for themselves. Excluding participants under SE, 92% of jobs for participants are with employers-comparable to the comparison group findings.

As Exhibit 4 illustrates, the job experiences of participant and comparison group members are highly similar. Notable exceptions are the income from employment and employment status. These are explored in more detail in Exhibit 5 for participants who were current claimants and the reach-back clients and the comparison group individuals who correspond to them. (Note that other characteristics presented in Exhibit 4 do not vary appreciably by client status.)

Exhibit 5 Selected Characteristics Of Post-EBSM Period
	Participants			Comparison Group
	Total	Current Claimants	Reach- Back	Total	Current Claimants	Reach- Back
Working At Some Time	75%	75%	77%	69%	70%	66%
Average Income Earned In Typical Week	$466	$466	$464	$538	$555	$485
Currently Working	65%	65%	67%	55%	55%	54%

As noted in Exhibit 5, participants are more likely to work in the post-EBSM period-75% compared to 69% for comparison group members. Reach-back clients are most likely to have worked (77%) while the group chosen to compare to them are least likely to have worked (66%)³ . This observation also holds for the percentage employed at the time of the survey. However, those comparison group members who do work tend to have higher average weekly earnings than participants who work.

Exhibit 6 compares the pre-EBSM period experience for participant and comparison group members. The pre-EBSM period is defined as the year preceding the start date of the EBSM for participants. For comparison group members, it is the year prior to the start date of the intervention for their matched participant. The exhibit illustrates data for all participants, active claimants and reach-back clients, as well as data for comparable groups from the comparison survey.

As the exhibit illustrates, our comparison group well matches participants in terms of their pre-EBSM labour market experience-especially for active claimants. However, what is striking is the reported labour market experience of reach-back clients. Fifty-seven per cent of participants who are reach-back clients report working in the year prior to their EBSM. Jobs are full time (average 39 hours per week), most are for an employer (84%) and lasted on average 15 months.

Comparison of Exhibits 4, 5 and 6 suggest:

• Participants did better than comparison group members in the post-EBSM period compared to the pre-EBSM period in terms of the percentage working. Reach-back participants had the largest improvement from 57% to 77%. Comparison group members, who were chosen to represent reach-back participants, experienced a drop in the percentage employed from 75% to 66%, in a similar period. Current claimant participant employment dropped from 78% to 75% between the pre- and post-EBSM periods. For comparison group members chosen to represent current claimants, the employment rate dropped from 78% to 70%.

• Results are not as unequivocal in terms of income earned in a typical week for those who are working. Reach-back participants have increased their earnings relative to the comparison group members. Reach-back participants gained $20, on average, compared to a loss of $110 for the reach-back comparison group between the pre- and post-EBSM periods. Current claimant participants, on the other hand, lost $59 compared to a gain of $60 for the relevant comparison group members⁴ .

We also assessed specific employment impacts for some EBSMs through our survey:

• Fifty-five per cent of participants ending TWS interventions continued to work with their TWS employer.

• Eighty-six per cent of participants ending SE interventions started their own businesses. Of those who had started businesses, 96% were continuing to run their business when surveyed and 25% had hired one or more employees to work with them.

G. Dependence on EI and IA in Post-EBSM Period

Exhibit 7 provides a summary table of the post-EBSM dependence on EI and IA of participants and comparison group members. In the post-EBSM period, 40% of participants and 37% of comparison group members had received EI. The figure for participants will include those still on claim when their EBSM ended and may include those whose EI was "extended" (EI Part II income support) through participation in EBSMs after the one selected for our survey. Note that 50% of participants who were current claimants received EI in the post-EBSM period, compared to 40% of comparison group members selected to be like current claimants.

Nine per cent of participants and 5% of comparison group members received IA in the post-EBSM period. The principal reason identified for being on EI or IA was a lack of work available. Seasonal layoff as a reason mentioned was higher for our comparison group members.

The percentage on EI was lower for participants (10%) than comparison group members (12%) at the time of the survey. Months on EI and benefit rates were comparable across the two groups. The only outlier is the current rate on EI for TWS participants which stands at 25% compared to 9% for all EBSMs. A possible explanation is that time spent on a TWS increases EI eligibility.

The percentage on IA at the time of the survey remained higher for participants. Average IA amounts were similar across the two groups. The average number of months on IA was less for participants compared to comparison group members on IA at the time of the survey.

H. Short-Term Incremental Impacts on Earnings, Hours Worked and Dependence on Income Support

We used regression analyses to identify whether participation in an EBSM had a significant effect in the short-term on the experience of participants in terms of their:

• Earnings per year.

• Hours worked per year.

• EI benefits received per year.

• IA benefits received per year.

This analysis uses data for a matched comparison group of non-participants to predict what would have happened to participants had their interventions not occurred. Individuals for the comparison group were chosen to reflect the characteristics of participants (determined through administrative sources) other than participation. The regression models compared the actual and predicted experiences, the difference reflecting the effect of participation in an EBSM.

These models corrected for selection bias. It is possible that some unknown factors, such as motivation, may be related both to participation and to outcomes. It is this relationship that creates selection bias. However, the modelling approach that we used was designed to correct for the effects of such bias..

A number of demographic and other variables were used to control for remaining differences between participants and comparison group members, including pre-EBSM values of some of the outcome variables.

These are tough tests for the EBSMs. Typically, EBSMs remove the participant from the labour market for a period of time. Over this period, their matched comparison group member remains in the labour market, may be able to get a job and begin to increase their employment earnings. Participants may need to catch up to comparison group members when their EBSMs are completed. Regression analyses are generally reserved for summative evaluations where the longer term analysis provides for a fairer test of the EBSM impacts.

Exhibit 8 presents significant findings from the short-term regression analysis related to the above outcome variables. Separate models were used for all EBSMs, all EBSMs excluding EAS, each separate EBSM, active claimants and reach-back clients. For each model, and outcome variable which are significant, we list the estimated value for the variable and the level of significance⁵ of the estimate.

Across all EBSMs, the annual EI benefits received by participants are estimated to have increased by $663 as a result of participation in an EBSM. For current claimants, EI benefits received on an annual basis increased through participation by $1,203. For reach-back clients, EI benefits received decreased by $798 compared to what would have happened in the absence of the intervention. For employment benefit measures only (EAS is excluded), there are no significant impacts on EI benefits through participation.

Participation did not result in significant changes in earnings, hours worked or IA benefits on an annualized basis across all EBSMs or all employment benefits. Note, however, that hours worked increased significantly for reach-back clients. Recall that the post-EBSM period is relatively short in the formative evaluation (typically less than one year). Participants have likely not reached their full labour market potential so soon after the intervention. The intervention also may have effectively removed the participant from the labour market compared to what would have happened in the absence of the program. It may take some time for participants to recover from this labour market absence. (The summative evaluation can provide evidence of the longer-term impact of participation.) Because fewer observations are included in the models for individual EBSMs, it is less likely that these models are able to detect a significant change when one exists. Significant changes by individual program were:

• An increase in EI benefits received by EAS participants of $1,058 per year. (Recall EAS may provide access to other EBSMs. Participants who are current claimants may have their EI "extended" (EI Part II income support) through participation in other EBSMs.)

• A decrease in annual EI benefits received by SE participants of $1,060. (Recall that self-employment does not result in insurable hours.)

• An increase in hours worked per year by SE participants of 132 hours.

I. Impact of Negotiated Financial Assistance

Participants are encouraged to contribute to the cost of their RTWAP, if appropriate, through NFA. Sharing in the cost of training may bring positive impacts in terms of attitudes, labour market impacts and dependence on income support. This section assesses such potential impacts. Note that the number of participants paying all of their training cost is small. As a result, we merge this group with those paying part of their costs of training.

1. Attitudes Toward Work

Participants were asked their degree of agreement or disagreement with a number of statements related to attitudes toward work. The proportion that individuals paid of their training costs is a strong indicator of their willingness to pay for training to stay off government support. Those who paid at least part of their training costs are more likely to think they have a number of good qualities. They are also much more likely to turn down a good paying job if they had to move to get it. Training may be seen as a way to avoid moving to get a good paying job. Those unwilling to move may be more willing to pay for training. Those who paid at least part of the costs of their training place more importance on being independent from EI and IA.

2. Labour Market Impacts

Participants who have paid at least part of their training costs were slightly less likely to be employed in the period after training. Seventy per cent of those who paid at least part of their costs of training were employed in the post-EBSM period compared to 76% who paid none of the costs of training. At the time of the survey, 62% of participants who paid at least part of the training costs were employed compared to 65% who paid none.

3. Dependence On EI And IA

The proportion on EI at some time following their training for those who paid at least part or none of their cost of training were 41% and 29%, respectively. The proportion on EI at the time of the survey was 6% for both groups.

Similarly, 14% of those who paid at least part and 7% of those who paid none of their cost of training were on IA at some point after their training. The proportion on IA at the time of the survey was 12% and 3% for those who paid at least part or none of their cost of training, respectively.

As a result, paying at least part of the costs of training does not appear to reduce the dependence on EI and IA.

J. Outcomes in Terms of Savings to the Employment Insurance Fund

In areas where there are no jobs, TWS may find short-term employment for participants who then become eligible for EI benefits again in the future. This may account for the high proportion of TWS clients on EI in the post-EBSM period (53%) and at the time of survey (25%).

This chapter reviews the LMDAs impacts on B.C. communities and its partners. In particular, it reviews community impacts in terms of changes in employment, social infrastructure, community involvement, resident mobility and volunteerism. It also reviews the impacts of JCPs specifically.

A. Impacts on Communities

1. Employment Impacts

The objectives of EBSMs are to assist individuals to prepare for, obtain and maintain employment. Communities are assisted when these objectives are achieved for the individuals who make those communities their homes. Employment results to date have been positive. Based on the results of a survey of 1,200 clients who participated in an EBSM before March 31, 1998:

• Seventy-five per cent of those no longer participating had found employment.

• Those who had finished participating under SE were most likely to have found (or created) work (90%).

• Eighty-six per cent of all SE participants had started their own businesses, of which 25% had hired one or more employees to work with them. At the time of the survey (September 1998), 96% of those who started a business were still running the business.

• Fifty-five per cent of those who had ended a TWS continued to work for their TWS employer when the EBSM support ended.

At the time of the survey:

• Sixty-five per cent of past participants were employed.

• The percentage employed by EBSM participated in ranged from 82% for SE to 44% for EAS.

2. Enhancement To Social Infrastructure

LLMPs are intended to increase the resources in a community devoted to social goals. To date there is little evidence to show that social infrastructure in communities is increased through EBSMs. There are specific instances of this occurring, such as a youth centre in one case study site. However, it is not clear how much of the youth centre is financed through HRCC EBSM funding and whether the centre would have proceeded had that funding not been available.

3. Community Involvement

It appears that, to date, city councils and other community organizations are not involved much with EBSMs. Other organizations, such as third party service providers and colleges, were involved with HRDC programming prior to the introduction of EBSMs. That involvement does not appear to have changed radically.

4. Mobility of Residents

Sixteen per cent of those participants contacted have moved since their intervention ended. The highest proportion of moves were in the North (21%), especially Terrace (24%). Fourteen per cent of comparison group members have moved since the date when the participant they were selected to represent should have completed their training. Areas with the highest proportion of moves were again the North (19%), including Terrace (18%). We cannot say, with certainty, that moves were outside of the community where the individual resided at the time of the intervention.

Thirteen per cent of the telephone numbers we attempted for participants were not in service compared to 19% for comparison group members. This higher proportion of disconnected services may indicate moves outside of the community. A more likely explanation is that the telephone numbers for comparison group members are less current than for participants.

Respondents to our surveys were asked to assess their degree of agreement with the statement, "I would turn down a good paying job if I had to move from my community to get it." A seven-point scale was used to assess agreement, where one means "strongly disagree" and seven means "strongly agree." Participants are slightly more likely to agree to this statement with an average score of 3.5 compared to an average score of 3.4 for comparison group members.

In our regression analysis participation in an EBSM was found to have a significant negative effect on mobility (at the 0.1% level of significance). Participants were 0.4 points (on a seven-point scale) more likely to want to stay in their community than they would have been without participation. Current claimants were more likely to want to stay (at the 0.2% level) by 0.5 of a point. There was not significant effect for reach-back clients. By EBSM, the only significant result was for SE (at the 0.1% level). Participation increased the reported agreement with the statement by 0.9 points on a seven-point scale. Potentially, participation under an EBSM is viewed as a substitute for moving from the community to obtain employment.

5. Volunteerism

Through our surveys of participants and comparison group members we identified the proportion of individuals who volunteered. Twenty-seven per cent of participants have volunteered for community organizations since their EBSM ended. This compares to 23% of comparison group members in the same period. The average number of hours per month spent volunteering in community organizations was 18 hours and 12 hours, respectively, for participants and comparison group volunteers. As a result, participants are slightly more likely to volunteer and spend more time per month, on average, volunteering. This greater volunteerism by participants may be as a result of participation in an EBSM. However, there could be other unobservable factors which influence both participation and volunteerism.

Forty-three per cent of those participants who had volunteered report spending more time volunteering now than they did before their involvement in their EBSM. This is stronger evidence of the effect of participation on volunteerism; however, it remains indicative but not conclusive.

6. Impacts Of Job Creation Partnerships

Job Creation Partnerships (JCPs) are community-based work opportunities projects. The objectives statement for JCP reads, "To create sustainable employment in the local community and to create meaningful work opportunities for unemployed clients."

JCP involves partnering with an employer or not-for-profit organization within the community. These partners are encouraged to top-up the EI Part II supported wage to the prevailing levels for the occupation in the community. Overhead, equipment and training costs may be provided for not-for-profit partners.

JCP is not a high profile EBSM. Staff time required to market and to form partnerships with community members has limited its use. A requirement for ministerial approval for HRDC contributions in excess of $150,000 has apparently not been a factor in limiting its implementation. The average contract award is less than 20% of this limit.

Examples of the type of activities supported under JCP include:

• In one case study site, one of the JCPs has been helping clients enhance their skills. It has helped clients with self-confidence, development of job specific skills and more general work skills. It provides them with work experience along with how to find contract work and create jobs on their own.

• In another case study site, one of the JCPs has been helping clients become trained and continue to work. It has helped clients by training them for new jobs in their old company (from which they had been laid off the previous winter). This led to (more) permanent positions for two individuals. However, additional training will be needed by these individuals in the future.

The first site has many JCPs, particularly with not-for-profits organizations; some in combination with TWS (consolidated administration). Almost 30% of its budget is devoted to the program. In addition to overhead support, JCPs allow not-for-profit organizations to take on projects they might not otherwise have been able to do.

Partners may provide training (academic or on-the-job) and work experience-in new/emerging/expanding skill areas. This is expected to lead to long-term employment for clients (which is not necessarily provided by the contractor).

Based on information from the participants' survey, 69% of those whose JCP intervention had ended have worked in the post-EBSM period. Fifty-eight per cent were working at the time of the survey.

B. Impacts on partners

We interviewed very few partners other than third party service providers. When we tried to identify candidates, we were advised that there were few, if any, other partners. Thus, we cannot address impacts on partners, other than to suggest that more work is required to broaden the partner base if EBSMs are to have a more major impact on communities.

Targets have been established for the LMDA in terms of the number of participants, returns to work (RTW) by participants, and unpaid EI as a result of participants RTW before they have used up their EI Part I entitlement. HRIB has the mandate to assess achievement against these targets.

Participants are identified in a number of ways:

• Through the National Employment Services System (NESS) Action Plan:

• Entered directly into NESS.

• Entered into Contact IV by third party service providers and uploaded to NESS.

• When an application for EI benefits is processed via the Online Information System (OLIS).

• Through participation in a Group Session if an active claimant. The Group Session Schedule Entry Screen (GSSE) must be updated to identify participation.

• Through intervention data entered into OLIS or CJS II.

Return to work is assessed in two ways. For those who are active claimants in receipt of EI Part I, a systems-approach is used. A RTW is recorded if the claimant has 12 consecutive weeks (or the remaining weeks on claim if less than 12) where EI benefits paid are 25% or less of entitlement. For those who are reach-back clients or who are active claimants and do not RTW before the end of the EI entitlement, a survey is conducted 12 weeks after the end of the EBSM. If the individual is working at the time of the survey, a RTW is recorded. Unpaid EI is estimated for those active claimants who RTW as total EI Part I entitlement minus EI Part I entitlement paid.

The evaluation has identified a number of problems with the way the LMDA's success is being measured and reported. These topics are discussed in this chapter.

A. Achievement of Short-Term Targets

Exhibit 9 outlines the short-term targets established for the LMDA for fiscal year 1997-98.

HRIB identifies 52,857 participants, 23,656 RTW and $93.8 million in unpaid EI. These represent short-term successes which occurred in 1997-98 but not necessarily for the LMDA. Our study evaluates the LMDA. As a result, we focus on participants under the LMDA and the successes represented by those participants.

Exhibit 10 identifies the short-term success that has been achieved by the LMDA between April 25, 1997 and March 31, 1998 according to HRIB data. These data are derived by restricting 1997-98 HRIB participant and results data to the participants and EBSMs after the April 25 start date of the LMDA in B.C. Information on mutual clients was determined by comparing HRIB participant data with MHR IA data.

The difference between HRIB's reporting of RTW and unpaid EI success for 1997-98 and success for the LMDA is accounted for by clients who pre-date the LMDA. In future years, annual assessments of success will deal exclusively with LMDA participants. However, there will remain a lag between participants in a given year and the results for those participants. In any one year, meaningful comparisons between reported participants, RTW and unpaid EI will be problematic if there are:

• Changes in the number or distribution of participants across EBSMs. (For example, the elimination of TP in June 1999 will create a significant aberration.)

• Changes in actual or perceived economic or social conditions.

It is unlikely that there will be sufficient stability in these variables across years to make meaningful comparisons within a single year.

The following sections assess the results identified by the HRIB data against the findings from the evaluation.

1. Participants Served

HRIB has a mandate to track the number of participants.

We observed anomalies between various HRIB data files-each intended to represent participants in 1997-98. The differences between the high and low numbers of participants identified was 8,000. Twelve per cent of those surveyed by the evaluation could not be found on HRIB's "official" participant file for 1997-98 despite being on an earlier HRIB participant file used for our sample and despite confirming their participation in our survey. Data from other HRDC data systems (Financial Administration System and Online Information System) identified 35% more participants taking comparable interventions and 130% more apprentices. Thus, we expect HRIB participant numbers to under-estimate actual participants.

Another anomaly, we identified 33% of non-apprentice participants as reach-back clients based on their eligibility at the time they began their last intervention. HRIB identifies the percentage of reach-back clients at 19% for 1997-98. Our review of HRIB data for participants after April 25, 1997 identifies 4,809 reach-back clients or 10% for all interventions (15% for long-term interventions). The difference is not attributable to apprentices who are few in number and, depending on the definition used to identify them, average about 45% reach-back. (Note that all apprentices should be active claimants to count toward the LMDA targets.)

The difference is attributable to the definition of "active". Once an active claimant the participant remains active so long as they participate in an EBSM. If they exhaust their EI eligibility, the active claim is extended to cover the period of the EBSM. So long as the individual has an active claim, he or she is eligible for further EBSMs and the active claim is extended by taking further EBSMs, to a maximum of 2 years. We found 23% of claims were still active after EI Part I entitlement had been exhausted. The average length of extension was 14 weeks and maximum was 52 weeks.

To identify the number of mutual clients, we used MHR data to identify whether participants were in receipt of IA at the start of their intervention. Total mutual clients were 6,927-fourteen per cent of all participants and less than one-third of the target set for mutual clients. The number of mutual clients may be under-estimated given our observations about under-reporting of participants.

2. Returns to Work

The HRIB results file identified 9,745 RTW for the 48,427 participants who began their last intervention between April 25, 1997 and March 31, 1998. This is less than the 23,656 RTW reported by HRIB for 1997-98. Almost 60% of the reported RTW were for individuals who participated prior to the signing of the LMDA.

The evaluation adopted two measures of RTW involving total and consecutive weeks of work. Within each definition it adopted least and most stringent definitions of these measures.

We assessed returns to work using these most and least stringent decision rules for each definition for participants in our survey who had a minimum of three months in the post-EBSM period. Exhibit 11 outlines the percentage of participants returning to work following these definitions and decision rules by EBSM.

Exhibit 11 Percentage Of Participants Returning To Work In Sample
	12 Consecutive Weeks (Most Stringent)	12 Consecutive Weeks (Least Stringent)	12 Weeks (Most Stringent)	12 Weeks (Least Stringent)
	%	%	%	%
EBSM:
EAS	48	52	48	52
TP	40	56	40	56
SE	70	77	71	77
JCP	49	56	49	56
TWS	57	61	57	61

Methodology note-Although there are considerable differences across definitions and across most and least stringent tests, our RTW estimates are very similar. The simplest explanation is that post-EBSM jobs, when they occur, tend to be long. Both definitions using the least stringent test produced identical results. A manual check of approximately 80% of cases showed that results were accurate.

The LMDA achieved an average RTW across EBSMs which ranged from 46% to 58%, depending on the methods used to calculate RTW.

Our survey covered non-apprentices in long-term interventions (EBSMs) only. Based on our survey results, we extrapolate to the population of non-apprentice clients as of March 31, 1998 expected to complete their EBSM by June 1998. This would provide a minimum of three months post-EBSM experience before our survey date, in keeping with our survey participants. Based on this population and the percentage of survey participants with a RTW, we estimate the RTW as of September 1998 for all participants under the LMDA as of March 31, 1998. Exhibit 12 provides the estimated RTW for non-apprentice participants under EBSMs.

The HRIB methodology identified 9,745 RTW by March 31, 1998 for all participants between April 25, 1997 and March 31, 1998. Our estimates of 9,442 to 11,743 bracket this value. However, our measures:

• Provide for a longer period of analysis (approximately 5 months). This would tend to increase our estimate.

• Exclude apprentices. This would tend to decrease our estimate. Of the 3,197 individuals who might be apprentices (maximum inclusion from available data), a total of 1,554 RTW. This represents 7% of the participants and 16% of the RTW calculated by HRIB.

• Exclude short courses. This would tend to decrease our estimate. HRIB data identify 2,814 RTW attributable to Group Service Interventions or 29% of the total RTW estimated by HRIB. Using the proportion of Group Services clients who RTW to approximate (because of missing data) Counselling Sessions clients' RTW, there may be a further 567 RTW.

Using HRIB-derived estimates for RTW by apprentices, Group Session and Counselling Session clients with our own estimates for other participants, there may be between 14,377 and 16,678 RTW by all LMDA participants. We also note that participant counts are under-reported. RTW by those participants who have not be counted are not included in this potential total.

3. Unpaid EI Benefits

The HRIB data for participants under the LMDA suggest that $42.3 million in EI Part I benefits were not paid. This is less than the $93.8 million estimated by HRIB for 1997-98. This difference is attributable to carry-over clients who did not participate in the LMDA.

To assess the unpaid EI for participants, we identify the number of current claimants who have a RTW. We then estimate for each their total EI entitlement minus the actual EI paid under their original claim from administrative sources. The difference is the unpaid EI benefit associated with a RTW but not necessarily attributable to a RTW. (More will be said about attribution in a later section.) This unpaid EI definition conforms to the definition used in LMDA evaluations in all jurisdictions.

The total unpaid EI benefit for non-apprentices based on this methodology ranges from $21.1 million to $23.2 million. This is approximately one-half of the unpaid EI for LMDA participants estimated by HRIB. Our estimate will exclude the unpaid EI of apprentices who, after taking the in-school training, return to their employer. The HRIB estimate includes $12.5 million in unpaid EI for apprentices. The evaluation method also does not include the unpaid EI for the 2,814 Group Services participants who return to work according to HRIB or the estimated 567 Counselling Sessions clients who may RTW. Applying the average unpaid EI estimated by the evaluation to these latter groups may add between $6.7 and $7.6 million to the total estimated above. The total across non-apprentice participants in long-term interventions, apprentices, and participants in short-term interventions may be from $40.3 million to $43.3 million

B. Case by case analysis of the HRIB and Evaluation assessment of short-term measures

Our survey identified 902 participants who had enough post-intervention experience at the time of our survey to assess their RTW status and unpaid benefit values. Only 794 of these participants could be identified on the December 1998 HRIB participant file. The remaining 108 could not be found despite being on an earlier HRIB data file used for our profile and despite confirming their participation in our survey. (This is another indication that HRIB may not capture information on all participants under the LMDA.)

To confirm that the 794 matched participants had the same last intervention in both sets of data, we excluded cases with different EBSMs (none removed) or different start or end dates by more than seven days (225 removed) or no dates in HRIB results data (23 removed).

We compared cases where HRIB did and the evaluation did not identify a RTW and alternatively cases where the evaluation did and HRIB did not identify a RTW. The proportions are about the same for reach-back clients for both methods. Both methods use a survey and appear to have a similar "error" rate according to the other's result. For active claimants, the evaluation identifies RTW for between 33% to 41% of those HRIB did not. This may be because the evaluation tracks clients over a longer period. Assessments of RTW by HRIB for active claimants are rejected by the evaluation in about 8% to 9% of cases. Note that matched individuals are those identified by HRIB to have participated under the EBSMs. We cannot speak to the RTW experience of those individuals who appear to participate under EBSMs, but who do not appear to be counted by HRIB.

Our evaluation methodology identified more RTW for active claimants but a slightly lower unpaid EI ($3,129 versus $3,470 by HRIB) per RTW. Some of the difference may be HRIB's 12-week 25% rule. The unpaid EI formula used by HRIB will ignore up to 25% of EI entitlement paid to the participant during the 12-week period used to calculate RTW. Our evaluation would count this amount producing a lower unpaid EI. More will be said about the 12-week 25% rule in the following section.

C. Appropriateness of Short-term Success Indicators and Methods to Calculate Them

Short-term success indicators (or targets) were established for the LMDA based on:

• Number of participants.

• Returns to work for participants.

• Unpaid EI for participants who return to work.

Earlier sections have identified the findings of the evaluation relative to these targets. This section discusses the appropriateness of the targets themselves and the methods used to calculate them.

1. Problems With the Measures of Success

Targets provide three key measures related to the success of the LMDA. These measures are intended to be indicators only. Their value as indicators should be based on their ability to provide unbiased insights into the likely success of the LMDA.

A significant portion of the targets represent the activities of apprentices who are provided income support through EI Part I to attend the in-school training component of their apprenticeship. These individuals may not be unemployed under a strict definition of the eligibility criteria for EBSMs and are not unemployed under a common sense definition of unemployment. When they continue the on-the-job component of their apprenticeship, their return to their employer is not equivalent to an unemployed worker's RTW. Similarly, there is no equivalent to the unpaid EI benefit for an unemployed worker since there is no intention that the remaining EI benefit will ever be paid.

As a result, to the extent that the targets include apprentices, they do not represent real impacts of the LMDA. They have a built-in bias to over-state the potential impacts.

It should be noted that unpaid EI benefits represent a reduction in the expenditure of EI Part I only. To achieve this reduction, an expenditure under EI Part II is being made. The expenditure under EI Part II may be less, equal to, or more than any entitlement under EI Part I. Viewed from the context of the broader EI Fund and not simply EI Part I, the unpaid EI benefit measure does not necessarily show a reduction in the amount paid to the client. Its name may be a cause of some confusion.

This is illustrated in the following example for TWS. Say an individual starts a TWS with 20 weeks remaining on their claim and a benefit rate of $300. The TWS is for 30 weeks at 40 hours per week and at a wage rate of $10 per hour. The program pays a 50% subsidy. (TWS can be for up to one and one-half years and up to a 60% benefit rate.) If the individual works at the end of the TWS or at the time of the 12-week follow-up a RTW is credited to the individual. The unpaid EI is $6,000. However, the expenditure under EI Part II has been $6,000. The benefit paid the individual is the same-the only difference is which Part of the EI fund is used to draw the payment.

The measure's stated purposes are to: indicate speed of RTW; to focus resources on those early in their claim period, and to compare current to past performance. (HRIB, Accountability Framework: Methods to Measure, September 23, 1998.) However, as the above example indicates, the method of calculation distorts use of the measure for these purposes. These purposes would be better served by a measure of the remaining weeks of claim when the individual returns to work. In the example noted above a "remaining weeks" indicator would be zero.

2. Problems With the Measurement of Success

The Human Resource Investment Branch (HRIB) is responsible for measurement of achievement against these short-term success indicators. HRIB has recently revised its methods of measurement and its algorithms to measure success. However, issues remain regarding the measurement of short term success. Not withstanding the inclusion of apprentices in these measurements we note the following issues related to the measurement against these targets.

Current Claimants

The method to calculate RTW for active claimants is referred to as the 12-week 25% rule. Under this rule administrative data on EI paid for the individual on claim are used to infer a RTW. A RTW is recorded for an active claimant (with an action plan or group session entered in NES, or if an apprentice entered in Benefits and Over-Payments (BNOP)) with 12 consecutive weeks (or the remaining weeks on claim if less than 12) where EI benefits paid are 25% or less of entitlement. The calculation assumes the "trigger" under BNOP has been pulled. This trigger deducts EI dollar for dollar if earnings exceed 25% of the benefit rate (or exceed $50 when the benefit rate is below $200). However, there is no recognition that a benefit rate of less than 25% does not imply earnings and does not mean that the individual has returned to work. (There are many reasons why a benefit will not be paid other than reported earnings from work. These are discussed in a later section.) The result is a potential over estimation of RTW for current claimants.

The calculation of unpaid EI savings is a point estimate only. The individual could go back on claim. The result is a potential over estimation of unpaid EI benefits. To overcome this potential HRIB adjusts its unpaid EI estimate at a later date based on the complete claim history. However, it continues to count unpaid EI before the RTW and exclude EI paid during the EI period.

Reach-Back Clients and Current Claimants After the Claim Period

Reach-back clients do not receive support under EI Part I. As a result their return to work cannot be assessed using the method outlined for current claimants. Instead a follow-up survey, 12 weeks after the intervention has ended, is used to capture data on whether the individual is working. A follow-up survey is also used in the case of current claimants who are not identified as having returned to work prior to the end of their claim.

To the extent that this follow-up is not made, the method will under-report RTW. However, if a follow-up is conducted, the answer to the survey question "Are you working?" may not be equivalent to "Have you returned to work?" The survey is valid only at a point-in-time. It has the potential to under or over report actual RTW.

Because reach-back clients do not receive EI Part I, there is no calculation of unpaid EI benefits.

Problems Related to Targeted Wage Subsidy

The method formerly used by HRIB identified an individual as having a RTW at a point 12 weeks after the start of their TWS, even though they continue on a subsidized work term. The approach has been changed to record a RTW if the case worker identifies the person being employed at the end of their TWS or if they are employed at the end of the 12-week follow-up. This approach suffers the same problems as the point-in-time estimate outlined above.

In the past, the formula HRIB used to calculate unpaid EI benefits to TWS clients could estimate a greater unpaid EI benefit than an individual's EI entitlement. The problem was a result of the individual's entitlement being extended under the TWS. The formula has been changed to eliminate this possibility.

For TWS clients identified as having returned to work, it now calculates unpaid EI after the point where the participant starts the TWS. This will avoid counting unpaid EI which is in the period prior to the TWS. However, the potential to capture unpaid EI before the intervention (non-attributable) exists with the formula used for all other EBSMs.

D. Alternate Measures of Short-Term Success

Earlier sections have identified problems related to the way HRIB success measures are calculated, especially for active claimants. The evaluation explored alternate methods to calculate RTW and unpaid EI for active claimants. (Focusing on active claimants, this reduced our effective sample size to 344 cases in the following analysis.)

We refine the calculation of HRIB return to work and unpaid EI measures by using more of the data available from the administrative systems.

For the RTW calculation, the alternate measure uses:

• Reported earnings to define work. This contrasts with HRIB's focus on benefits paid, which can be tricked by non-work-related reasons for reduced benefits, including severance and vacation pay, recovery of over-payment, failures to comply, and leaving the country. As a result, HRIB's measure can identify a RTW when it does not occur. (For comparison to HRIB's methodology, reported weekly earnings had to be sufficient to reduce EI benefits paid to 25% or less of entitlement.)

• Non-reporting to define work. We observed that participants who RTW are unlikely to return their weekly report cards if their earnings reduce their EI benefit to zero. Reporting results in a reduction of entitlement weeks where non-reporting does not. By not reporting, the participant "protects" these entitlement weeks and may draw on them later (but before they lapse) if they become unemployed. By not recognizing non-reporting as a normal behaviour by a participant who is working, the HRIB method can fail to identify a RTW when it occurs.

The alternate unpaid EI measure looks at EI remaining on the claim and compares it to EI actually paid after the start of the RTW. This is an improvement over HRIB's approach which:

• Looks at the entire claim period.

• Ignores EI paid during the period of the RTW calculation.

• Includes as unpaid any weeks of zero benefit paid before the RTW and, therefore, not attributable to the RTW.

• Can attribute EI benefits which lapse.

Examples will more clearly demonstrate the difference between the HRIB and alternate unpaid EI measures.

Exhibit 13 presents the hypothetical example of four individuals (A, B, C and D), all with 24 weeks of entitlement and a $400 weekly benefit, an intervention in week 1, but with different weekly benefits paid over their claim. Earnings (not shown) are the cause of the reduced weekly benefits resulting in a RTW calculated in week 13 for individual A, B and C and week 1 for individual D under the HRIB and alternate methods.

The unpaid EI calculated for each individual and method is shown in Exhibit 14. The alternate measure's formula subtracts EI benefits paid after the start of the RTW from the EI remaining at the start of the RTW. The alternate measure focuses only on that period over which an attributable impact can occur. This is not the case for the HRIB approach.

The alternate method gives the true unpaid EI in each case. The HRIB method calculates:

• Too much unpaid EI for individual B because it ignores benefits paid in the RTW period.

• Too much unpaid EI for individual C because it incorrectly attributes unpaid EI before the RTW period.

• Too little unpaid EI, as for individual D when, after the RTW period, the individual goes back on claim and receives a benefit less than their entitlement, but greater than zero.

Not illustrated in the example, the HRIB method can also calculate too much unpaid EI by including EI which has lapsed. Again, the alternate measure will correctly measure the unpaid component of this benefit.

3. Comparison of HRIB and Alternate Success for Participants

Exhibit 15 compares the HRIB and alternate approaches for the 344 active claimants in our sample. Both the HRIB and alternate approaches identify RTW for the same 51 cases. Unpaid EI is similar under both methods.

HRIB identifies 42 RTWs that are not identified by the alternate approach. Here the HRIB approach attributes a period where benefits are not paid as work time. Unpaid benefits are low for this group, likely as a result of benefits continuing to be paid after the RTW was identified. In one example reviewed, an individual had 84 weeks of severance pay. In week 60, the individual started a five-week EAS. After the EAS, the 12-week 25% rule counted benefits not paid (because of severance pay) as work and identified a RTW at the end of 12 weeks. The individual continued to use up severance pay, then went on to receive EI Part I until the claim terminated at week 104. HRIB calculated the lapsed EI as an $18,585 unpaid EI.

The alternate method identifies 98 cases as individuals having RTW where HRIB does not. On inspection, this difference is a result of two causes:

• The longer data history used by the alternate method. The HRIB measure is based on RTW occurring on or before March 31, 1998. The alternate measure is based on data extending to the end of 1998.

• The absence of reporting. As an example, the alternate measure identified 12 weeks of non-reporting during an individual's claim as work while HRIB did not. On inspection, the individual reported vacation pay in week 13, suggesting that the individual had in fact been working.

This chapter reviews the cost-effectiveness of the LMDA overall, the delivery of the EBSMs, as well as the cost-effectiveness of individual "successes."

If the LMDA is shifting costs from one part of the EI fund to another (as discussed previous), this has significant implications for the overall cost-effectiveness of the LMDA. In total, $206.2 million was spent on EBSMs (under Part II) in fiscal year 1997-98. In addition, various other costs were incurred to implement the LMDA, such as the costs of operating the Secretariat and other co-management structures, as well as in-kind costs (e.g., time) incurred by HRDC, MAETT and MHR. The extent of these costs is unknown. Restricting HRIB data to those who participated in the LMDA in 1997-98, approximately $42 million in unpaid EI ( to Part I) occurred in the short term. Long term impacts and the overall cost-effectiveness of the LMDA will be topics for the summative evaluation.

A. Program Cost-Effectiveness

Many interviewees felt that program delivery was already fairly cost-effective. However, most interviewees agreed it is always possible to become more cost-effective. For example, costs might be reduced if one government or agency delivered all EBSMs as this would eliminate the need for combined meetings and planning, etc. It would also address the problems identified regarding contracting.

It may well have been cheaper to have HRCC staff administer the programs, rather than have them administer contractors who administer the programs. Some contracts include funding for overheads, others do not. Thus, some third party service providers in the same area provide the same services without such subsidies.

Fewer contracts would reduce the contract administration effort. Longer contracts would enable contractors to negotiate better deals (e.g., for leases) and thus realize savings that could be passed on either to clients or to the EI Account.

Views regarding effectiveness vary from interviewee to interviewee. Provincial staff, for example, question the effectiveness of some EBSMs. One senior provincial representative commented that the province, based on its work with employers, does not consider TWS to be effective. HRDC representatives and employers interviewed in focus groups who had participated in TWS reported the exact opposite.

B. Costs Per Success

While individual third party service providers are able to identify their costs and identify outcomes on a client-by-client basis, and thus calculate a cost per success ratio, it appears that government cannot do the same in a reliable way, partly because of inadequate ongoing management and performance data. In addition, those interviewed generally could not identify the costs per success.

Principal conclusions of the formative evaluation of the LMDA are summarized in this chapter.

A. Relevance

The EBSMs are consistent with the intent of the EI legislation. The evaluation found no direct duplication between the EBSMs and provincial programs. The EBSMs generally promote reduced dependency on EI. As well, considerable efforts are being spent on co-management and local level decision making is occurring.

The evaluation concluded that EBSMs are highly relevant to the employment needs of clients. While many clients receive multiple interventions, these are sequential, not concurrent. All groups contacted feel that most eligible active EI clients are being reached or being made aware of the EBSM programs and services.

EBSMs also seem to be accessible to clients with special requirements or who are members of under-represented groups. While demand for services in French is low in BC, such services are available through a variety of arrangements in most areas of the province (though not always in the client's area of residence). Some contracts also provide English as a Second Language assistance. However, illiteracy is estimated to be as high as 20% among some client groups, which may reduce access to EBSMs in either official language.

B. Delivery

The degree of partnerships vary by area and level. Within government, there are problems with relationships at the senior level, both federally/provincially and intra-provincially. While the two secretariats work well, the effectiveness of the joint partnering structures varies, in part because of inadequate resources and frequently changing memberships. Partnerships with community groups are happening to a limited extent, but partnerships with employers are not common. Staff report a lack of time to develop these partnerships.

An enormous amount of effort and time has gone into joint planning, especially at the local level. To date, a province-wide annual plan for 1998-99 has not been signed because of disagreement over IA savings. Despite this, fourteen sub-provincial plans were developed locally and reviewed centrally for the 1998-99 fiscal year.

Almost all interviewees report that there is sufficient flexibility at the local level and that this flexibility has been used effectively to respond to local needs. However, a high level of local autonomy may result in some inconsistencies and inefficiencies.

Case management is widely used, though definitions of what constitutes case management vary. In particular, it is being used by third party service providers and for longer, more complex or multiple interventions that involve an action plan.

Negotiated financial assistance (NFA) appears to result in a sharing of training costs for some, but the savings realized are not recorded. The level of knowledge and the application of NFA also varies by area.

Information systems and the contracting process are two of the most frequently cited areas for potential improvement. While information gathering has improved over time, there are still significant concerns about the completeness, reliability and accuracy of some data. There are opportunities for enhancing the connectivity between systems, the use of consistent definitions, and the timeliness and accuracy of reporting.

The federal and provincial governments' approach and philosophy to contracting vary, and attempts to merge them have proved problematic. There have been no joint contracts. In some situations, the two governments have contracted with the same organization, but they use two separate or "parallel" contracts. Such contracts typically share common start and end dates but vary in terms of value and possibly services contracted for. These parallel contracts appear to be working.

C. Client Outcomes

Findings from the survey suggest, but not conclusively, that clients take on more responsibility through participation. Key informants note that this varies by factors which include: the extent to which a client is job-ready, the prevailing economic conditions and the availability of jobs, especially entry-level or lower-paying jobs.

Almost one-half of the participants surveyed who left their EBSM before completion, did so because they found employment. Almost one-quarter discontinued for negative reasons (e.g., the withdrawal of funds or dissatisfaction).

A majority of the participants whose EBSM involvement was finished at the time of the survey gave their EBSMs a positive satisfaction rating in the survey. Participants rated TWS as the most useful EBSM in obtaining full-time employment, followed by SE and TP.

Individual clients had the following outcomes:

• 23% took further training-almost one-half as a result of their EBSM.

• 75% have been employed since their EBSM ended (as compared to 69% of comparison group members). While participants are more likely to work than comparison group members, they tend to have lower average weekly earnings.

• 40% have received EI since their EBSM ended (versus 37% of comparison group members), while 9% have received IA (versus 5% of comparison group members). Paying at least part of the costs of training does not appear to reduce dependence on EI and IA in the short-term-even though those who paid at least part of the costs of their training placed more importance on being independent from EI and IA than other clients.

D. Outcomes on Communities

EBSMs appear to have had some employment impacts on communities-in particular, as a result of SE participants who have started their own business and have hired others to work with them. Participation in EBSMs also seems to reduce the willingness of residents to move and may result in greater volunteerism among participants. To date, there is little evidence to show that the social infrastructure in communities is increased through EBSMs.

E. Short-Term Success

Short-term success targets-number of participants, returns to work (RTW) and savings to the EI account (unpaid EI)-were established for 1997-98. According to Human Resources Investment Branch (HRIB), these targets were not achieved. Most RTW and unpaid EI identified for 1997-98 represent success for clients who pre-date the LMDA.

The evaluation focused on non-apprentice participants of the LMDA in long-term EBSMs in 1997-98. As of September 1998, approximately 10,000 of these 25,000 participants had RTW with unpaid EI of about $22 million.

There are problems related to the measurement of success by HRIB. Participant numbers may be under-reported by 35%. The actual number of apprentices may be 130% higher. The "12-week 25% rule" used to calculate returns to work (RTW) for active clients identifies participants as having RTW when they have not and as having not RTW when they have. Once a RTW has been calculated (rightly or wrongly) the unpaid EI measure focuses on the entire claim instead of that portion after the RTW begins. As a result it identifies unpaid EI which should not be attributed to the EBSM. There are other problems including the lack of compatibility of the point-in-time estimate for RTW used with TWS clients and with clients assessed using the 12 week follow-up survey. Estimates of success include apprentices. But, apprentices are not strictly unemployed when they attend the in-school component of their training. When they go back to the employers, who provided their "lay-off of convenience", their return is not equivalent to RTW by the unemployed. Also there is no expectation of receiving the unused portion of their EI. Including apprentices produces biased estimates of the success of the LMDA.

F. Cost-Effectiveness

In fiscal year 1997-98, $206 million was incurred in EBSM Part II costs to implement the LMDA. Other costs were incurred but have not been quantified. Restricting HRIB data to those who participated under the LMDA, an estimated $42 million in EI Part I benefits was not paid to approximately 10,000 participants who had returned to work by April 1, 1998. In the longer term participants may increase their earnings from employment. Fewer transfers to participants in the form of EI or IA benefits and more transfers to government in the from or increased taxes may result. Determining the cost-effectiveness of the LMDA requires a longer term analysis of impacts.

Many interviewees felt that program delivery was already fairly cost-effective. However, most agreed it is possible to become more cost-effective. Costs per success could not be identified. Individual third party service providers are able to identify their costs and identify outcomes on a client-by-client basis. However, government cannot do the same in a reliable way, partly because of inadequate ongoing management and performance data.

Footnotes

1	(Ref. Number 13295, April 1999.)
2	The level of significance identifies the risk or probability that an estimated result, which is judged to be significant, is in fact not different from the actual result. For a result to be considered significant, a maximum risk of 5% is generally accepted. Smaller levels of significance represent less risk and, therefore, greater certainty that the stated result is significant. We report the level of significance for all results identified as significant.
3	On further inspection, we found that 42% of the reach-back comparison group had worked throughout the EBSM period. This compares to 23% of reach-back clients who claimed their post-EBSM job had started prior to their intervention start date. Forty-four per cent of the latter, who worked, worked for themselves. These participants may not have technically been unemployed. However, because our comparison group also includes those who were employed during the EBSM period, we are able to determine the incremental impact of participation through our regression analysis.
4	A loss in income is sometimes observed for participants in the short term. Participation results in their absence from the labour market. During this absence, comparison group members may find a job and begin to increase their earnings. When participants return to the labour market, their earnings may be lower than comparison group members. Participant earnings must "catch up" to earnings of those who have remained in the labour market. However, over the longer term, the rate of increase in wages for participants may be greater than for comparison group members. Earnings in the medium and long term may be greater for participants. This issue will be explored by the summative evaluation.
5	The level of significance identifies the risk or probability that an estimated result, which is judged to be significant, is in fact not different from the actual result. For a result to be considered significant, a maximum risk of 5% is generally accepted. Smaller levels of significance represent less risk and therefore greater certainty that the stated result is significant. We report the level of significance for all results identified as significant.