 |
|
Executive Summary
The governments of British Columbia and Canada entered into a joint Labour Market Development Agreement (LMDA) on April 25, 1997 to facilitate the co-management of labour market development programs in B.C. Each has agreed to work together in the design and management of Employment Benefits and Support Measures (EBSMs) and in the operation of the National Employment Services (NES). Delivery, in British Columbia, remains the responsibility of Human Resources Development Canada (HRDC). Most delivery is achieved through contractual arrangements between HRDC and third party service providers. Eligibility under the LMDA is restricted to unemployed individuals pursuant to the Employment Insurance Act. Clients must be either a) active employment insurance (EI) claimants; b) former EI claimants whose benefit period has been established or ended within the past three years; or, c) those who have established a claim for maternity or paternity benefits within the past five years and who are returning to the labour force for the first time since leaving work to care for a newborn or newly adopted child. The latter two groups are eligible through reach-back. A subset of clients are also current or eligible clients of provincial income assistance (IA). These clients are classified as mutual clients. The LMDA incorporates EBSMs as the mechanisms to achieve the LMDA objectives. As of 1997-98 there were four employment benefits-Job Creation Partnerships (JCP), Targeted Wage Subsidies (TWS), Self-Employment (SE) and Training Purchases (TP)-and two support measures-Employment Assistance Services (EAS) and Local Labour Market Partnerships (LLMP). The objective of these EBSMs is to assist individuals to prepare for, obtain and maintain employment resulting in savings to the EI Account. In so doing, a return on investment will be achieved through a reduction in dependency on EI and IA and additional tax revenues through increased employment. Targets established for fiscal year 1997-98 were 68,885 EI clients served, of whom 26,000 are mutual clients; with employment returns of 25,008 and savings to the EI Account (unpaid EI) of $117.4 million. (It should be noted that a considerable but unspecified portion of the employment returns and unpaid EI targets reflect the activities of clients who pre-date the LMDA.) Under the Accountability Framework included in the Agreement, a two-phase evaluation (formative and summative) is to be conducted. This report presents findings from the formative evaluation. Other reports relating to this evaluation include Methodology Report, Technical Report, Evaluation Brief, Recommendations, Lessons Learned and What Worked Well. The formative evaluation uses multiple lines of evidence (both qualitative and quantitative) to explore seven types of evaluation issues-relevance, design/delivery/ implementation, impact on participants/employers, impact on community, assessment of short-term success indicators, overall impact/cost-effectiveness and EBSM-specific issues. Qualitative data were collected in the summer of 1998 (approximately 15 months after the LMDA was signed through focus groups, key informant interviews and literature and document reviews. They focus on how co-management of the LMDA is proceeding throughout B.C. as well as more specific information on the operation of the Agreement in three local areas. Quantitative data were collected in the early fall of 1998 (approximately 17 months after the LMDA was signed through a telephone survey of 1,200 EBSM participants across the province and a separate, comparative survey of 400 individuals who were matched to participants on key variables, but who did not participate in EBSMs. Survey responses were analyzed using univariate, bi-variate and regression techniques. Key findings for the period of the evaluation follow. A. Relevance The relevance of EBSMs were assessed against five key priority areas: Consistency With Intent Of EI Legislation—All sources contacted generally considered the EBSMs to be consistent with the intent of the EI legislation. The six EBSMs are harmonized to ensure no unnecessary overlap/ duplication. No direct duplication with provincial programs was identified. The EBSMs generally promote reduced dependency on EI. However, in the short term, EAS may increase dependence by increasing awareness of other EBSMs. Also, because TWS provides supported employment and insurable hours of work, it may act as a mechanism to get reach-back clients back on EI, thus increasing or sustaining dependency in the short term. Considerable effort has been expended on joint activities related to the co-management of the LMDA by HRDC, the Ministry of Advanced Education, Training and Technology (MAETT) and the Ministry of Human Resources (MHR). The EBSMs are broad and sufficiently flexible to allow significant decisions at the local level to reflect local circumstances and needs. In many respects, the LMDA has been implemented within a framework for success. An Accountability Framework exists for the measurement of success within B.C.; however, there have been serious issues with the systems used to report on success. Ongoing management/performance information is inadequate, unreliable and not timely. Consistency With Federal And Provincial Priorities-EBSMs appear consistent with Federal and Provincial priorities. However, the traditional client base of the federal and provincial governments are markedly different. Federal clients tend to have more work experience and a stronger attachment to the labour market. They tend to have recently lost their job and be eligible for EI income benefits compared to IA clients who are either not or are no longer eligible for EI. Provincial clients tend to have more barriers to employment. These tend to be different barriers than the job skill barriers of the typical federal client. Friction exists related to who should be the clients of the LMDA. This friction may be aided by the low number of mutual client participants relative to targets. Bilingualism/Language Of Delivery—The demand for services in French is very low, estimated as less than 2% of clients, even in some designated bilingual communities. Service in French is available through a variety of arrangements in most areas of the province. However, some clients may not be served in their area of residence. There are some contracts with third party service providers who deal with clients whose mother tongue is neither English nor French and who require English as a Second Language (ESL) assistance. Illiteracy, which is estimated by some third party service providers to be as high as 20% among their clients, has implications for access to EBSMs not only in an official language, but for potential access in alternate languages. Clients—All groups contacted feel that most eligible active EI clients are being reached or are being made aware of the EBSM programs and services. Only 9% of all participants surveyed report any difficulty accessing EBSMs. (The methodology may have excluded others who for reasons such as poor literacy and poor skills in an official language were unable to access EBSMs.) However, evidence suggests that the LMDA may have under-achieved its participant targets. Reported participants appear to be less than 80% of the LMDA's share of the 1997-98 the targets. The number of mutual clients served is less than one-third of the anticipated target. However, other evidence suggest that the official count of participants is under-reported. Reasons for the under-achievement of targets are complex. Some of them (e.g., client flow) are being addressed. Others, such as poor public transportation which limits participation in EBSMs and access to jobs and literacy, are less readily resolved. Many interviewees are concerned that changes, which pre-date the LMDA, in EI legislation and eligibility criteria (1996), as well as in provincial programming and eligibility criteria (implementation of BC Benefits welfare reform initiative (1996)), result in a substantial group of unemployed persons who are not eligible for existing services. Under-Represented Groups—The survey and administrative document review found that approximately 50% of participants are female, 4% to 6% are of Aboriginal descent, about 4% have a disability and somewhere between 2% and 9% report themselves to be members of a visible minority. Because of the need to self-identify, these figures may understate the situation (other than for gender). B. Delivery HRDC, MAETT and MHR have established a variety of mechanisms to assist with joint planning, decision-making and program delivery. Findings related to LMDA delivery are: Joint Planning—An enormous amount of effort and time has gone into joint planning, especially at the local level. A good deal of co-operation also exists at upper levels of the organization, but below the executive level. The Agreement requires the Management Committee to prepare an annual plan. No separate annual plan was prepared for 1997-98 as the Agreement itself was felt to be adequate. To date, a province-wide annual plan for 1998-99 has not been signed because of disagreement over IA savings. While this has not prevented those in the field from moving ahead on their own plans, it did result in their planning being conducted in a vacuum. Further, it provided an unfortunate signal to many trying to work co-operatively on their own planning exercise. Fourteen sub-provincial plans were developed locally and reviewed centrally by Secretariat and field staff. There is a significant level of variation among the sub-provincial plans we reviewed. Deficiencies observed include lack of results information, lack of detail on how non-mutual clients are to be served, use of mixed units for reporting purposes (making comparisons difficult), lack of estimates by specific EBSM and a general lack of detail. Community Input—The level of community input into LMDA business planning varies enormously from area to area. However, in all cases it seems that only a limited group in the community, primarily third party service providers, was involved. The community consultation framework prepared in April 1998 by the Communications Working Group had not been fully implemented at the time of the evaluation. Local Decision-Making—Almost all interviewees report that there is sufficient flexibility at the local level and that the flexibility has been used effectively to respond to local needs. However, a high level of local autonomy does have some drawbacks and may result in inefficiencies. Target Setting—Targets for the number of participants, proportion who are active versus reach-back, number of all participants returning to work and unpaid benefits by current claimants who return to work are set by National Headquarters (NHQ) of HRDC. Targets for mutual clients are negotiated within British Columbia. Mutual client targets exist at both the Human Resource Centres of Canada (HRCC) and the sub-HRCC level (MHR regions). While no targets are set for each EBSM, HRCCs base current year planning on previous year's numbers. In general, targets are not internally consistent. There is no link between the types of clients targeted and the demand for and supply of EBSMs. Client Flow—Client flow for mutual clients is another area where local autonomy has been used successfully. However, even in the best situations there have been problems. Each area, and in some situations sub-area, has developed its own process. Interviewees noted a need to refine the process but most also felt that client flow has improved recently. System problems inhibit the distribution of data on mutual clients and some tracking of mutual clients is still done manually. Contracting Process—Contracting is one of the most frequently cited areas of discontent. There have been no joint contracts due to financial and legal difficulties. In some situations, the two governments have contracted with the same organization, but they use two separate or "parallel" contracts. Such contracts typically share common start and end dates but vary in terms of value and possibly services contracted for. Data Collection, Monitoring, Tracking—There is an Information Sharing Agreement in place between the two governments. It came somewhat late in the process of LMDA implementation and its absence in the early months caused problems. There appear to be some differences in the way this Information Sharing Agreement is interpreted. The lack of an interface between the computer systems of the two governments/three departments causes additional problems in sharing information. Information gathering has improved over time. Increased familiarity with Contact IV has resulted in improved information gathering. Nevertheless, there are still some manual systems in place to overcome some system connectivity problems. There is concern regarding the completeness, reliability and accuracy of some data. For example, there is no agreement or consistency in terms of the unit of measurement to be used, nor are there definitions for specific units. Reporting remains a problem. The software in use does not produce all the reports required for day-to-day management. Other reports are produced by NHQ. They are not timely and their accuracy is questioned by some. HRDC's ongoing management and performance data do not allow it to identify how many/which clients are receiving which EBSM. Thus, without a major effort such as a survey, it is not possible to evaluate the effectiveness of these measures as determined by outcomes. Partnering—Processes to support partnerships vary from area to area, as does their effectiveness. Co-operation with community groups is happening to some extent. However, it varies from area to area. Most of the community groups involved in consultation and partnerships to date appear to be either the same as, or closely allied with, third party service providers. Co-operation and partnerships with employers at the local level are not common. Employers are not automatically brought into the consultation and decision-making loop. As a consequence, the impact of most EBSMs is limited to the supply side of the labour market. Their effectiveness may be further constrained if the supply side impact does not match a demand side requirement. Third Party Service Providers—In the three field sites, virtually all programs are delivered through third party service providers. Duplication/Overlap—The potential for duplication exists as federal and provincial governments continue to offer employment-related programs. However, our evaluation identified no duplication as the programs target different clients or the same clients at different stages in their progression toward a return to work. Efficiency And Effectiveness—Some interviewees noted efficiencies through the LMDA, primarily in the area of parallel contracting. However, some inefficiencies were identified. These include having too many third party service providers offering the same service in a geographic area, contractors having to compete with each other through advertising to maintain their share of a finite client base, and short contract lengths that have forced contractors into shorter and more expensive leases and equipment rental agreements. Contractors reported that uncertainty in the contracting process creates uncertainty for their staff, many of whom leave in search of more secure positions. Inefficiencies also result from the lack of congruent boundaries for local delivery areas of the three co-management partners. Excessive time is spent co-ordinating the co-management function. Clients—The EBSMs collectively offer a broad range of RTW activities. In addition, there is significant flexibility at the local level regarding how EBSMs are applied to meet local needs. The qualitative data suggest that the EBSMs are highly relevant to the employment needs of clients. While many clients receive multiple interventions, these are sequential, not concurrent. Survey results indicate participants are satisfied with their EBSMs and give them positive ratings in terms of their usefulness. Employers—Employers' main involvement with the LMDA is through TWS. They are not involved in community consultation and those in our focus group sessions knew little about the LMDA and co-management. It appears that lack of awareness, not the characteristics of TWS and LLMP, is the factor limiting greater participation by employers. Information And Communications—The unemployed have a variety of labour market and employment resources available to them. Eighty-two per cent of participants in our survey had made use of resources on a self-serve basis. There are significant differences between participants and comparison group members in terms of their use of labour market information and employment resources. Participants were more likely to use the Internet, printed career and work search information and technology provided for their use. Survey respondents rated the information they had received highly. There has been limited communication with community groups, including employers. Case Management—Case management is widely used, primarily by third party service providers. Not all clients are case-managed, particularly those going through short interventions. Usually, however, those going through longer, more complex or multiple interventions that involve an action plan are case-managed. Negotiated Financial Assistance—Clients are expected to contribute to their Return To Work Action Plan (RTWAP) if they are able. Negotiated Financial Assistance (NFA) is the process used to determine ability to contribute and level of contribution. The extent to which NFA is used varies by area, HRCC, third party service provider and even within each specific contract. The amount of knowledge of NFA also varies. Many, but not all, third party service providers have received training or assistance on NFA from their HRCC staff. There is some inconsistency in the application of NFA. NFA appears to result in a sharing of training costs for some—5% of surveyed participants said they paid all of the costs of their training, while 46% said they paid part of the costs. This compares to 42% and 8%, respectively, for comparison group members. C. Impacts/Outcomes on Clients EBSMs are expected to produce a number of impacts and outcomes for participants. Short-term impacts and outcomes identified by the formative evaluation are: Assuming More Responsibility-The EBSMs are intended to encourage clients to assume more responsibility for identifying employment needs, sharing in the cost of assistance, if appropriate, increasing employment and reducing their dependency on EI/IA. Views regarding achievement of this goal varied widely from area to area and from agency to agency within an area. Influencing factors include the extent to which a client is job-ready, the prevailing economic conditions and the availability of
jobs, especially entry-level or lower-paying jobs. Where jobs are available, EBSMs are seen as reducing dependency. Where there are few long-term jobs, EBSMs are seen as being less successful in securing employment. However, they are still seen as making clients more marketable and better able to market themselves. Action plans are seen as assisting or even forcing clients to take more responsibility for their employment. Discontinuation—Almost one-half of participants surveyed who left their EBSM before completion, discontinued because they had found employment. In 24% of cases, discontinuation was a negative result of the program-the withdrawal of funds or dissatisfaction. Satisfaction—A majority of the participants whose EBSM involvement was finished at the time of the survey gave their EBSMs a positive satisfaction rating. Participants rated TWS as the most useful EBSM in obtaining full-time employment, followed by SE and TP. Participants also gave their EBSMs positive ratings in terms of their usefulness in building their confidence, focusing their career goals and learning how to do a certain job. EBSMs were also considered to be useful, though to a lesser extent on average, at helping participants develop job skills such as managing time and organizational skills, obtain full-time employment and improve or reach their earning potential. Attitudes To Work—Most of the participants who had finished their EBSM report that their attitudes toward work and the future have become more positive or stayed the same since their EBSM ended. Further Training—Twenty-three per cent of participants have taken further training or courses since the EBSM ended. Almost one-half of these participants took this training as a result of their involvement in their EBSM. Employment—Seventy-five per cent of participants no longer involved with their EBSM have been employed subsequently. For members of the comparison group, 69% had worked after the date when the participant they were chosen to represent was scheduled to complete their intervention. Excluding SE participants, 64% of whom were in business for themselves, 92% of jobs in the post-program period for participants were with employers (91% for comparison group members). Although participants are more likely to work than comparison group members, they tend to have lower average weekly earnings. Dependence On EI And IA In The Short-Term—In the short post-EBSM period, 40% of participants (including those on EI at the end of their EBSM) and 37% of comparison group members had received EI, while 9% of participants and 5% of comparison group members had received IA. The percentage on EI at the time of the survey was lower for participants (10%) than comparison group members (12%), possibly indicating a reduction in dependence over time. (This will be a topic for the summative evaluation.) Months on EI and benefit rates for those on EI were comparable across the two groups. The only outlier is the current rate on EI for TWS participants which stands at 25% compared to 9% for all other EBSMs. A possible explanation is that time spent on a TWS increases EI eligibility. The percentage on IA at the time of the survey remained higher for participants. Impact Of Negotiated Financial Assistance—Those who paid at least part of the costs of their training placed more importance on being independent from EI and IA. However, in the short period reviewed under the formative evaluation, paying for training had not reduced the dependence on EI and IA. Those paying for training were also much more likely to turn down a good paying job if they had to move to get it. They were also slightly less likely to be employed in the period after training. D. Impacts/Outcomes on Communities Impacts On Communities—The following impacts were observed on communities: EBSMs appear to have had some employment impacts, in particular, those SE participants who have started their own business and have hired others to work with them. To date there is little evidence to show that social infrastructure in communities is increased through EBSMs. Nor are city councils and other community organizations involved much with EBSMs. Participation in EBSMs reduces the willingness of residents to move-potentially, participation is viewed as a substitute for moving from the community to obtain employment. Greater volunteerism by participants may result from participation in an EBSM. However, there could be other unobservable factors which influence both participation and volunteerism. Impacts On Partners—We were unable to identify partners other than third party service providers. Staff report a lack of time to develop partnerships within their communities. E. Assessment of Short-Term Success Indicators This section examines the achievement of short-term targets (the number of participants served, returns to work and unpaid EI benefits) and the appropriateness of success indicators. Participants Served—According to Human Resources Investment Branch (HRIB) data on EBSM interventions which started between April 25, 1997 and March 31, 1998, there were 48,427 participants. 97% of the active claimant, 20% of the reach-back and 27% of the mutual client target for fiscal year 1997-98. There were between 2,845 and 3,197 apprentices represented, depending on the method used to identify apprentices. There are discrepancies between various HRIB data on participants. Evidence from other HRDC systems suggest that actual numbers of participants may be 35% higher overall and 130% higher for apprentices. Returns To Work—According to the HRIB results file, 9,745 of participants under the LMDA had returned to work by March 31, 1998. This is less than the 23,656 returns to work reported by HRIB in 1997-98-the difference being carry-over clients who participated before the LMDA started. Our evaluation focused on non-apprentice participants under long-term interventions (EBSMs). The evaluation found 9,442 to 11,743 of this group had returned to work by September 1998. Accepting HRIB RTW numbers for apprentices and for short-term interventions there may be as many as 16,111 RTW for participants under the LMDA. Unpaid EI Benefits—HRIB data suggest unpaid benefits of $42.3 million for participants as of March 31, 1998. This is less than the $93.8 million reported by HRIB for 1997-98 again because of carry-overs. Our estimate, using a longer time frame and non-apprentices under EBSMs, identifies from $21.1 million to $23.2 million. Our estimate will exclude the unpaid EI of apprentices who, after taking the in-school training, return to their employer. The HRIB estimate includes $12.5 million unpaid EI attributable to individuals who may be apprentices. The evaluation method also does not include the unpaid EI for the 2,814 Group Services participants who return to work according to HRIB. The evaluation found additional problems with the measures and measurement of short-term success. The evaluation explored alternative short-term measures that may be used in the place of current HRIB measures. These proposed measures are system-derived. The alternative RTW measure uses earnings or the lack of reporting while on claim to identify work. The alternative unpaid EI measure focuses on the part of the claim period where an attributable impact may occur. These measures are presented and compared to the HRIB measures in the report. F. Cost-Efficiency/Cost-Effectiveness Program Cost-Effectiveness-Views regarding efficiency and effectiveness vary from organization to organization and interviewee to interviewee. Most interviewees agreed it is always possible to become more cost-effective. Management inefficiencies result from the lack of congruence in the geographic coverage of local delivery areas within the three organizations. Co-ordination is time consuming as a result. This time could be reduced through re-alignment of the boundaries. Many felt that delivery was already fairly cost-effective. Fewer contracts would reduce the contract administration costs. Longer contracts would enable contractors to negotiate better deals (e.g., for leases) and thus realize savings that could be passed on either to clients or to the EI Account. Costs Per Success-Interviewees generally could not identify the costs per success. G. Conclusions Principal conclusions are summarized below: The LMDA delivered EBSM that were highly relevant to the unemployment needs of clients. While multiple interventions can be received they are sequential not concurrent. EBSMs are accessible to clients. Those who ask for programs and services are being served, and in their official language of choice. While demand for services in French is low, such services are available through a variety of arrangements. Participants are satisfied with their EBSMs. TWS is rated as most useful in obtaining full-time employment, followed by SE and TP. Although more participants than individuals who compare to them obtain work, after the EBSM, wages tend to be lower for participants. An enormous amount of effort and time has gone into joint planning under the LMDA. Planning is highlighted as a success of the LMDA. However, there have been disagreements which have lead to no province-wide annual plan for 1998-99. The federal and provincial governments' approach and philosophy to contracting vary. Attempts to merge them have proved problematic. "Parallel" contracts offer some efficiencies and appears to work There is little evidence to date of the social infrastructure of communities increasing as a result of the EBSMs. There are significant concerns about the completeness, reliability and accuracy of some data from information systems. There are opportunities to enhance the connectivity between systems, the use of consistent definitions, and the timeliness and accuracy of reporting. Short-term success appears to be less than established targets. However, annual targets and official estimates of success do not reflect well the activities of participants under the LMDA. Participant counts are likely under-reported. The current RTW and unpaid EI measures may be significantly improved. Problems related to the measurement of the number of participants, RTW and unpaid EI limit accountability.
|
![[Previous Page]](/web/20060127005525im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127005525im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127005525im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}