|
|
     |
||
 |
     |
 |
|
Formative Evaluation of Employment Benefits and Support Measures Under the Canada/Nova Scotia Labour Market Development AgreementFinal Report
Evaluation and Data Development
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Table of Contents]](/web/20060127000311im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif) |
Under the direction of an Evaluation Committee composed of representatives of Human Resources Development Canada (HRDC), the Nova Scotia Department of Community Services and the Nova Scotia Department of Education and Culture, the evaluation was conducted by Power Analysis Inc. of London, Ontario. Subcontracting to Power Analysis, Dr. Craig Riddell of the University of British Columbia conducted and wrote up the econometric analysis (Chapter 9). Omnifacts Research Limited of Halifax, Nova Scotia conducted the participant and non-participant surveys. Martell Consulting Services of Tantallon, Nova Scotia conducted most of the key informant interviews, case studies and focus groups.
The evaluation team would like to thank all those who contributed to the study, especially officials in both the federal and provincial governments who gave of their time and experience to assist the evaluation team. We would also like to thank the many employment insurance clients who generously shared information about the impact of Employment Benefits and Support Measures (EBSM) on their lives.
|
On April 24, 1997, the Canada/Nova Scotia Agreement on a Framework for Strategic Partnerships was signed providing a framework for Canada and Nova Scotia to undertake joint activities in the area of labour market programs and services in Nova Scotia. Under this Labour Market Development Agreement (LMDA), Canada and Nova Scotia continue to maintain separate responsibility for the delivery of their own labour market programs and services within the existing authorities legislated by their governments while agreeing to work closely together in identifying areas of common interest.
The federal government has established several job creation, training and support options — incorporated under the Employment Benefits and Support Measures (EBSM) —to help clients find work and reduce their dependency on insurance benefits and other income transfer. Nova Scotia's EBSM consists of the following elements:
The primary purpose of this project was to carry out a formative evaluation to assess the management and operation of EBSM under the Labour Market Development Agreement. As specified by the Terms of Reference, the objectives of the formative evaluation were to:
EBSM is consistent with the EI legislation and with the priorities of HRDC. However, program guidelines such as targeting, selection, and case management are not always closely adhered to. At some offices targeting and selection is sometimes omitted: targeting and selection criteria exist but staff sometimes move away from them in their efforts to help all clients. In many cases an action plan is developed only for high-need clients — clients in need of upgrading and/or skills training — but none is developed for low and moderate need clients. In addition, there is no typical use of case management procedure — some offices are dedicated to it while others give it a lower priority.
Because of local level flexibility it is fair to conclude that EBSM has good potential to be relevant to the needs of the community. Every community agency representative interviewed in the three communities where case studies took place was pleased with the program and its new emphasis on community capacity building. All three communities had examples of successful partnerships with employers and the community.
The Canada/Nova Scotia Agreement has been credited for re-establishing some positive partnerships that had been forged between the province and the federal government. The evaluation, however, uncovered some challenges with the committee structure governing the partnership between HRDC and the province - although this is to be expected at the early stages of any partnership. Concerns raised by key informants include a cumbersome committee structure, committee members being on multiple committees, and communication problems. LMDA partners are addressing most of the concerns uncovered in the evaluation by streamlining the committee structure.
Most informants were of the opinion that joint cooperation had minimized unnecessary overlap or duplication between governments, especially for Targeted Wage Subsidies and Self-Employment.
Asked what the strengths of the organizational structure are in terms of the program's ability to meet its objectives, managers' consensus response was "flexibility." EBSM puts most of the discretion in the hands of local staff. All interviewees agreed that there is sufficient flexibility to allow decision making at a local level.
The flexibility of EBSM and the devolution of responsibility have been embraced by staff who are long-term employees of an HRCC; but they are quick to point out that the flexibility of EBSM and devolution of power to local staff could be a challenge to new workers. Interviewees indicated that even experienced workers need some parameters.
This level of flexibility also raises some concerns particularly with regard to the lack of consistency between and within zones. Different interpretations of how to apply the measures can result in differential treatment of clients. Lack of rules can also lead to a lack of consistency among HRCCs in terms of application of EBSM guidelines.
The interviewees were generally of the opinion that implementation went smoothly. In part, this was because of the high degree of flexibility of EBSM. Moreover, the component programs of EBSM were continuations of pre-existing HRDC labour market programs. There were some implementation issues, but for the most part they had to do with organizational changes taking place at the same time. The Region, for example, was reducing staff - some HRCCs lost many of their experienced staff. This required a change in the way HRCCs operated, forcing some HRCCs to use external agencies for the administrative components of EBSM. Another implementation challenge was the limited timeframe. Interviewees asserted that there was a great deal to do but very little time in which to do it. One key activity — staff training - fell short of what was required according to interviewees. Most concurred that the training was too general.
Several constraints may diminish the ability of the program to meet its objectives. The constraint cited most often, especially at the HRCC level, were those related to the limitations of the management information system. Current administrative systems were believed to be inadequate for providing timely information to help manage the program and demonstrate accountability.
EBSM clients who responded to the survey were asked to indicate their level of satisfaction with the program by assigning a letter grade, where A meant excellent, B good, C average, D below average, and F fail. Most participants (78 percent) thought the program was good or excellent. Few gave the program a failing grade (4 percent) or a D (5 percent). Clients in every EBSM component gave a mean overall grade of B, except for Employment Assistance Services clients who gave a B-. The reason most frequently offered for a low grade was lack of encouragement or help (15 percent) and not enough financial assistance (11 percent).
Discontinuation was not a major issue in Nova Scotia. About a quarter of EBSM clients had no Action Plan at any time, but most of those who did completed it. According to the administrative data 93 percent of EBSM clients successfully completed their intervention. No single reason predominated for not completing the intervention.
Survey findings support these results. The vast majority of participants on job placements stayed on the job for the entire period of the placement: 84 percent for Targeted Wage Subsidies and 86 percent for Job Creation Partnerships. Only 11 percent of the Purchase of training trainees failed to finish their course. The reason most often given for non-completion was that the participant was still in the course.
According to HRCC staff, clients who develop action plans — normally high need clients who require upgrading and/or training — generally take the onus for developing and implementing their action plan. Many come into counseling knowing what they want and, judging by the completion rate, follow through on their plans. Counselors ensure clients know about future employment prospects and help clients to identify their strengths and weaknesses, but otherwise participants have the primary role in deciding what type of intervention they need.
The survey of participants showed that about 43 percent of Purchase of training respondents were identified as being able to contribute to the cost of their training: 40 percent actually did so. In addition, all but 5 percent of Self-Employment clients made an initial capital investment in their new business. Nearly all informants held that the client is much more committed to the plan when it is not being entirely financed by HRDC. They have found clients to be more motivated to succeed if clients pay a share. Econometric results suggest that those who contributed to the cost of their intervention did better in terms of earnings than those who did not contribute.
Client outcomes refer to how participants and non-participants fared with respect to the intended results the program. Note that outcomes do not isolate the impact of the program; such impact analysis are shown in the following section.
A survey of program participants indicated that about 70 percent were employed for at least 12 consecutive weeks following participation in EBSM. Self-Employment and Targeted Wage Subsidies clients were the most successful in this regard.
EBSM participants out of the program in 1997 spent about 31 percent of 1998 unemployed, 64 percent working, 3 percent in school and 2 percent in school and working. Non-participants spent 32 percent of 1998 unemployed, 65 percent employed, 1 percent in school and 2 percent working and in school. The only statistically significant difference between the groups was time spent in school.
Prior to EBSM, about 45-55 percent of participants had been on Unemployment Insurance (UI) at some time during each year between 1992 and 1995. That proportion rose to 64 percent during 1996 and about 73 percent by 1997. Non-participants started off about the same as participants, but had a much greater tendency to be on UI by 1995. In 1996, the two groups were again about equal. In 1997, non-participants were much less likely to have been on EI than participants. The proportions of clients on EI fell by 1998, but this is not necessarily due to EBSM, since the fall was just as dramatic for non-participants.
In general, around 10 percent of EBSM clients had been on social assistance during each year from 1993 to 1997 compared to about 6 percent of non-participants over the same time frame. Employment Assistance Services clients were consistently the most likely (16 percent) to have relied on social assistance before participating. Limiting the analysis to those who completed EBSM in 1997, 11 percent of participants received social assistance in 1998, twice as high as the proportion for non-participants.
Program impacts were determined by comparing the outcomes of participants who completed the program in 1997 to those of a comparison group who did not participate in EBSM. It should be noted that the impact results presented here are preliminary only and that more definitive results will be presented in a summative evaluation where more time after program completion will have elapsed.
EBSM does not appear to have had any impact on the time spent in school. With respect to time spent working and unemployed the evidence is more mixed. The differences between participants and non-participants in the time devoted to these activities narrowed following the program, and in the desired directions: toward more time working and less time unemployed. However, these changes were not large enough for EBSM to have a statistically significant impact on time spent unemployed, at least in the short term.
Targeted Wage Subsidies appears to have had a positive impact of approximately 1.2 to 2 months on time spent working during 1998 but this conclusion is not supported by all the evidence as there was no significant impact on time spent unemployed. Self-Employment and Job Creation Partnerships may have had a modest positive impact on time spent working and a similarly desirable modest negative impact on time spent unemployed; sample sizes were too small to reach a more definitive conclusion. Employment Assistance Services and Purchase of training components did not significantly alter these outcomes.
There is some evidence that EBSM modestly reduced EI use, measured in terms of both weeks of benefits and total benefits received. To the extent that this reduction occurred, it came principally from the Self-Employment component that had a large impact on the use of EI (a reduction in the range of 7 to 14 weeks), and to a lesser extent from the Job Creation Partnerships component. The Purchase of training component may have reduced EI use to a modest degree, but the evidence supporting this finding is not conclusive. Employment Assistance Services and Targeted Wage Subsidies had no discernable effect.
EBSM did not have a significant impact on earnings in the year following program completion. Participation in EBSM did not reduce reliance on social assistance.
The formative evaluation of the Canada/Nova Scotia LMDA indicated that EBSM implementation went smoothly — a finding mostly attributed to the high degree of flexibility of EBSM. This flexibility provided for decision making at the local level and thus good potential for the EBSM to be relevant to the needs of the community. However, some concerns were raised in regard to consistency among HRCCs in terms of application of guidelines.
The LMDA has been credited for re-establishing some positive partnerships that had been forged between the province and the federal government. There were also many examples of successful partnerships with employers and the community undertaken through the EBSM interventions.
Most EBSM participants viewed their program favourably; 78 percent thought their program was good or excellent. About 70 percent were employed for 12 consecutive weeks following participation. Preliminary impact results also appear favourable suggesting increased employability for participants and reduced use of Employment Insurance.
Some concerns were identified that could have an impact on EBSM delivery. Timely client results data were generally not available to assist management and delivery staff in fine-tuning programs and services under EBSM. There was not a coordinated and consistent approach to delivery throughout the province, and there was uncertainty about rules for the delivery of some program components.
The primary reasons for conducting a formative evaluation is to determine what is working well and not so well and use those findings to provide advice to managers and policy makers on how to improve the program.
Program Monitoring
Program monitoring was identified as an area of concern. Given the importance of sound results measures to policy formulation, HRDC should provide the resources needed to collect relevant data and ensure managers and staff have accessibility to the information. HRDC should reiterate the importance of collecting and using results information to line managers.
HRCCs must also ensure that third parties take monitoring seriously and follow through to ensure the third parties are collecting required data. This could be written into the contract.
There is also a need to reexamine the computer systems related to EBSM and other HRDC programs. Ideally, there should be one system, simple for non-technical people to use and simple for management to draw out essential monitoring information. HRDC should come to an agreement as to what types of data are required for appropriate monitoring and design the computer system around these requirements.
|
The formative evaluation of the Employment Benefits and Support Measures under the Canada/Nova Scotia Labour Market Development Agreement (LMDA) identifies major areas of positive implementation as indicated by the high degree of satisfaction stated by evaluation participants. The evaluation also recognizes improvements required in order to modify and facilitate the delivery of Employment Benefits and Support Measures (EBSM). It is important to note that the evaluation was conducted during the early stages of the Labour Market Development Agreement and that important changes have been made since the evaluation was conducted in the summer of 1998. Human Resources Development Canada (HRDC) has been making improvements to the delivery of EBSM with a continual focus on partnerships and results.
The recommendations from the evaluation report and action planned or taken to date by HRDC are noted below.
Recommendation 1: Policy
Many of those interviewed for the evaluation indicated that the flexibility of EBSM and devolution of power to local staff present challenges to new and experienced workers. HRDC should examine the degree of flexibility it has granted the delivery network in the application of EBSM. A brief policy manual laying out how and when to use each EBSM component and specifying rules that cannot be broken should be developed.
Response
A guiding principle of EBSM is to grant local offices the flexibility needed to allow them to make significant decisions about implementation at a local level to better assist their communities. Management, however, has recognized that flexibility can lead to some inconsistencies in service delivery and, as a result, a number of steps have been taken to bring coherence to program delivery within Nova Scotia.
HRDC Nova Scotia Region has developed an Accountability Policy Framework and a process to establish priorities and procedures for the effective and accountable delivery of programs and services. The approach incorporates four strategic directions: the establishment of an accountability framework; the development of tools for data capture, evaluation, monitoring and quality assurance; the provision of training and support for the development and implementation of the accountability framework; and capitalization of effective risk management practices.
Policy papers have been developed for Nova Scotia priority areas such as youth, persons with disabilities, and access to information technology. The policy papers establish parameters and provide guidelines around how HRDC and the partners can assist specific client groups. The policies are first approved by the Strategic Planning and Policy Committee and then forwarded to the Service Delivery Committee for the development of an action plan. Once an action plan is developed and approved, it is shared with all relevant service providers.
A document detailing appropriate uses of each EBSM component has been developed and is currently being validated with Regional and National staff. This tool also contains a series of Questions and Answers that will assist delivery staff in moving towards a coherent approach in the delivery of EBSMs to clients.
Recommendation 2: Administration
Central administrators and local managers should ensure that delivery staff adhere to the guidelines established for the program . To the extent such guidelines as targeting and selection, case management, and action plans are deemed important for effective program delivery, HRDC should require their use at the local level (which may entail further resources and training).
Response
The annual business plans set goals and priorities for the local area including clients targeted for services based on a community needs analysis as well as local office capacity to respond.
To foster and support program effectiveness, and to ensure that proper procedures with respect to targeting and selection, action plans and case management are followed, specific activities are set out in the Accountability Framework Implementation Plan:
In addition, a study is currently underway in the Region to identify characteristics that put EI clients at greater risk of moving to social assistance. The results will be useful in the targeting and selection process to identify "at risk" clients.
Recommendation 3: training
Many local program staff are attempting to carry out their duties under considerable uncertainty regarding the rules around EBSM delivery. training for newly implemented programs must be more specific, especially when the program has a high degree of front-line discretion. Key policy rules that must be followed must be made clear. training on the appropriate application of Job Creation Partnerships and Labour Market Partnerships especially are needed.
Response
HRDC has been delivering training and developing tools required for an effective and accountable delivery of programs and services. The region has recently held training sessions for program officers on the delivery of employment interventions to provide them with the skills to negotiate, contract and monitor agreements within the Terms and Conditions of EBSMs.
In addition, a variety of tools and desk aids that provide staff with documentation requirements to ensure accurate and complete data capture of client case management and action plans have recently been developed to assist delivery staff. Specific tools and training include a National Employment Service System (NESS) user guide, Contact IV user guide, contracting with external service providers, and a monitoring guide for those managing external service providers. An evaluation tool has also been developed to measure employment service delivery in the local office and to identify the integration of training as well as training gaps in the workplace.
Recommendation 4: Program Monitoring
Program monitoring needs to be strengthened. Given the importance of sound results measures to policy formulation, HRDC should provide the resources needed to collect relevant data and ensure managers and staff have access to the information. HRDC needs to reiterate the importance of collecting and using results information to line managers.
Human Resources Centre of Canada (HRCCs) must also ensure that third parties take monitoring seriously and follow through to ensure the third parties are collecting required data.
In support of the monitoring function, HRDC should come to an agreement as to what types of data are required for appropriate monitoring, and design one data system which is simple to use for data input as well as for the extraction of results data.
Response
Effective monitoring and quality assurance is an area emphasized in the Accountability Framework Implementation Plan. A Regional Monitoring Policy has been developed and was recently issued to all relevant staff as part of the Accountability Implementation Plan.
As mentioned previously, training that was recently delivered to program officers had a large component devoted to project monitoring. A monitoring guide was also developed for program officers responsible for managing external service providers to ensure compliance with the terms of the agreement as well as proper client documentation.
An information package is also being developed to ensure external service providers are familiar with HRDC's accountability policies and are putting them into practice.
The HRDC management information systems are under constant review for relevancy. A recent NESS and Contact IV streamlining exercise has been completed.
|
No longer is passive income maintenance acceptable for many of Canada's million plus unemployed. They need help to meet the requirements of a labour market that has become more demanding as Canadian business strives to compete in an increasingly competitive world.
The federal government has established several job creation, training and support options — incorporated under the Employment Benefits and Support Measures (EBSM) —to help clients find work and reduce their dependency on insurance benefits and other income transfers. Under the Labour Market Development Agreement (LMDA), EBSM is undertaken in partnership with provincial governments with a major aim of fostering a more proactive approach to meeting the labour force challenges of the 1990s and beyond.
Given that Canada's future prosperity depends on a productive workforce, it is imperative that the government's active efforts to assist the unemployed are successful. A principal way of ensuring success is to carefully evaluate these programs to identify what aids or impedes their success, and to recommend improvements.
This report presents a formative evaluation of EBSM in Nova Scotia. The evaluation examined the benefits and measures themselves, as well as the working relationship between Human Resources Development Canada (HRDC) and provincial departments, and the impact of those relationships on the success of the program.
The primary purpose of this project is to carry out a formative evaluation that will assess the management and operation of EBSM. Formative (also called process) evaluations answer the question, "How is the program operating and how can it be made better?" The main objective is to provide feedback to managers on whether the program is being carried out as planned and in an efficient manner. It should provide a detailed account of the program as implemented and compare it to what was intended. To the extent that the implementation diverges from program design, the appropriate treatment may not be delivered, the right target group may not be helped, or both. Even very well planned programs will have unanticipated results and unwanted side effects that arise during implementation; through a process evaluation, problems can be identified and addressed in time to benefit those who depend on the program. Either the program can be modified so it is carried out as planned, or the plan itself can be modified if it is found wanting. As well, a good process evaluation lays the foundation for a good summative evaluation, in part by ensuring appropriate data are being collected.
As specified by the Terms of Reference, the objectives of the formative evaluation are to:
The evaluation also studies the partnership between Nova Scotia and Canada in delivering EBSM under LMDA.
Chapters 2 and 3 set the context for the evaluation by describing the methodologies and presenting a summary of program design. Chapter 4 reviews the implementation and operation of the program. Chapter 5 summarizes the Canada-Nova Scotia Labour Market Agreement. Chapter 6 describes how each EBSM component has been implemented in the case study sites. The EBSM client profile comprises Chapter 7.
Chapter 8 turns to issues of outcome, analyzing preliminary outcomes in terms of receipt of income assistance and occupational success. Chapter 9 builds on the descriptive analysis of Chapter 8, with an econometric analysis of impact. Using longitudinal and cross-sectional models, it presents a thorough analysis and interpretation of the results. The final chapter summarizes the major findings, and draws together the different lines of evidence to address the evaluation questions.
|
Six sources of information were used to evaluate Employment Benefits and Support Measures (EBSM): a review and analysis of administrative data; interviews with program officials; a survey of participants and a matched sample of non-participants; focus groups with Human Resources Centres of Canada (HRCC) staff; case studies at three HRCCs; and an econometric analysis to determine preliminary program impact.
The overall purpose of the document review was to enable the evaluators to learn about the program and its context. This is imperative for the conduct of a formative evaluation wherein a thorough understanding of the program as designed is the foundation for all subsequent work. The Committee provided various documents during the initial meeting, which were read before preparing the draft research instruments.
HRDC provided electronic files and file descriptions of administrative data concerning EBSM. The main purposes of the analysis of the administrative files were: to produce a profile of the program and its clients; to assess the monitoring and performance measurement system; and to validate the primary employment results indicator. In addition, the data were used to select samples for the participant and non-participant surveys, to address several evaluation issues, and to examine the data sets for completeness and accuracy.
The purposes of the interviews were: to assess program implementation, management and operation; to determine informants' understanding of the goals and objectives of the program; to identify any major obstacles to achieving program objectives; to examine the federal-provincial partnership; and to gather suggestions for making the program more successful.
The first step was to obtain a list of interview subjects from the EBSM evaluation committee. Concurrently, interview guides were designed to govern the interviews.
Somewhat different guides were needed to reflect the different perspectives of HRCC managers, zone managers, regional managers, and provincial officials. Interviews with 25 HRDC and provincial government managers were completed.
Two separate survey instruments were created, one for participants and one for non-participants. There was a lengthy core of questions focusing on post-program activities common to both questionnaires to set the stage for an econometric analysis of preliminary impact.
The questionnaires were reviewed by the evaluation committee, then pre-tested with about 40 respondents. Respondents had virtually no problems with the questions or response categories, with the length of the questionnaire, or with recalling details of interest.
While the questionnaires were being crafted, random samples of participants and non-participants were selected from administrative databases supplied by HRDC. Participants who started their intervention on or after January 1, 1997 and ended on or before June 30, 1998 were eligible for selection. Selection was stratified by EBSM component to end up with approximately a 6 percent margin of error for each component. The final number of cases by component is as follows:
| Component | N |
| Employment Assistance Services | 209 |
| Job Creation Partnerships | 196 |
| Purchase of training/Feepayer | 284 |
| Self-Employment | 116 |
| Targeted Wage Subsidies | 228 |
| TOTAL | 1,033 |
As for non-participants, a random sample was selected very similar to the sample of participants. The literature points to a few key traits for matching including age, sex, education, program eligibility, and employment/unemployment history. Program eligibility was a given since the non-participant file came from Employment Insurance (EI) files. Because administrative data on education are not available on all clients, this variable was not used for matching. Samples were matched on age, sex, and unemployment history.
A computer-assisted telephone interview (CATI) system was used to facilitate the phone surveys. Because of the problems with finding people at home and with invalid telephone numbers, up to 20 attempts were made to reach each person in the sample before replacement. Most telephone interviews took place in the evenings or on weekends. On average, they lasted about 18 minutes for participants and 13 minutes for non-participants. Response rates were respectable for this target group: 62 percent for participants and 61 percent for non-participants. An analysis of non-response concluded there should be no substantial biases.
CATI generated a ready-made computerized file. It was carefully edited and imported into SPSS for statistical analysis.
The standard error is the key measure of the accuracy of results. For the calculation, the gender variable is used. The standard errors and associated margins of error1 with a 95 percent confidence interval are:
| SE | Margin of error | |
| EBSM | 0.0145 | ± 2.9% |
| Employment Assistance Services | 0.0314 | ± 6.2% |
| Job Creation Projects | 0.0305 | ± 6.0% |
| Purchase of training | 0.0274 | ± 5.4% |
| Self-Employment | 0.0325 | ± 6.4% |
| Targeted Wage Subsidies | 0.0296 | ± 5.8% |
The purposes of the focus groups were: to assess program implementation; to gain a better appreciation of how EBSM activities are carried out; to examine how the program is monitored; and gather suggestions for improving the program.
Focus groups were held with all HRCC staff who could attend at the case study sites as well as in Halifax. A protocol to cover the issues was submitted to the committee for approval. Sessions lasted for two to three hours. The discussions were transcribed, then analyzed.
Case studies were conducted at HRCCs in Antigonish, Sydney and Yarmouth. The case studies were meant to explore how EBSM — a program characterized by a great degree of local flexibility — was implemented in different areas of the province. The case studies consisted of an administrative data review, interviews with the manager, employers, outreach groups and third party providers, a focus group with staff, a group meeting with a regional committee, and observations. Discussions were transcribed and analyzed.
Although it is too early at the formative evaluation stage to render a definitive verdict on program impact, it is important to look for early signs of program success. The operative question: Does the program seem to be accomplishing its objectives at this early stage?
This evaluation used a quasi-experimental design to estimate program effectiveness. This approach was required because participants and non-participants were not randomly assigned as would be the case in a true experiment. A variety of econometric and statistical techniques were used to assess whether the program had an incremental impact on work activity, earnings, receipt of employment insurance or social assistance, the allocation of time to work and/or school, and attitudes toward work and social assistance.
To measure the program's impact, pre-program data were obtained beginning two or more years prior to referral to the program. This information was important because it permitted a determination of the incremental impact of the program by controlling for biases caused by unobserved individual differences.
|
Employment Benefits and Support Measures are programs and services designed to assist current and former Employment Insurance (EI) clients prepare for, find, and keep jobs. EBSM began as a part of the EI restructuring in 1996. The chief objectives of the reform were "to ensure that the system was responsive to the realities of today's labour market and to remove disincentives and inequities in the system." Active re-employment assistance was emphasized to help the unemployed find work, to improve work incentives, and to simplify administrative processes for employers.
Part II of the EI legislation contained six important changes related to EBSM:
Reflecting this context, EBSM principles include:
EBSM in Nova Scotia currently encompasses the following elements:
Targeted Wage Subsidies Offers employers a wage subsidy to provide on-the-job experience for individuals who have been unemployed for a long time or face special barriers to employment. The subsidy is meant to encourage employers to hire eligible clients who would not normally be considered without the subsidy. The maximum subsidy is set at 60 percent of the wage paid by the employer. The maximum duration is 78 weeks. Contracts are negotiated by the Human Resources Centres of Canada (HRCC) or delegated partners.
Self-Employment Assistance to unemployed individuals with viable business ideas. Individuals may qualify for financial support, planning assistance and ongoing support while getting the business up and running. A special priority is to be placed on designated groups: women, visible minorities, Natives and the disabled. For delivery, HRCCs contract with local delivery agents with expertise in business development. These third party agencies select appropriate clients, provide training in key aspects of running a business, and negotiate client contribution and length of contract (not to exceed 52 weeks, or 78 weeks for the disabled). Income support — EI and some expenses such as child care — is not insurable.
Job Creation Partnerships To help develop the local economy in areas where jobs are scarce, these projects encourage community partnerships to help claimants get work experience. Partners may include provinces, the private sector, labour and community groups. Its aim is to create "sustainable employment" for clients; that is employment that lasts beyond the term of the job creation project. Job Creation Partnerships projects should not be "make-work," nor should they fund existing community activities. They are meant to fund new community economic development initiatives. Partners are expected to contribute capital and material costs and perhaps top up income support. Targeted clients are EBSM eligible individuals with multiple barriers to employment, or more job-ready clients without employment opportunities. Job Creation Partnerships is funded through contribution agreements up to 52 weeks in duration. Income support may be EI or non-insurable remuneration.
Training Purchases and Skills, Loans and Grants (Purchase of training) As mentioned above, training purchases must be phased out by the middle of 1999 to be replaced by Skills, Loans and Grants, an initiative providing loans or grants to help with course fees and living expenses for those who need new skills to return to work. Purchase of training normally consists of "buying" seats at a local community college or private training school, which are reserved for eligible clients. Purchase of training clients are excused from searching for work while in training; they receive EI benefits and have course costs covered. "Feepayers" are excused from job search requirements but have to pay course costs. Maximum duration is 52 weeks.
Employment Assistance Services Under Employment Assistance Services, HRCCs or community-based organizations deliver a range of services and Human Resources Development Canada (HRDC) programs to help clients prepare for, find and keep jobs. Services may range from self-service information to full case management. Employment Assistance Services gives HRCCs the flexibility to decide what services to contract out to whom, based on local needs and resources. Employment Assistance Services are normally used by job-ready clients who need labour market information, help with job search, access to tools such as resume writing software and the Internet. In some areas, higher need clients such as the disabled may be referred to Employment Assistance Services providers with appropriate expertise.
Local Labour Market Partnerships Local employers, labour groups, community organizations and government agencies forge partnerships to help create local employment opportunities for the unemployed. The objectives of Local Labour Market Partnerships are to support community efforts to identify labour market needs, identify the capacity of the community to meet those needs, work with partners in building "community capacity" to address local labour market needs, and minimize duplication and gaps in service. As such, Local Labour Market Partnerships fund research, community planning, and innumerable other activities in the name of community economic development. HRCCs are to identify needs (perhaps through research and developing local planning groups), to help mobilize community resources to meet the identified needs, to provide information and financial support as warranted, to negotiate partnerships, and to monitor the results.
No claimant is automatically eligible for employment benefits. Many offices target those who need extra help and are prepared to make a personal commitment to a return-to-work Action Plan. An HRCC staff member helps determine client needs and eligibility for EBSM measures. Clients develop a personal Action Plan that must be approved by an HRCC staff or external Case Manager. This plan sets out the steps the client agrees to take to get back to work, and the actions HRDC will take to support them. Case Managers support and monitor client progress, and provide additional assistance as needed.
|
This chapter examines the implementation and operation of EBSM in Nova Scotia. The first section briefly recounts the implementation of EBSM in Nova Scotia, using information from the key informant interviews. The second section summarizes how EBSM is operating in Nova Scotia, using information from the key informant interviews, focus groups and case studies.
Implementation of EBSM in Nova Scotia took place in July 1996. Interviewees were generally of the opinion that implementation went smoothly. In part, this was because of the flexibility of EBSM. Moreover, many staff interviewed felt the component programs of EBSM were continuations of pre-existing Human Resources Development Canada (HRDC) labour market programs. Purchase of training, Self-employment Assistance and various wage subsidies and job creation programs had been offered by HRDC for a number of years. Several interviewees referred to similarities with the Canadian Jobs Strategy (CJS).
One important change from the previous delivery mechanism is the new focus on outcomes: Does the client get and keep a job? Several informants noted a shift in philosophy and culture. They claimed the new philosophy emphasizes caring about what happens to a client, i.e., working with clients until they are self-sufficient.
The change from entitlement to negotiated financial assistance with the client contributing to the cost of the intervention has also required a change in mindset. Under negotiated financial assistance, counselors work with clients to calculate the costs of taking training, and an agreement is negotiated to apportion these costs between HRDC and client. One interviewee perceived this as the most fundamental change. Managers across the province were very much in favour of cost-sharing with clients. They have found clients to be more motivated to succeed if they pay a share, saying it makes them more accountable and successful in their training intervention. Also staff found that clients who put their own money into the plan do more preparatory work to make sure it is the right program for them.
A final shift identified by interviewees was the greater use of partnerships: using third parties to deliver the programs and working with other levels of government. Chapter 5 centres on the partnership with the province and Chapter 6 contains many examples of the beneficial use of partnerships at the field level.
In general, the implementation of EBSM was perceived to have gone smoothly because the components were essentially the same as before. Challenges encountered dealt primarily with organizational changes taking place at the same time.
The Region, for example, was reducing staff. Some HRCCs lost many of their experienced staff. This was especially critical to a program that is based on flexibility. Some HRCCs turned to external agencies for delivery of some components of EBSM.
Another implementation issue was the limited timeframe. Interviewees asserted that there was a great deal to do but very little time in which to do it.
During implementation, one key activity fell short of what was required according to interviewees: staff training. Although there were disagreements concerning whether or not there was enough or even too much training, most respondents concurred that the training was too general. There was a series of nationally designed training sessions focused on delivery mechanisms and accountability measures. But informants claimed they were poorly timed, since the sessions took place too far in advance of EBSM implementation. As well, the information relating to the measures kept changing. Most importantly, according to staff the sessions were very general or philosophical in nature. Lack of "how to" detail has meant that staff have had to improvise, which has led to inconsistencies in how the measures are delivered throughout the province.
As previously mentioned, implementing some individual components posed little challenge because of their similarities to pre-dated EBSM in one form or another. The only measure with which field staff were not familiar was Local Labour Market Partnerships. Many informants said they didn't know the specific rules for its use.
Lack of familiarity with guidelines gets back to the need for program-specific training. Interviewees identified a pressing need for training particularly relating to what constitutes a Job Creation Partnerships or Local Labour Market Partnerships and what constitutes community capacity building. Project officers indicated they were not comfortable in how they were doing things in the field and wanted to learn the correct way.
Asked if there were any particular options that have been more difficult to implement, interviewees made Skills Loans and Grants the virtually unanimous choice. Skills Loans and Grants, which was to be phased in by January, 1999 is temporarily on hold awaiting the outcome of the provincial meetings with the federal HRDC minister. The Provinces are concerned about the loss of capacity from the federal government getting out of direct training purchases in June 1999. Field staff were anxious to start the process on January 1st and felt it would be good for clients. Some had already started negotiating loans with their clients when the program was put on hold. Most informants were of the opinion that it is urgent to get Skill Loans and Grants under way before Purchase of training expires.
To ensure a thorough and systematic analysis of EBSM as it currently operates, this section looks at EBSM in terms of its objectives, structure, communications, monitoring, activities, and environment.
One good test to determine if a program has been implemented as planned is to ask those responsible for day-to-day program operation about the cardinal objectives they are trying to reach. If the field is striving for something different than program planners had in mind, chances are good that the program as operating is not faithful to its design.
There was virtually unanimous agreement among those interviewed that the most important objectives of EBSM are jobs and unpaid Employment Insurance (EI) benefits, which corresponds to the main goals of EBSM. Several managers also mentioned community capacity building as the new vision of HRDC. Moreover, there is strong evidence that managers at every level are taking the new emphasis on results seriously. Most talked about the importance of focusing on client outcomes.
But are these objectives likely to be achieved? Several constraints may diminish the ability of the program to meet its objectives.
Constraints to Meeting Objectives
The constraints cited most often by a wide margin, especially at the HRCC level, were those related to the limitations of the management information system. Current administrative systems were said to be inadequate for providing timely information to help manage the program and demonstrate accountability. Some interviewees indicated having no knowledge of how well they were doing, and thus no ability to adjust workload or make changes based on results. Certainly, the inability to measure results works at cross-purposes with HRDC's new orientation to results.
Besides the clear connotations for managing and accountability, the limited capacity to measure what the programs are doing has other unfavourable implications. Insufficient results monitoring may be affecting decisions on what interventions to use. Key informants stated they haven't been able to measure how well their programs work so they tend to rely on measures with which they are familiar and know have provided good results in the past such as Targeted Wage Subsidies.
The problem with monitoring also hinders the Region's ability to determine how well different zones or HRCCs are doing in a relative sense. One office may include all clients who go through a group information session, send them to the intervention before targeting and selection and count the results. Another may have different monitoring rules.
A few other constraints were mentioned as well. Some interviewees raised the issue of limited funding for resources. The chief result of this was said to be reductions in staff, which in turn has lead to:
A final constraint to achieving employment results identified by key informants was poor labour market conditions. The labour market varies dramatically across the province, with the Halifax region enjoying relatively low unemployment, but areas of high unemployment elsewhere. High unemployment was said to be a large problem for Job Creation Partnerships because the low demand for workers could seriously hinder the chances of employment after the subsidy ends. This would also apply to Targeted Wages Subsidies participants.
At the centre of EBSM's delivery structure is the Regional Office, which has two units directly involved with the delivery of EBSM: Delivery Support and Strategic Alliances.
Strategic Alliances deals with partnerships, sector councils, community capacity building, and designated groups2 as well as offshore activities. As such, Strategic Alliances doesn't deal extensively with the field offices, but deals mostly with external agencies.
Delivery Support deals with programs that are delivered in the field offices and with clients. It is responsible for the counseling and training areas as well as employment services. With the highly decentralized structure of EBSM, Delivery Support has limited roles with respect to program delivery. Their role is primarily to provide expertise and advice. Purchase of training is still administered by the Region, but this is being phased out by June 1999. Delivery Support also carries responsibility for the management information function and financial management for EBSM.
The Region's role was much more extensive during the implementation phase of EBSM. Regional office then stepped back and let ongoing program delivery fall to the zones and HRCCs.
The next level comprises four delivery network zones or geographic regions of the province. The map below shows the four zones: Halifax Metro, Industrial Cape Breton, South Shore-Valley, and Northern. The zone office is located in one of the HRCCs in each zone: Halifax, Sydney, Bridgewater, New Glasgow. In addition to employment-related functions, the zone offices house most administrative functions for the zone and process Employment Insurance claims.
HRCCs within the zones provide services and programs to clients including: Employment Insurance, Human Resources Investment, Income Security programs and Labour.
Different levels in the organization are still sorting out their roles. There is a good deal of uncertainty about the respective roles of the various parts of the organization. Zones and HRCCs are not clear on what the role of the Region is especially in the case of pan-regional activities where the Region can still become involved, but in this era of decentralization, it is seldom clear who has the authority to make decisions.
Asked what the strengths of the organizational structure are in terms of the program's ability to meet its objectives, managers' consensus response was "flexibility." EBSM puts most of the discretion in the hands of front-line workers, and several informants said they liked having a high degree of flexibility.
No one interviewed disagreed that there is sufficient flexibility to allow decision making at a local level. HRCCs are pleased with their power to make decisions.
But flexibility can also have its drawbacks. Rules help to ensure the intervention is delivered as designed. Without them, lack of consistency may be introduced between and even within zones. Some HRCC managers interviewed agreed that there was lack of consistency among the HRCCs within service delivery zones. For example, Employment Insurance Officers who were trained in the service delivery model to do certain functions may not be if the HRCC manager does not see it as a priority. Moreover, according to two informants, clients are finding out they're being treated differently because of different interpretations of how to apply the measures in different areas.
The flexibility of EBSM and the devolution of responsibility have been embraced by staff who are long-term employees of an HRCC. However, many of those interviewed for this project indicated that the flexibility of EBSM and devolution of power to local staff could be a challenge to new workers and that even experienced workers need some parameters. Some experienced informants said it was hard to work with so few guidelines.
Ironically, then, flexibility may be simultaneously the greatest strength and weakness of the delivery structure.
The communication system is the cornerstone of any organization and must work well for the organization or program to work well. Communication was explored at the corporate level (see next chapter) and field level of the organization, and between the levels.
Communication at the Zone Level
The Northern Zone has a unique committee — the HRIF Coordinating Committee — to facilitate communications within the zone. This committee is made up of representatives from each of the Northern Zone offices, as well as corporate services representatives. At first the focus of the committee was the impact of the new legislation on program delivery, working out the details of implementation and case management. Over time the role of the committee has evolved to include budgeting, first in an advisory capacity and more recently with some authority for decision making. This zone-wide committee also provides an opportunity to address staff questions and concerns and to share ideas on best practices. Local office staff and committee members feel that this committee allows for strong communication within the zone.
The evaluators learned of no similar committee in the other three zones. There are occasional meetings of managers/staff from the HRCCs in other zones to discuss matters of policy and procedures.
No interviewee felt well informed about what other levels of the delivery network (i.e., region, zones, HRCCs) are doing. In part because of the degree and pace of recent changes, there seems to be a lack of understanding of what the other levels are doing and where they fit in. HRCC managers and staff expressed concern about the communication between zones and between the zone offices and regional offices. The role of regional staff is also unclear at a local level. With more emphasis on decision making at the local level, the role of the regional office has changed to one of consultant rather than decision-maker; however, as one person stated, staff don't always know whom to contact for consultations.
Communication at the HRCC Level
For the most part HRCC management and senior staff felt well informed about EBSM itself. Most said the materials and communications were clear and ample for them to understand the general outlines of the program well.
At the delivery level communications was identified as more of an issue. Informants were especially dissatisfied with the state of communications with external service providers. For example, in one area even very basic information, such as who the case manager is, was not communicated between the HRCC and its main external service provider. However, with such a large change in operations — the shift to comprehensive use of external agencies — one might expect initial problems in communication and operation while the various parties learn their new roles.
Any communication system works on two levels: formal and informal. On an informal level, the system was said to work well. The informal communications system is cordial. HRCC staff said they have a good working relationship with their third party contacts. Most third party representatives interviewed confirmed they enjoyed a good working relationship with their contact person at the HRCC. HRCC staff are perceived to be open and friendly.
Periodic activity reports comprise much of the formal communication system. External agencies are generally required to provide monthly or quarterly reports to the HRCC. But HRCC staff included in the focus groups did not believe the reports are adequate for their intended purposes of monitoring and accountability.
Program monitoring encompasses several key aspects of administering a program: monitoring aspects of the delivery structure under each level; monitoring external service providers; monitoring clients on placement; and management of information. This section examines each in turn.
Overall, key informants were dissatisfied with the monitoring of EBSM. Senior officals were worried about the lack of monitoring at the local level. HRCC managers were concerned about lack of monitoring of clients on work experience projects. Local delivery staff were worried about their own ability to monitor external agencies under contract.
Monitoring of Local Labour Market Partnerships and Job Creation Partnerships was particularly thought to be inadequate. Any monitoring that does take place tends to focus on the delivery of services or activities and most often takes the form of a phone call or other informal contact with those involved. HRCC staff believe that this casual approach to monitoring is insufficient to meet accountability requirements.
HRCC staff note that downsizing has contributed to their inability to monitor agencies and clients. HRCC staff lack the time to visit the sites. Some staff also expressed concern about lack of time, and in some cases expertise, to provide financial monitoring (to ensure money is spent appropriately).
Whereas staff in some HRCCs are committed to maintaining up-to-date client information and record any interventions with a client on the system, others report they do not record all interventions and do not always have the time to keep the system up-to-date.
Management Information Tools
Few positive notes were sounded on management information tools available to help in the management of the program. A few informants liked the Contact IV system for external service providers. Otherwise, feedback was largely unfavourable.
According to virtually all interviewees at the local level, management information tools were inadequate. There is little timely information about interventions or contracts.
The complexity of the management information system, along with the aforementioned lack of resources dedicated to the monitoring of data input, may well be causing data quality problems. For example, the administrative data show a lot fewer Self-Employment cases than key informants said were contracted with annually in the three case studies sites. Staff were also worried that the system was incapable of picking up duplication of service.
Both HRCC and external service providers called for third party access to HRDC systems. Constant phoning by external agencies to check on EI eligibility was universally considered to be inefficient and time-consuming.
In the case studies and focus groups informants were asked what a typical EBSM client would experience upon arriving at the HRCC or external service delivery agency. The process differs somewhat by HRCC, but, in general, it unfolds as follows.
A typical client would go to the HRCC to apply for EI benefits, or to seek assistance of some type at the HRCC or the external service delivery agency. In some areas, the first step is to attend a group information session, where information on Employment Insurance and related programs and services is imparted. The sessions last about an hour.
Other HRCCs do not run group information sessions. Two of the three case study sites used to conduct them but stopped because of staff and space limitations. Also, Halifax staff reported that not many people came to the sessions, and many who did wanted individual attention after the session. It would seem counter-productive, however, to stop group information sessions because some clients want individual attention — one would presume that they were instituted as an efficient way to inform clients of programs and services. Unless this information is no longer being transmitted to clients individually, an obvious deduction is that the process across all clients must take longer without group information sessions.
Targeting and Selection
By the book, the next step should be targeting and selection. Staff from Regional Office ran sessions on this with local staff to help them understand the rationale and importance of appropriate targeting and selection. There is a template for targeting and selecting: it includes all the national targeted groups and youth. Regional informants believed the process is working well.
At some offices targeting and selection is sometimes omitted: targeting and selection criteria exist but staff sometimes move away from them in their effort to help all clients. Offices that did attempt to use targeting and selection found that they often ended up with too many clients and not enough resources because the targeted population is so broad. As a result, it seems that targeting and selection is being reviewed and adjusted in some offices, while essentially being overlooked in others.
Needs Determination
Most HRCC managers and staff said participant needs are identified through needs determination interviews. In case study sites for example, the interview aims to classify clients into one of three categories based on education, work experience and general employability needs: high need; moderate need; and low need.
The next step depends on the level of client need. There are self-service options for low need (i.e., job-ready) clients who require no active interventions. These clients are often referred to a local third party agency contracted to provide Employment Assistance Services.
Moderate need clients — defined as those who don't need upgrading or training, but in need of some employment assistance (usually work experience) — see an Employment and Insurance Officer who sets up a case file and then would select the most appropriate intervention such as Targeted Wage Subsidies or Employment Assistance Services. Clients who have an interest in pursuing self-employment would usually be referred to an external service provider contracted to deliver Self-Employment.
In cases of high need — clients who need career decision-making assistance, the client is referred to an employment counselor. The client and the counsellor develop an action plan which outlines the intended goals and the steps required to reach these goals and the case management process begins.
According to policy, all EBSM clients are supposed to have an action plan. But it is typical for HRCCs to set aside the action plan for low need clients. There are also some offices who do not develop an action plan for moderate needs clients. According to administrative data, about a quarter of EBSM clients had no action plan at any time.
Action plans were described as being a joint effort between clients and employment counsellors. Whereas some clients come in with a specific goal in mind, others need to explore various options before committing to a specific plan. Before approving an action plan HRCC staff look at various factors including the resources the individual has, labour market information for that occupation, and viability of the plan.
Counseling
High need clients are defined as those who need career decision-making assistance as well as upgrading and/or skills training to improve employability. All high need clients see an employment counsellor. Counseling sessions usually last between 90 minutes and two hours and includes the identification and clarification of the employment difficulty, the development of an action plan and implementation and evaluation of the plan.
Since many clients have low education levels, they are classified as high need. Long-term interventions are the norm for these clients. Administrative statistics confirm the overwhelming majority of clients (67 percent) are in the Purchase of training option.
Case Management
Case management provides employees with a process that ensures clients, who are involved in activities designed to meet identified employment needs, receive the on-going support and follow-up necessary to promote successful completion and positive results.
There seemed to be no typical case management procedure. Some offices are dedicated to it while others give it a lower priority. Even within the same area, there is great divergence in the application of the case management process.
For example, in Yarmouth, the HRCC and the Burridge campus of the Nova Scotia Community College, had completely different positions respecting case management. HRCC staff stressed how time consuming it was with resources already stretched while the perception of case management at Burridge was, conversely, very favourable.
Case management has been a priority in the Antigonish office since long before the advent of EBSM. Processes were put in place some time ago at the behest of the local manager. Staff feel that local emphasis on case management allows them to better track the client and the success of the intervention.
There is a need at the management level to ensure field workers understand the importance of case management and to provide adequate resources to permit it to take place. And there is also a need to clarify roles and responsibilities of the various parties. For example, in some HRCCs it has not been made clear whose responsibility it is to monitor clients of external service providers.
Individuals and other organizations in its environment substantially affect an organization. In terms of EBSM, perhaps the most relevant aspect of the environment is use of external service providers.
Why do some HRCCs use external service providers to deliver EBSMs, whereas others have chosen to keep most services in-house? The answer given by most informants is "capacity." Loss of staff at many offices made it impossible to carry out all the tasks internally, providing a compelling reason to use external providers.
Related to capacity is access to expertise. No HRCC respondent believed their office had the money to pay for all the required expertise in-house. This is most evident for the Self-Employment program. Most HRCCs — even Antigonish, which doesn't otherwise use external providers — contract out to local agencies with expertise in guiding new business start-ups. Expertise goes beyond the substantive subject matter; it includes the capability of generating trust with client groups.
There are also basic philosophical differences from one area to the next on the benefits and merits of using external agencies. Undoubtedly a principal reason for such extensive use of external providers in Yarmouth is the philosophy of the HRCC management. They strongly believe in HRDC's new community capacity building mandate.
Besides providing needed expertise, working with external agencies has also helped to stimulate contributions from others in the community in quest of employment for the unemployed and help for local employers (Chapter 6 gives some examples).
However, local staff felt that extensive use of external providers can mean a loss of quality control by the HRCCs. In particular, staff questioned how well external agencies and their clients are monitored. The solution goes beyond a better, more accessible data entry system, although that is an important part. According to HRCC staff, it requires support people to carry out the work-intensive tasks required for monitoring and to make the communication system work.
Many of the HRCC staff included in the focus groups indicated they would prefer to have the work done in-house again.
The critical question in any formative evaluation is: Does the program as operating faithfully reflect the program as designed? That's never an easy question to answer, but it is particularly difficult when it comes to EBSM, because its flexibility makes it difficult to define programs and services. The HRDC Handbook on EBSM provides broad descriptions on each component, essentially leaving interpretation of EBSM and how to apply the guidelines up to discretion of the Human Resources Centres of Canada (HRCC).
Clearly the program's design envisages local flexibility to respond to the needs of the community. EBSM has that to a high degree. In that sense, the program was implemented as planned. However, this level of flexibility raises some concerns particularly with regard to consistency between and within zones.
Another key finding of the evaluation was the absence of timely data on client results that was necessary for fine-tuning programs and services under EBSM. Downsizing, lack of resources, and a need for greater commitment on the part of management have meant local offices do not have access to much-needed results data.
Two other shortcomings identified that could have an impact on program delivery are the lack of a coordinated and consistent approach to delivery throughout the province and uncertainty about rules for the delivery of some program components.
|
Employment Benefits and Support Measures (EBSM) is undertaken in partnership with the Nova Scotia government under the Labour Market Development Agreement (LMDA). The main objective of the Labour Market Development Agreement is "to provide a framework for Canada and Nova Scotia to work jointly to find opportunities through strategic partnerships to benefit Nova Scotians."3 This chapter reviews the partnership between the province and the federal government.
The Canada/Nova Scotia Agreement on a Framework for Strategic Partnerships was signed on April 24, 1997 to undertake joint activities in the area of labour market programs and services in Nova Scotia. EBSM predated this LMDA, having been inaugurated in July of 1996.
Under this agreement, Nova Scotia has adopted a unique cooperative and collaborative arrangement in its dealings with the federal government and labour market programming. While agreeing to work closely together and cooperate in identifying areas of common interest, Canada and Nova Scotia continue to maintain separate responsibility for the delivery of their own labour market programs and services within the existing authorities legislated by their governments.
Areas of joint cooperation and collaboration enabled by the Agreement include, but are not restricted to:
The Canada/Nova Scotia approach is in contrast to other provinces, which accepted the full devolution of Employment Insurance (EI) funded programs and services (New Brunswick, Alberta, Saskatchewan, Quebec and Manitoba), or those which agreed to co-manage employment benefits and measures (Prince Edward Island, Newfoundland).
To comprehend the Canada/Nova Scotia Agreement, it must be understood that it is neither a prescriptive nor monied agreement (unlike the previous training Agreements in place between the federal and provincial government, or the earlier Social Assistance Recipient Agreement). Instead, the Agreement provides a strategic framework through which the two governments jointly identify priorities in which there is mutual advantage for collaboration. The emphasis in Nova Scotia is on serving clients. The Canada/Nova Scotia Agreement represents, in the words of one respondent, "a continuous trust building exercise, a whole new way of doing business." It gives the parties to the Agreement more ability to negotiate on an ongoing basis.
The Canada/Nova Scotia Agreement has been credited by some senior officials for bringing the province and the federal government back together after a period of drifting apart. Under the earlier SAR Agreement, for example, some positive partnerships had been forged between Community Services and Human Resources Development Canada (HRDC) staff. The formalized liaison committees ceased with the ending of the agreement, although informal arrangements continued in some parts of the province. Since the implementation of the Canada/Nova Scotia Agreement these relationships have been re-established across the province.
A hierarchy of committees governs the partnership. The committees determine sharing of costs, mandate, mutual responsibilities and accountability for results. The committee structure is headed by the Joint Management Committee. Co-chaired by the HRDC Nova Scotia Region Director General, and the Senior Advisor with the Labour Market Development Secretariat for the Province, this committee is mandated to collaborate and coordinate efforts to improve federal and provincial labour market programs and services for the benefit of Nova Scotians. Deputies from the Nova Scotia Departments of Community Services, Education and Culture, and Economic Development and Tourism sit on this committee together with their federal colleagues from Atlantic Canada Opportunities Agency, Enterprise Cape Breton Corporation, and Industry Canada.
Under this committee is the Committee of Senior Officials, co-chaired by the Senior Advisor of the Labour Market Development Secretariat and the HRDC Associate Director General. A large committee (18 members or more), its make-up includes representatives from each department attached to the Agreement. On the federal side, members are drawn from field directors and the regional office; head office staff attend from the Province. In addition, co-chairs from the various sub-committees, are invited to attend the Committee of Senior Officials' meetings. The mandate of the Committee of Senior Officials has been to:
Specific roles and responsibilities have been to:
At the base are 12 sub-committees. Membership was drawn from provincial head office and field office staff, and from regional and local Human Resources Centres of Canada (HRCC) staff on the federal side. The function of most of the subcommittees is clear from its name: Co-location (co-location of HRDC, Community Services, Education and Culture and allied Non-Government Organizations); Community Capacity Building; the Glace Bay Project (joint service delivery); Labour Market Information Products; Literacy; Reach-back Client; Skills, Loans and Grants; Support to Job Creation; Youth4; Evaluation; and Communications.
The theory of the committee structure is as follows: By bringing senior officials from the federal and provincial governments to a common table and inviting regional and field staff to participate in joint discussions, differences which in the past may have prevented joint provincial federal cooperation, would be replaced by an understanding of what each jurisdiction has in common.5
One of the first concrete activities of the Canada-Nova Scotia Agreement focused on the development of common approaches and guiding principles in the application of the Targeted Wage Subsidies and Self-Employment program by HRDC and Community Services staff in their work with clients. These approaches and principles were summarized in a manual, and training was provided to all federal and provincial employment staff. This activity was overseen by the TWS/SEA sub-committee, and resulted in a harmonization program launched on April 1, 1998.
One of the significant principles includes the following: That the local office and the individual staff person will hold the decision-making power regarding the appropriateness of using a harmonized program option with a client. Since the harmonization program, provincial staff in some regions of the province have reported a significant shift from their usual centralized decision-making model to local decision-making. Regional staff in the Halifax region, for example, report direct responsibility for most decisions respecting client interventions. This is in sharp contrast to the pre-agreement days when decision-making required head office approval.
The work of the SLG Sub-committee enabled federal and provincial players to understand each others' issues and concerns and, in the opinion of the co-chairs, "has enabled a good deal for Nova Scotians, one of which we can be proud."
That said, the committee structure was not easy. It entailed many hours of discussion and many repeat visits to the drawing board. Nova Scotia also piloted the Skills, Loans and Grants program in two locations (the only province to implement such a pilot) in an effort to jointly work out the many details and issues that arise in a newly launched and cross jurisdictional program.
In the end, the commitment by both jurisdictions to stick with the discussions has resulted in an agreement that includes the HRDC principle of client-centered programming, while guaranteeing the provincial Community College system continued funding for training costs. The agreed-upon model, re-named the Skill Development Employment Benefit, has replaced the earlier proposed loan component with a federal guarantee to the Province to cover the difference in actual cost of training and the cost of tuition. Clients will be expected to contribute towards their training based on their ability.
The negotiations of the committee have also resulted in the addition of enhanced service components including an introductory one-week course aimed at building/refreshing study skills, and use of the Prior Learning Assessment services to a job placement service upon graduation. In the opinion of senior officials, these incremental components, were only possible because of the goodwill generated through the framework of the Agreement.
The emphasis of the co-location committee, to date, has been on identifying sites throughout the province appropriate for bringing HRDC staff together with Community Services staff under one roof. Co-location has occurred in a number of sites, and a report has been completed on the impacts of co-location in two pilot sites.
The existence of the committee has enabled issues arising from the co-locations to be discussed at a common table so that lessons learned can be shared and applied to future co-locations. The committee is increasingly broadening its role from one of planning for co-location, to providing active support to co-locations in the field.
During the interviews related to the partnership, several concerns were raised by informants:
With so many committees and subcommittees, communication is vital to the success of the partnership structure. Unfortunately, as one might predict with the cumbersome committee structure, communication has often been problematic. There was broad concurrence that communication needs to improve to support the Agreement. Moreover, as some informants stated, there hasn't been a lot of output from the committees.
Committee co-chairs had questioned the sheer number of committees initially established under the Agreement; others had indicated that there was not always as much commonality between the jurisdictions as initially thought; still others had questioned the efficiency of the meeting schedule. Many interviewees maintained that LMDA committee duties were given to them without any relief from other duties. Related to this is overlap in committee duties and membership: the committees are required to meet monthly and many people are on several committees.
Leadership has recognized the need to streamline the committee structure and is focusing its efforts on a smaller number of priorities. It has also recognized the need to shift its focus from consultation at the senior levels, to that of support for the growing number of partnerships in place and in development at the regional and local levels. Accordingly, the Agreement is undergoing a period of transition.
A few informants (including some provincial staff) thought the Province had been somewhat disinterested in the Agreement. Sparse attendance at some meetings was offered as evidence of this. Another reason was offered by one provincial informant who said: "It hasn't been a provincial priority to reduce the amount of time a person has been on EI." If the province views the Labour Market Agreement as a vehicle to reduce reliance on EI, it is possible some officials view it as a threat to the social assistance budget.
Some interviewees indicated that little has changed in terms of service delivery at the field level since the signing of the Agreement. In the Halifax area, for example, Community Services was said to be delivering some aspects of EBSM, but some front-line workers claimed there was little communication between themselves and the provincial workers. Even where federal and provincial offices have been co-located, there is some evidence that not much has changed. "Even in those offices where co-location exists, the two departments still deliver the services in isolation from one another."
Although the LMDA partnership is experiencing some challenges, this is to be expected at the early stages of any partnership. The LMDA management committee is addressing most of the concerns uncovered in the evaluation. For example, there is a proposal to completely overhaul the cumbersome committee structure to not only streamline the administration of the Agreement but also to involve local staff more directly in LMDA activity. Local committees reflecting each of the priority areas agreed to will be established in each region to directly ensure and oversee work on initiatives.
A major strength of the Canada-Nova Scotia Agreement is its capacity to evolve to overcome identified weaknesses and to meet unforeseen contingencies. As the Agreement continues to evolve, it is hoped that the work carried out under the Agreement will no longer be seen as an add-on but rather as an integral part of staffs' ongoing responsibilities.
|
Using information from the case studies, this chapter outlines how each EBSM component is delivered in the three sites visited. The case studies were meant to explore how EBSM was implemented in different areas of the province. The case study, then, is directly tied to the primary purpose of the process evaluation: to determine how the program has been implemented and compare that to program design.
Given the program's high degree of flexibility, it should come as no surprise to learn that there are a variety of procedures in place for delivering each EBSM component. The three Human Resources Centres of Canada (HRCCs) were selected to represent the range of models represented in Nova Scotia. Case studies were conducted at HRCCs in Antigonish, Sydney and Yarmouth.
Yarmouth HRCC was chosen as one location because it makes extensive use of external service delivery agents in quest of capacity building. At the other end of the spectrum is Antigonish HRCC, which has chosen to stay with the traditional Human Resources Development Canada (HRDC) client-centred approach, with very little reliance on external agencies. This HRCC did not suffer the loss of staff seen in other offices during the downsizing that took place in 1995. The staff here are experienced and established in the community and have chosen to keep the work in-house. Sydney HRCC was selected to represent a model of community capacity building in between the extremes.
The chapter begins with a brief section to set the labour market context for the three sites.
The Antigonish HRCC is responsible for the delivery of HRDC programs and services in the counties of Antigonish and Guysborough, a large, sparsely populated area in Northern Nova Scotia. Much of the area has traditionally relied on the fishery and forestry industries for employment. Loss of jobs in these industries has resulted in pockets of high unemployment - the town of Canso was particularly hard hit by job losses.
The Guysborough Regional Development Authority describes the county as experiencing "chronic unemployment, seasonal employment and underemployment. The average taxable income is 75 percent of the Nova Scotia average while 55 percent of the workforce has an active Employment Insurance (EI) claim." Guysborough's unemployment rate is currently around 20 percent. Despite the current situation, the area is optimistic about the future because of the Sable development that is taking place which is hoped to offset some of the losses, particularly in the community of Guysborough. HRCC staff indicated that because of the influence of the university, hospital and service industry, the local rate of unemployment in Antigonish is currently about 10 percent. The Antigonish HRCC Operational Plan states the local priorities as tourism, information technology and the Sable Offshore Energy Project.
The geographic area served by HRCC Sydney includes the Cape Breton Regional Municipality, which amalgamated eight former municipal units on August 1, 1995. The population served by HRCC Sydney is approximately 128,750 (1991 census). The annual average rate of unemployment for Cape Breton as a whole was 23.2 percent for 1996, which is significantly above the 1996 Nova Scotia annual average of 12.6 percent. Unofficial statistics put the unemployment rate of Industrial Cape Breton at 40 percent or higher. The recent announcements to close out DEVCO will drive these rates even higher.
The Industrial Cape Breton area faces enormous labour market challenges. High unemployment has placed increased pressure on agencies responsible for delivering income support, including HRDC. In addition, pressure is increased on agencies mandated to deal with the social problems, which tend to accompany severe unemployment. Out-migration has been particularly common among the area's youth population.
The Yarmouth area is predominantly rural. Fishing, and manufacturing are the primary economic sectors for the Yarmouth HRCC. Future areas for employment growth for the Zone as a whole are projected to be in aquaculture, recycling and home health care. The tourism industry also holds potential for an increased emphasis.
The unemployment rate for the Southern Zone as whole has been steadily increasing over the past five years: Yarmouth's unemployment rate stood at 13.8 percent in April, 1999. In the area covered by the Yarmouth HRCC (Yarmouth and Shelburne counties), the downswing in the ground fishery has led to unemployment for large numbers of ground fish harvesters and processors. In addition other sectors of the fishery, including shellfish, have recently suffered displacement because of fleet restructuring. Mechanization will continue to put labourers out of work in the fishing and forestry industries. Many of the displaced workers have low levels of education and literacy, and a high proportion of these individuals are not "job ready" in the sense they lack a broad range of work skills/experience and have fairly narrow industry specific skills.
Over the last decade jobs that have disappeared in primary resources and manufacturing occupations were replaced by jobs in sales and service. Restructuring has been a challenge given the disparity of wages between the two, and the higher levels of education required for the new jobs.
HRCCs may contract with a local external service delivery agency to provide Employment Assistance Services — most of which involve job search assistance — or may provide such services in-house. In most cases, external agencies deliver the services. External agencies show clients how to do job searches, walk them through computer programs used for the search, and review application procedures, résumé writing, and interview techniques. Job postings and the job bank are also featured. At the discretion of the HRCC, external agencies may also offer full case management services on behalf of the HRCC.
Employment Assistance Services may provide service to all Canadians, not just EI or reach back clients (e.g., self-service information on the labour market or job openings). Employment Assistance Services can open access to job search services to those who are otherwise ineligible for EBSM. Employment Assistance Services is also delivered through outreach services to some targeted groups (Visible minorities, women re-entering the labour market, persons with disabilities, and youth for example).
In Antigonish, the HRCC takes care of Employment Assistance Services without external agencies. Yarmouth contracts out most Employment Assistance Services, even case management of some clients. Sydney also contracts out most such services (except case management) to a local agency.
Local Labour Market Partnerships are generally handled by the project officers in the HRCCs. HRCCs with an emphasis on community capacity building, such as Yarmouth, go out in the community to identify potential partners for Local Labour Market Partnerships projects. In other places such as Antigonish and Sydney, HRCCs more often wait for community groups to come to the HRCC with proposals for partnering activity. HRCC staff in Antigonish do not go out and actively search out potential projects, but they do represent HRDC on a variety of community boards and committees where ideas for projects are often broached. Enough organizations seek HRDC assistance in Antigonish and Sydney to make community partnerships a significant element of EBSM in each area. During the study period of January 1997 to June 1998, there were 25 Labour Market Partnerships in Yarmouth, 17 in Antigonish and 14 in Sydney.
Although Sydney had the least number of Local Labour Market Partnerships, they tended to be the largest in terms of numbers of participants and HRDC contribution.
| Area | Number of LLMPs | Mean Number of Participants | Mean HRCC Contribution |
| Antigonish | 17 | 1.8 | $17,688 |
| Sydney | 14 | 4.6 | $37,560 |
| Yarmouth | 25 | 2.3 | $26,412 |
Antigonish and Yarmouth have Regional Development Authorities (RDAs) that play important intermediary roles in the Local Labour Market Partnerships component. These organizations work with HRDC to identify the best approach and best program (from HRDC and others) for employers, community agencies and others who are looking for assistance with labour market programs. RDAs also have ongoing consultations with the community to identify economic development activities and appropriate partners.
Local Labour Market Partnerships have been used to fund a wide variety of projects including: hiring coordinators for groups serving the EBSM target groups, financial support for career fairs, creating community economic development advertising materials, funding research to support community development, and financial support for other regional development activities.
One project in Yarmouth combined Local Labour Market Partnerships and Job Creation Partnerships (although the partnership predated EBSM). A few years back, when Dominion Textiles decided to close their plant, approximately 300 employees were put out of work. The Yarmouth County Industrial Commission resolved to study the situation to determine if there was a way to make use of the facility. Enlisting the support of various government departments including HRDC, the Commission arranged for the building to be renovated so local employers could purchase or rent space. Three parts were sectioned off to be sold, and the money was reinvested to enable further renovation. Clients were hired under the job creation partnership to do the renovation work. At present, several employers operate there on a full time basis.
Another example of a Local Labour Market Partnerships is the Yarmouth Learning Centre. It has four partners: HRDC; the Chamber of Commerce (initial sponsor); Community Services; and Burridge Community College. In addition to EI-eligible clients, the new Centre will take social assistance clients, youth and other people with no EI attachment. HRDC is bringing funding to the partnership through Local Labour Market Partnerships, Purchase of training and Employment Assistance Services. The Community College is providing the facility and overhead costs and access to its resources. Community Services is providing some funding for their clients.
A completely different Local Labour Market Partnerships example comes from the HRCC Sydney case study. The Virtual Firm, a special project of the University College of Cape Breton in partnership with HRDC, is a learning model for the unemployed. It functions like a real business enterprise, interacting with other Virtual Firms in a closed network. In partnership with a sponsor company, the Virtual Firm sells a product in a "virtual" marketplace to meet the demands of other network participants.
The key focus of the Virtual Firm is to help participants find employment and use hands-on training to enhance their professional knowledge and skills in a 15-week period. The Virtual Firm model allows individuals to develop and enhance their business skills and enables them to network, make business contacts and benefit from the business expertise of well-qualified supporters with a primary interest in helping individuals find meaningful employment.
In Sydney's Virtual Firm, clients run the business on Tuesdays, Thursdays and Fridays to improve their skills. Mondays and Wednesdays are spent job searching. Employers, were said to be beginning to buy into the concept, accepting the firm as real experience.
The Antigonish County Adult Learning Association is another example of a Local Labour Market Partnerships, although the association was in existence prior to the establishment of EBSM. The organization was small and felt that a coordinator position was needed so it could better meet the needs of the community. There are now a number of partners involved: the HRCC and the National Literacy Secretariat jointly fund the coordinator position, the province funds instructor salaries, a small grant from the Provincial Literacy Coalition helps with learner related expenses, the school board and municipality provide space, and other private sector partners offer technical support, accounting services and consulting. The Association's Board of Directors, which meets monthly, is made up of the partnership and community representatives.
These examples, which seem to have no commonalties whatsoever, illustrate the range of activities supported through this measure. For the most part, use of the measure was considered to be beneficial by the partners.
Some interviewees, though, attested that Local Labour Market Partnerships are used in a different manner from what was intended. They said the measure is used when staff want to support something in the community that doesn't fit anything else. This, however, does not seem inconsistent with policy - the HRCC handbook on EBSM is very broad in its definition of what types of projects might be acceptable. A broad interpretation of the handbook would be that any labour market initiative fits under Local Labour Market Partnerships as long as there is at least one community partner.
The Job Creation Partnerships benefit in practice is administered in a similar fashion to the Local Labour Market Partnerships. Project officers administer the contracts and are responsible for monitoring.
Staff in HRCC Yarmouth, for example, go out into the community to identify and develop projects. The Regional Development Authority helps identify projects and works with HRDC and community groups to develop them. At the other extreme, Antigonish and Sydney are less proactive in this area. Community partners generally approach the HRCC to request funding for a Job Creation Partnerships.
The approach to developing Job Creation Partnerships (proactive versus reactive) does not appear to affect the number of agreements signed. During the study period of January 1997 to June 1998, there were 27 Job Creation Partnerships in Yarmouth, 23 in Antigonish and 29 in Sydney. As with the Local Labour Market Partnerships, the Job Creation Partnerships projects tended to be the largest in Sydney.
| Area | Number of JCPs | Mean Number of Participants | Mean HRDC Contribution |
| Antigonish | 23 | 2.1 | $1,359 |
| Sydney | 29 | 7.3 | $8,049 |
| Yarmouth | 27 | 3.3 | $2,289 |
Positions are most often short-term and tend to be used to help the community as well as provide the client with some valuable work experience. Typically the client is receiving EI and continues to receive benefits while working on projects under the Job Creation Partnerships. These benefits may be topped up by the employer or the HRCC. Antigonish staff indicated that the local policy is for the employer to top up the benefits if the partnership is within the private sector and for HRCC to top up the benefits if the partnership is with a non-profit organization, whose contribution to the partnership would be more "in kind."
Job Creation Partnerships is supposed to create opportunities for employment and work experience leading to economic growth, long-term jobs and reduced reliance on EI benefits. Some informants said the component was supposed to create "sustainable employment." That phrase shows up in some HRDC descriptions of Job Creation Partnerships. These interviewees maintained that the intent was to move away from the former Section 25 make-work project to subsidizing sustainable jobs. However areas of high unemployment may pose a challenge to the requirement for job sustainability for approval of projects. Some staff say they had to take some decisions around what is sustainable.
Other informants said there was no such requirement for job sustainability. The HRDC Handbook on EBSM states that "Job Creation Partnerships (JCP) can be used wherever there is a development need and an opportunity to give unemployed workers the chance to gain some work experience that can lead to long-term employment." The handbook seems to place no constraints whatever on the use of Job Creation Partnerships for creating permanent jobs: HRCCs may define for themselves how best to use the measure.
In any case, HRCCs sometimes overlook job sustainability because they feel it is simply not realistic. Managers and project officers claim they are taking a broader view of job creation. They can fund a project based on the hope that new skills, enhanced experience, and a stronger resume will benefit the client in the long run, even if the current project does not lead to full time employment. As well, HRCCs are looking at the community development aspects of Job Creation Partnerships.
A good example raised by HRCC staff in Sydney was a boardwalk funded through Job Creation Partnerships. HRCC staff knew the people involved in building the boardwalk would not be employed at it long-term, but several businesses — boutiques and a restaurant — followed, which did produce permanent jobs. In this case, the workers claimed the infrastructure construction lead indirectly to these new jobs.
The Dominion Textiles project cited above was posited as a good example of the use of Job Creation Partnerships in Yarmouth. Clients were hired under Job Creation Partnerships to carry out the renovation work. They were not expected to get permanent jobs at the site, but the work they did was expected to facilitate the creation of other jobs at the facility.
Another example of Job Creation Partnerships in the Sydney area is Two Rivers, which used to be a provincially run wildlife park. The park is now a not-for-profit organization. HRDC provided funds to hire about six persons. One of these was hired through Job Creation Partnerships to do the organization's books: she has since been brought on full-time by Two Rivers. Two or three others were carpenters hired to build pens for the animals, trails, and keep up the grounds. They have also been brought on full-time.
HRCC staff and external service deliverers, however, were as one in condemning the non-insurable aspect of Job Creation Partnerships.
Targeted Wage Subsidies was intended as a tool to provide clients with direct work experience that can lead to long-term employment. In many HRCCs, including Yarmouth and Antigonish, counselors give letters to the clients indicating their eligibility for the program, so they can do their own marketing. Targeted Wage Subsidies is thus seen as a client oriented program. For those who are considered job-ready, the subsidy is seen as a means of gaining entry to what is hoped to be long-term employment.
In some areas or in particular circumstances, however, Targeted Wage Subsidies is employer driven. Not only does the HRCC get calls from employers looking to hire an EI client with a wage subsidy but the employer may have a particular client in mind.
This approach by employers, choosing an employee and asking HRDC to share the wages, does not appear to counter policy for program delivery.
The HRDC Handbook on EBSM seems to give HRCCs this latitude, stating the following: An employer's proposal to use a targeted wage subsidy may be evaluated based on the value of the work experience it offers to participants, on the likelihood that the jobs will continue in the long term, or on both of these criteria. If an EI eligible client has taken the initiative to market himself with employers to generate his own opportunities and a wage subsidy agreement is what this client needs, then staff are to make a decision based on what makes the best sense for that client to achieve results.
Staff feel the measures meet the needs of employers. In areas where the measure is client-centred, interviewees were less disposed to think that employer needs are met. For example, when an employer calls asking for a subsidized worker, staff in client-centred HRCCs are supposed to determine if the employer has had subsidized employees previously and what the outcome was. If past clients were laid off when or soon after the subsidy ended, staff are less disposed to sending another subsidized client to that employer.
The wage subsidy is generally 50 percent of the hourly wage. The length of the placement is negotiated with the client. Usually the placement lasts for 16 to 26 weeks.
During the interviews with employers, some positive examples of the use of Targeted Wage Subsidies for training and retaining EBSM clients were raised. A bakery in Sydney was about to lose a baker who was relocating. Before the baker left, the bakery hired an EI client under the Targeted Wage Subsidies program and had the original baker train the client. Targeted Wage Subsidies paid 50 percent of the salary and the client is now employed full-time with the bakery. Later they needed to bring on an additional employee and again looked to the Targeted Wage Subsidies to subsidize the wages (50 percent) while the second employee was in training. This client is also now employed full-time by the bakery.
In Antigonish, an employer has used the program on several occasions to subsidize employees that require training on the job to carry out specific duties. Generally the positions that were being filled were said to be created by a growing industry. The employer stated that he does not submit a proposal to HRDC for a subsidy unless clients get full-time employment at the end. Local HRCC staff were praised for their follow up even after funding ends. One area for improvement, in this employer's opinion, is better screening of clients by HRCC staff.
As mentioned already, most EBSM clients participated in a training intervention. In general, counselors have an interview with clients to assess the employment need and to develop a plan of action. In many instances, the client has made a career choice and comes to the counselling interview with a good idea of the training he or she desires. Counselors make sure all clients know about their labour market prospects for a chosen career. Career development and exploration workshops are offered through Employment Assistance Services for those who need assistance with career choice.
Included in the development of the action plan is negotiation of financial assistance with the client - an assessment of what the client can contribute to the intervention. Nearly all staff interviewed felt that the client is much more committed to the action plan when it is not being entirely financed by HRDC. Also staff find that clients that are contributing into their plan do more preparatory work in terms of making sure it is the right career path for them.
The extent of the contribution is very much influenced by the individual's situation, with some individual's contributing only their EI benefits while others are unable to contribute. Office vary in their approach to financial negotiations with clients. One local office stated the departure point on the negotiations is 60/40, i.e. it is assumed the client can cover 40 percent of the cost of training unless he or she proves otherwise. Few do prove otherwise: staff estimated that 75 to 80 percent of the people they deal with fit that 60/40 bracket.
Although training in negotiating these financial agreements was available to staff and most had gone through such training, there is still considerable variation in the extent of the negotiations, with some offices expecting more from the client than other offices, and even some staff within the same office expecting more than others.
The delivery of Self-Employment was common in the three HRCCs visited for the case studies. Self-Employment is delivered by external service providers, mostly Business Development Agencies.
Clients who express an interest in self-employment are referred to the responsible Agency. Some agencies offer an information session on what is involved in starting and running a business, what is required in a business plan, what the program eligibility requirements are, and what other sources of financing are available for clients. Others begin with an in-depth interview covering applicant background and present status.
After the initial meeting, the external agency determines client eligibility through HRCC staff. EI eligibility (or reach back status) is a necessary but insufficient condition. The decision to approve a client for Self-Employment is based on a viable business plan.
After an orientation session, some preliminary training and planning, clients are asked to do some research on their business and develop a business plan. On the basis of the viability of the plan, the external delivery agent makes a recommendation to HRCC. By and large, the HRCCs follow the recommendation.
According to case study informants, the most important benefit for clients is the ability to collect EI benefits — income to rely on while getting the business under way (this was confirmed by clients in the survey). Another important benefit is business related courses, taught by outside experts, in key areas such as taxation, bookkeeping, and understanding financial statements.
Self-Employment clients can stay on the program for up to 52 weeks. It is up to the delivery agent to recommend the length of the intervention. In Antigonish, the full 52 weeks is only available to certain target groups such as the disabled, visible minorities and youth. Other areas grant 52 weeks of benefits to all clients.
Also negotiated with the client is a personal investment in the business. The Self-Employment component requires that clients have 25 percent equity in the business (20 percent in Antigonish), which can be cash or in-kind.
The external agencies monitor the performance of the new business through site visits and by looking at the monthly statement of income and expenses that has to be submitted. If any problems are evident, assistance is offered.
All three agencies involved in the case study sites had done surveys of former clients and all maintained the Self-Employment option was highly successful. The uniformity of their findings increases confidence in the results: Yarmouth and Antigonish found that 76 percent of the clients they had contracted with since 1992/93 are still in business; Sydney's success rate was reported at 77 percent. Survey results suggest the success rate for Self-Employment is close to 70 percent.
Types of businesses started include: service, retail, tourism, forestry, agriculture, fisheries, manufacturing children's clothing, electrical business, video distribution, photography, manufacture of heavy equipment, carpet cleaning, a barter service, pet grooming, graphic works, tool sales, furniture refinishing, fast food restaurant, dessert parlours, property maintenance, accountants, daycare, trucking, massage therapy, prosthetics, hair salon, shuttle service, second hand clothing shop, crafts, aquaculture, restaurants, interior decorator, shoe store, specialized printing services, automobile diagnostic services, floor installation, greenhouse and a sawmill. Several of these businesses have resulted in the creation of new jobs for additional employees, however, support of some sort (possibly EBSM) is often sought for these positions.
It is clear from the three case studies that there are wide variances between HRCCs in how they have chosen to deliver EBSM. Is any one delivery model better?
One measure is the clients' perceptions. Survey data can be used to compare the mean grades given by clients to EBSM overall, and to various services by case study area. Overall, clients gave Antigonish and Yarmouth a B +, and Sydney a B-, a significant difference.6 Table 6.3 shows satisfaction levels with key aspects of EBSM.
| Service | Antigonish | Sydney | Yarmouth | Province |
| Increased Motivation to Work | B + | B - | B - | B |
| Helped Develop Career Action Plan | B - | C + | B - | B |
| Improved Job Skills | B + | B | B - | B |
| Improved Job Search Skills | B - | C | C + | B - |
| Helped Find a Permanent Job | C | D + | D | C |
| Provided Work Experience | B | B — | B + | B |
| Upgraded Educational Skills | B | B + | B | B + |
| Overall Grade | B+ | B - | B + | B |
For the most part Antigonish gets the highest grades, which one might predict based on its client-centred approach. In one area, educational upgrading, Sydney clients gave the highest marks, corresponding to Sydney's focus on training and youth upgrading.
Survey data can also be used to provide some preliminary outcomes by case study area. Please note, however, that these outcomes are not necessarily the result of the program: clients in the different areas could have been very different and have different labour market conditions.
Clients in the HRCC Yarmouth area were most likely to be working at the time of the survey: nearly three-quarters were employed, as opposed to just over half in Sydney and Antigonish. But, the same outcomes were experienced for individuals from the same areas who did not participate in the program.
In terms of earnings improvement in each area for participants and non-participants, there were no significant differences across the three offices in earnings improvement pre- and post-program (using 1996 and 1998 earnings). Neither were there significant differences in earnings gains between participants and non-participants in any region.
Clients in Antigonish and Sydney received slightly less EI benefits in 1998 than did non-participants in those areas, though the differences were not significant (Table 6.4). In Yarmouth, however, participants relied on EI benefits to a significantly lesser extent than did non-participants in 1998 (at the 10 percent level of significance). Again, though, this does not necessarily connote negative or positive impacts of the program: the groups may have been very different before the program.
| Area | Participants | Non-participants |
| Antigonish | $3,793 | $3,884 |
| Sydney | $4,027 | $4,171 |
| Yarmouth | $2,786 | $3,709 |
| Province | $3,157 | $2,032 |
Without a sophisticated econometric analysis, no definitive verdict can be rendered on the impact of EBSM in each individual region. In any case, it is not apparent that the type of model used in the HRCCs has affected the impact of the program to any great extent. The local labour market is probably a much more important factor.
|
This chapter draws profiles of Nova Scotia EBSM participants and non-participants. Its value is to give the reader a good understanding of the program and its clients in advance of presenting outcomes and impacts.
Data were drawn from the follow-up surveys and Human Resources Development Canada (HRDC) administrative databases. Demographic information such as age, gender, mother tongue, and ethnicity were drawn from the administrative files. Where administrative data are not available or incomplete (for example, the education data are not available on all clients) survey results are presented.
The main thrust of the analysis to follow is to compare participants in the five components with each other and with non-participants. It begins with a brief look at the number of clients by component7. The chapter then turns to client demographics, followed by an examination of overall client satisfaction with EBSM.
In total, 10,956 individuals participated in Nova Scotia EBSM with a start date after January 1, 1997 and an end date before or on June 30, 1998, the time period designated for the study.8 Because some clients participated in two or three different EBSM interventions, the total number of interventions exceeds the number of individuals participating.
To avoid counting any individual more than once in the analysis, all multiple intervention participants were classified into one component as follows: Since the total number of Self-Employment cases was low, all individuals who participated in it were designated as Self-Employment, even if they also participated in a different EBSM. All other multiple-intervention individuals were assigned to the component with fewest cases. That yielded the distribution in the last column of Table 7.1.
| EBSM Component | Number of Interventions | Number of Individuals |
| Employment Assistance Services | 1,368 | 1,276 |
| Job Creation Partnerships | 803 | 803 |
| Purchase of training/Feepayer | 7,845 | 7,361 |
| Self-Employment Assistance | 232 | 232 |
| Targeted Wage Subsidies | 1,297 | 1,284 |
| Whole Program | 11,545 | 10,956 |
About 5 percent had more than one different type of EBSM intervention. In total 553 clients had two different EBSM interventions and 18 had three during the study period. Three-quarters of those who had multiple different interventions began in a Purchase of training course: the usual pattern was Purchase of training to Employment Assistance Services to Targeted Wage Subsidies.
With a start date on or after January 1, 1997 and an end date on or before June 30, 1998, there were 294 Job Creation Partnerships and 221 Local Labour Market Partnerships in Nova Scotia.
According to administrative data the average (mean) number of weeks spent in EBSM was 16.3 weeks (including multiple interventions). The mean differed by component (Table 7.2) ranging from 10.2 weeks for Employment Assistance Services interventions and 20.8 for Self-Employment.
| EBSM Component | Mean Weeks |
| Employment Assistance Services | 10.2 |
| Job Creation Partnerships | 16.4 |
| Purchase of training | 16.7 |
| Self-Employment | 20.8 |
| Targeted Wage Subsidies | 18.8 |
| EBSM | 16.3 |
| N = 10,955 | |
Chart 7.1 shows the distribution of EBSM clients by region of the province. Although the largest proportion of clients reside in the Halifax Metropolitan area — as might be expected on the basis of population — the overall distribution is fairly even across the four regions.
Distribution by region of the province differed somewhat across EBSM components (Table 7.3). Purchase of training predominated in all four regions, but beyond that each region tended towards a different component: Job Creation Partnerships in Industrial Cape Breton, Employment Assistance Services in Halifax Metro; Targeted Wage Subsidies in South-Shore Valley and Northern Nova Scotia.
| EBSM Component | Industrial Cape Breton | Northern Nova Scotia | Halifax Metro | South Shore-Valley |
| Employment Assistance Services | 6.4% | 5.7% | 25.4% | 6.7% |
| Job Creation Partnerships (clients) | 15.9 | 7.3 | 3.6 | 4.4 |
| Purchase of training | 70.0 | 69.6 | 62.5 | 61.8 |
| Self-Employment | 1.0 | 3.3 | 1.9 | 3.0 |
| Targeted Wage Subsidies | 6.8 | 14.1 | 6.6 | 24.1 |
| Note: percents add down. N = 10,268 |
||||
As Table 7.4 shows, use of Local Labour Market Partnerships was predominant in the Northern zone. The greatest percentage of Job Creation Partnerships was in Halifax Metro.
| EBSM Component | Industrial Cape Breton | Northern Nova Scotia | Halifax Metro | South Shore-Valley |
| Local Labour Market Partnership Projects | 9.1% | 41.5% | 28.0% | 21.3% |
| Job Creation Partnership Projects | 20.2 | 28.4 | 32.9 | 18.5 |
| Note: percents add across. N = 456 | ||||
As the female/male ratio of non-participants suggests, the population of those eligible for EBSM is slightly more male than female. Yet two-thirds of those who participated in EBSM during the study period were men (Chart 7.2). The only component that women and men were equally likely to use was Employment Assistance Services. The ratio of men to women in the Purchase of training option was 7 to 3.
The average age of EBSM participants was 35 years; non-participants were somewhat older on average, 39.6 years (Table 7.5). Self-Employment clients were the oldest on average, Purchase of training clients the youngest.
| EBSM Component | Mean Age | N |
| Employment Assistance Services | 35.3 | 1,232 |
| Job Creation Partnerships | 36 | 785 |
| Purchase of training | 34.7 | 7,313 |
| Self-Employment | 40.2 | 227 |
| Targeted Wage Subsidies | 34.8 | 1,266 |
| EBSM | 35 | 10,823 |
| Non-participants | 39.6 | 16,290 |
English was the mother tongue for about 99 percent of those participating in EBSM (although data was missing for about 10 percent of the cases). Only 0.3 percent said their first language was French and still only spoke French.
Participants most comfortable speaking French were asked in the survey if they received employment services in the language of their choice. Eleven percent said no (but note this represents only three respondents).
Approximately 3 percent of EBSM participants were coded as belonging to a visible minority group. There was little difference across components.
Nearly 4 percent of participants were Aboriginal. There was more of a difference by component in this target group, ranging from 1 percent of Targeted Wage Subsidies clients to 10 percent of Job Creation Partnerships clients.
Some 7 percent of participants and 9 percent of non-participants affirmed that they were limited in the kind or amount of work they could do because of a long-term physical condition, mental condition or health problem. The proportion of participants with disabilities differed significantly across EBSM components: 15 percent of Employment Assistance Services, 14 percent of Self-Employment, 7 percent of Targeted Wage Subsidies, 6 percent of Purchase of training and 4 percent of Job Creation Partnerships clients had a disability.9
Chart 7.3 displays clients' overall level of satisfaction with EBSM. As is evident, most participants (78 percent) thought EBSM was good or excellent. Few gave the program a failing grade (4 percent) or a D (5 percent). The mean overall grade was B.10 Clients in every EBSM component gave a mean overall grade of B, except for Employment Assistance Services clients who gave a B-.
Women rated EBSM slightly higher than did men: B + versus B.11 Northern Nova Scotia clients and South Shore-Valley clients gave EBSM a B+; Industrial Cape Breton clients gave a B, Halifax Metro clients gave a B - .12
The most often given reason for a low grade was lack of encouragement or help from staff (15 percent). Other reasons included not enough financial assistance (11 percent), not informed about a particular service (11 percent), lack of help finding employment (9 percent), too long a wait for service or check (8 percent), bad outcome (7 percent), would not continue funding (5 percent), did not receive assistance sought (4 percent), not enough available staff (4 percent), poor program organization (4 percent), and poor service (4 percent).
13The next table lists the mean grade given to EBSM by its clients in achieving its major goals. Results are presented by EBSM component. Purchase of training clients were most pleased with the program: they were significantly happier with their intervention than were Employment Assistance Services clients.14
By EBSM goal, the lowest grades by a considerable margin were given to the program's ability to help its clients find a permanent job.
| Service | EAS | JCP | POT | SEA | TWS | EBSM |
| Increased Motivation to Work | B - | B | B + | B | B | B |
| Helped Develop Career Action Plan | C + | B | B | B | B - | B |
| Improved Job Skills | C + | B | B + | C + | B | B |
| Improved Job Search Skills | B - | B - | B - | C + | B - | B - |
| Helped Find a Permanent Job | C - | C | C + | C + | C + | C |
| Provided Work Experience | — | B + | — | — | B | B |
| Upgraded Educational Skills | — | — | B + | — | — | B + |
| Overall Grade | B - | B | B | B | B | B |
| Stats (between EBSMs) | F = 3.4, df=4/998, p <0.02 | |||||
| N = 1,003 | ||||||
The grade bestowed EBSM for helping to get a permanent job does not necessarily imply that clients were disappointed with how well they were prepared for a new job. In fact, as the next graph demonstrates, 88 percent of the participants felt fairly or very well prepared for a new job as a result of their intervention. The low grade for finding a permanent job may therefore be a comment on client perception of the job market rather than the program.
|
The purpose of this chapter is to lay a solid foundation for the econometric analysis to follow in the next chapter. The descriptive findings on outcome15 are more intuitive than the much more complex econometric analysis; a good understanding of the basic outcomes presented in this chapter will help the reader understand the econometric analysis. A note of caution is in order, though: differences between participants and non-participants cannot be attributed to Employment Benefits and Support Measures (EBSM) until the econometric models control for outside influences. This modeling will be presented in the next chapter.
This section examines the major outcomes of interest to labour market programs such as EBSM: time spent working, time spent in school and unemployed; earnings; use of income support; and educational achievement.
For the longitudinal econometric analysis to follow, it is necessary to establish preprogram and post-program labour market activity. The survey asked participants and non-participants to account for their time spent working, unemployed, and in school since 1995. The results are portrayed in Table 8.1.
Before EBSM, non-participants spent more time than participants working and less time in school. Pre-program time spent unemployed (and not in school) was somewhat higher for participants. For participants, time spent unemployed held steady over the four years. Non-participants showed a similar pattern although time spent unemployed was somewhat higher in 1997 and 1998 than it had been earlier. During 1997, when the survey sample was participating in EBSM, participants' time spent in school increased markedly. Participants were significantly less likely to be employed than non-participants before the program. But this wasn't because they were more apt to be unemployed; rather they were more likely to be in school.
| Activity | 1995 | 1996 | 1997 | 1998 | ||||
| Parts | Non-parts | Parts | Non-parts | Parts | Non-parts | Parts | Non-parts | |
| In school and not working | 1.0 | 0.4 | 0.9 | 0.2 | 1.9 | 0.2 | 1.4 | 0.2 |
| In school and working | 0.3 | 0.3 | 0.3 | 0.3 | 0.5 | 0.2 | 0.4 | 0.2 |
| Working and not in school | 7.3 | 8.3 | 7.3 | 8.1 | 6.3 | 7.7 | 6.7 | 7.7 |
| Not working and not in school | 3.4 | 3.0 | 3.5 | 3.4 | 3.3 | 3.9 | 3.4 | 3.9 |
| Total Months | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 |
The next chapter will isolate the impact of the program, but for now the influence of being in the program can be removed by looking at how those who finished participating in EBSM in 1997 spent their time in 1998. Chart 8.1 shows that participants out of the program in 1997 spent about 31 percent of 1998 unemployed, 64 percent working, 3 percent in school and 2 percent in school and working. Non-participants spent 32 percent of 1998 unemployed, 65 percent employed, 1 percent in school and 2 percent working and in school. The only statistically significant difference between the groups was time spent in school.16
Table 8.2 shows the average annual earnings of participants and non-participants from 1992 to 1998. Average pre-program earnings were low, never reaching $20,000 for any subgroup in any year (note that those with zero income are excluded). By component, Job Creation Partnerships clients had consistently earned the least. Until 1996, Self-Employment clients earned the most. Non-participants consistently earned more than participants. The familiar "Ashenfelter Dip," where earnings fall just before participation in a labour market program, is in evidence for participants in every component, especially Self-Employment. By 1997, Self-Employment clients, most of whom were attempting to get their business going, had the lowest earnings.
Participants reported mean gross earnings of $17,599 from all jobs in 1998; non-participants earned $19,572, a difference of $1,973. The difference did not quite reach statistical significance, however.17 The jump for participants' gross earnings from 1997 to 1998 is certainly encouraging: it was over twice as large as the rise for the comparison group.18 From Table 8.1 it is clear that participants worked only 6 percent more time in 1998 than in 1997 on average, so most of the increment must be attributable to better-paying jobs. It is possible however that some of the earnings gains are subsidized since many were still participating in 1998. But if the analysis is limited to EBSM clients who completed their intervention before 1998, the picture looks better: mean 1998 gross earned income for participants was $20,241, $669 more than non-participants. This difference however falls short of significance.19
In 1998, Employment Assistance Services clients still earned less than they had preprogram. Every other EBSM group earned more in 1998 than they had before the program.
| EBSM Component | 1992 | 1993 | 1994 | 1995 | 1996 | 1997 | 1998 |
| Employment Assistance Services | $14,036 | $14,192 | $14,004 | $13,890 | $13,289 | $10,444 | $12,159 |
| Job Creation Partnerships | $9,555 | $9,205 | $8,877 | $9,032 | $7,606 | $7,282 | $12,586 |
| Purchase of training | $13,064 | $13,043 | $13,448 | $14,166 | $13,712 | $13,285 | $18,963 |
| Self-Employment | $18,337 | $19,065 | $19,200 | $17,067 | $11,353 | $5,794 | $21,578 |
| Targeted Wage Subsidies | $11,865 | $11,681 | $11,480 | $10,761 | $9,310 | $9,520 | $15,531 |
| EBSM | $12,893 | $12,855 | $13,059 | $13,446 | $12,726 | $12,022 | $17,599 |
| Non-participants | $15,058 | $15,167 | $15,049 | $14,847 | $15,419 | $17,161 | $19,572 |
Data in this section looks at UI/EI20 histories of EBSM participants and non-participants since 1992 (Table 8.3). Prior to EBSM, about 45-55 percent of participants had been on UI at some time during each year between 1992 and 1995. That proportion rose to about two-thirds during 1996 and about three-quarters by 1997. Non-participants started off about the same as participants, but had a much greater tendency to be on UI by 1995. In 1996, the two groups were again about equal. In 1997, non-participants were much less likely to have been on EI than participants. The proportions of participants on EI fell by 1998, but this is not necessarily due to EBSM, since the fall was just as dramatic for non-participants.
Differences by component are also apparent in Table 8.3. Employment Assistance Services clients were consistently the least likely, Job Creation Partnerships the most likely to use UI prior to participation. The most notable drop in EI use after the program was experienced by Self-Employment clients. This may reflect the entitlement rules for EI in the sense that self-employed individuals are generally not entitled EI. The percent of Job Creation Partnerships clients receiving EI in 1998 is high considering their program earnings were not insurable.
| EBSM Component | 1992 | 1993 | 1994 | 1995 | 1996 | 1997 | 1998 |
| Employment Assistance Services | 28.1% | 30.5% | 31.6% | 32.4% | 39.1% | 56.4% | 48.6% |
| Job Creation Partnerships | 49.9 | 54.0 | 60.0 | 67.6 | 68.0 | 56.8 | 52.8 |
| Purchase of training | 50.3 | 52.7 | 55.8 | 60.0 | 68.4 | 79.1 | 69.0 |
| Self-Employment | 43.1 | 47.0 | 53.0 | 57.8 | 70.7 | 52.6 | 19.8 |
| Targeted Wage Subsidies | 40.2 | 42.6 | 46.7 | 53.4 | 62.4 | 68.8 | 55.8 |
| EBSM | 46.4 | 48.9 | 52.2 | 56.5 | 64.3 | 72.8 | 62.9 |
| Non-participants | 50.6 | 52.9 | 60.8 | 69.7 | 65.8 | 53.6 | 42.5 |
About one-quarter of the participants and non-participants had been on UI for at least part of every year from 1992 to 1996 (Table 8.4). Only 5 percent of non-participants but 17 percent of participants were never on UI during these years.
Regarding participants, the average number of weeks in receipt of UI held fairly stable until it jumped in 1996 and again in 1997, before dropping in 1998. Although both groups (participants and non-participants) experienced the 1998 decline, the decline was steeper for participants.
| EBSM Component | 1992 | 1993 | 1994 | 1995 | 1996 | 1997 | 1998 |
| Employment Assistance Services | 7.7 | 8.5 | 8.8 | 7.9 | 9.4 | 14.7 | 9.4 |
| Job Creation Partnerships | 15.0 | 15.0 | 17.1 | 17.2 | 17.8 | 13.7 | 10.6 |
| Purchase of training | 13.2 | 13.5 | 13.6 | 12.8 | 15.6 | 20.0 | 13.7 |
| Self-Employment | 11.5 | 12.0 | 12.6 | 13.6 | 20.3 | 13.1 | 3.5 |
| Targeted Wage Subsidies | 10.7 | 11.3 | 12.3 | 12.6 | 15.9 | 14.5 | 10.3 |
| EBSM | 12.5 | 12.8 | 13.2 | 12.6 | 15.2 | 18.2 | 12.4 |
| Non-participants | 11.9 | 12.4 | 13.5 | 14.0 | 13.0 | 10.9 | 8.0 |
Mirroring the mean weeks on UI/EI are the mean benefits received by year (Chart 8.2). The two groups are similar before 1996, but diverge radically after that. The slope for participants is steeper than that for non-participants between 1997 and 1998.
As reported earlier, Employment Assistance Services clients were the least likely of all EBSM participants to use UI prior to participation. But, as the next table shows, they were consistently the most likely to have relied on social assistance before participating. In general, around 10 percent of EBSM clients had been on social assistance during each year from 1993 to 1997. About a fifth of the participants and 14 percent of non-participants had been on social assistance at some point during the five years previous to 1997.
At some time in 1998, 13 percent of participants but only 5 percent of non-participants were on social assistance.21 This is a continuation of the pre-program pattern of social assistance use, although the divergence between the groups grew after the program.
Limiting the analysis to those who completed EBSM in 1997, 11 percent of participants received social assistance in 1998, still twice as high as the proportion for non-participants.22 The econometric analysis will help determine if this is a consequence of participating in EBSM.
| EBSM Component | 1992 | 1993 | 1994 | 1995 | 1996 | 1997 | 1998 |
| Employment Assistance Services | 13.0% | 16.4% | 16.4% | 16.5% | 16.9% | 22.3% | 22.1% |
| Job Creation Partnerships | 7.7 | 13.8 | 10.7 | 14.4 | 16.3 | 17.3 | 16.3 |
| Purchase of training | 5.4 | 8.1 | 7.4 | 7.4 | 7.7 | 7.8 | 11.3 |
| Self-Employment | 5.1 | 9.2 | 8.0 | 7.7 | 8.7 | 6.4 | 6.9 |
| Targeted Wage Subsidies | 8.9 | 11.0 | 10.3 | 11.9 | 13.2 | 14.1 | 10.5 |
| EBSM | 6.8 | 9.8 | 9.0 | 9.5 | 10.0 | 10.8 | 12.6 |
| Non-participants | 4.7 | 6.2 | 5.7 | 6.2 | 6.4 | 5.9 | 5.4 |
Participants and non-participants were very different in terms of highest level of education completed (Chart 8.3). Non-participants tended more to the extremes than participants, having been more apt to drop out of school before finishing high school, but also more likely to complete university. The large difference in proportion completing college may be the result of Purchase of training, since most courses were held at community colleges (and 74 percent of participants who completed community college were Purchase of training clients).
Knowing that it would be hard to render a definitive verdict on the program's impact on education (because of the lack of pre-program data), the evaluators posed a further question to survey respondents to get a sense of whether the program might have influenced the propensity to upgrade their education. It turns out that 23 percent of EBSM participants had gone back to school, college or university since taking part in the program. About 8 percent were in a course at the time of the survey, and another 6 percent had successfully completed a course. Participants took a wide variety of courses including GED, computers, automotive repair, hospitality, and electronics.
By comparison, 24 percent of non-participants had taken further education or training since the start of 1997. When one considers that the question covered a two-year period for all non-participants but 59 weeks for the typical participant, it appears that participants were more inclined to upgrade their skills.
About 70 percent of the non-participants who had taken a course had paid for their own training; the remainder were enrolled in government training programs. Most of those paying for their own course (79 percent) had taken a skills training course at a community college or private training school. Others went back to school (12 percent) or university (12 percent).23
The typical non-participant course ran for 17.6 weeks. Thus nearly one-quarter of the comparison group experienced a significant training intervention in the last two years.
This section examines client outcomes in each EBSM component.
Employment Assistance Services include a range of services and Human Resources Development Canada (HRDC) programs to help clients prepare for, find and keep jobs. HRCCs or third party agencies contracted by Human Resources Centres of Canada (HRCCs) may deliver Employment Assistance Services. Of those Employment Assistance Services clients responding to the survey, 67 percent received services from an HRCC, 18 percent from a third party, and 15 percent from both.24 Third party agencies included employment resource centres (36 percent), social services offices (14 percent), and private training schools (8 percent).
Chart 8.4 displays the Employment Assistance Services received by clients. In general, third party agencies provided more services than did HRCCs. For example, HRCC clients were unlikely to get access to volunteer work, whereas nearly 21 percent of third party clients had such access. The only type of service that most HRCC clients received was a workshop on job search skills.
A key element of EBSM is the development of an action plan to get back to work. Often Employment Assistance Services clients did not develop such plans according to survey data. The previous graph shows that only 39 percent of HRCC clients and 62 percent of external service delivery clients received help to plan their strategy to get back to work. Fewer still developed an action plan: only 19 percent of HRCC Employment Assistance Services clients and 42 percent of third party clients did so. This conforms with the findings presented in Chapter Four: most low-need clients who use Employment Assistance Services do not develop an action plan.
Employment Assistance Services clients may also avail themselves of various services without the active assistance of HRCC or third party staff. Chart 8.5 reveals that 80 percent of HRCC clients and two-thirds of those going to third parties for Employment
Table 8.6 lists the mean grade given by Employment Assistance Services clients to the services they received. Most marks are above average: overall, HRCC clients gave a Band third party clients gave a B to Employment Assistance Services (not a significant difference).
| Service | HRCC Clients | Third Party Clients |
| Job Finding Clubs | B | B - |
| Workshop on Job Search Skills | B | B |
| Volunteer Work | B | B - |
| Referral to a Job, training Program or Other Services | B | B - |
| Help from an Employment Officer | B | B |
| Job Bank Kiosks/Job Board Listings | B - | C + |
| Labour Market Information from computer | B - | B - |
| Computerized Job Search Tools | B | B |
| Internet Job Listings | B | B |
| Information on training and Education Programs | B - | B - |
| Written materials on Labour Market or Employment Services | B | B |
| Overall EAS | B - | B |
The Purchase of training component provides training courses to clients to improve job skills. Almost 85 percent of Purchase of training clients surveyed asserted they could not have taken the course without the financial support of the program.
Chart 8.6 presents the occupation for which Purchase of training clients were training. Sciences, engineering and health, as well as construction were the most popular fields.
Asked how closely related their current job is to the job training they received, 58 percent of Purchase of training participants said very closely related, 17 percent said somewhat related and 26 percent said not at all related.
The typical course lasted for 20.2 weeks, but the length ranged from 1 day to over 100 weeks. The median course length was 9 weeks. Most of the courses were full-time: 77 percent ran for 30 or more hours per week. On average the courses were 35 hours per week. Multiplying the number of weeks by the number of hours per week produces the mean course length: 707 hours. Some 44 percent of the courses included a work placement component.
Approximately 43 percent of Purchase of training respondents were determined to be able to contribute to the cost of their training and 40 percent actually did so. The average course contribution was $1,171; but this figure is skewed by nine clients who paid $4,000 or more for their course. The median contribution was $400.
Only 11 percent of the trainees had not finished their course by the time of the survey. The main reason given ( by about 40 percent) was that they were still in the course while the remainder had not finished the course for various other reasons.
Purchase of training clients seemed impressed with nearly every aspect of the training they received. Overall, they gave the course a B+. In fact, most every aspect was graded B+. Only the monthly income available was marked lower.
| Service | HRCC Clients |
| training Course Overall | B + |
| training Facilities | B + |
| Instructors | B + |
| Relevance to your Career Plans | B + |
| Amount Learned | B + |
| Monthly Income provided | B - |
| N = 278 | |
Chart 8.7 shows that client placements by the type of organization mirrors the objectives of each of these work experience programs. Over 80 percent of Targeted Wage Subsidies placements were with private sector employers, whereas 44 percent of Job Creation Partnerships placements were with non-profit community agencies.
Types of work done for placement employers also differed widely between Targeted Wage Subsidies and Job Creation Partnerships clients (Chart 8.8). Targeted Wage Subsidies jobs were most likely to be in the service, sales or clerical fields. Job Creation Partnerships jobs were chiefly in construction, services or administration. This conforms to the type of projects funded through Job Creation Partnerships.
The typical client worked 39 hours per week on the placement job. There was no difference between Targeted Wage Subsidies and Job Creation Partnerships. Two-thirds of the work experience clients worked 40 hour weeks. Only 4 percent worked for less than 30 hours weekly.
The vast majority of participants in these two components stayed on the job for the entire period of the placement: 84 percent for Targeted Wage Subsidies and 86 percent for Job Creation Partnerships. The most often given reason for leaving before the placement ended was finding a job with another employer (45 percent); other reasons included fired or laid off (11 percent); disagreement with employer (9 percent); moved (5 percent); and disliked the job (5 percent).
In many cases, the wage subsidy worked as intended.26 Targeted Wage Subsidies clients had a much higher probability of receiving such an offer, however: 62 percent of Targeted Wage Subsidies participants versus 38 percent of Job Creation Partnerships participants were offered a job after the sudbidized placement ended.27 This variance is to be expected given the different nature of the two components, although both are supposed to subsidize "sustainable" jobs. Reasons respondents offered for not being offered a permanent job by the subsidized employer differed between the components: over two-thirds of the Job Creation Partnerships clients, but only 38 percent of the Targeted Wage Subsidies clients said there were no available positions or no money to hire. Other reasons given by Targeted Wage Subsidies clients for not being offered a permanent job included company closed (17 percent); seasonal employment (10 percent); not interested in working for the employer (9 percent); and don't know (12 percent). Other reasons given by Job Creation Partnerships clients included company closed (6 percent); seasonal employment (6 percent); and not interested in working for the employer (5 percent).
Unfortunately, in many cases, the job did not turn out to be permanent. Of those who were retained by their employer after the subsidy, 58 percent were no longer at the same job at the time of the survey. Job Creation Partnerships clients, less likely than Targeted Wage Subsidies clients to be kept on in the first place, were also more apt to have left or lost that job: only 34 percent hired after the subsidy expired were still working with the employer in early 1999. Just under half the Targeted Wage Subsidies clients (48 percent) were still with the placement employer at the time of the survey.28 One-third of work placement clients were on temporary lay-off, so it is probable the picture would be better in the summer. Also, about a quarter of those who left the job went to what they perceived to be a better work opportunity. In the end, then, only 19 percent of all Job Creation Partnerships clients and 35 percent of all Targeted Wage Subsidies clients were still with the placement employer in January 1999.
On average, Job Creation Partnerships clients retained after the subsidy but who subsequently left, were employed for 13 unsubsidized weeks. The corresponding figure for Targeted Wage Subsidies clients was 23 weeks.29
Table 8.8 presents the satisfaction ratings of Targeted Wage Subsidies and Job Creation Partnerships participants with various aspects of their intervention. For the most part, these participants rated their work experience good to very good. Guidance on services available after the placement was rated somewhat lower. There were no significant differences in satisfaction between the two groups.
| Service | Targeted Wage Subsidies | Job Creation Partnerships |
| Help provided by Employment Officer before your placement | B | B + |
| Help provided by Employment Officer during your placement | B | B |
| Suitability of your placement to your job skills | B + | B + |
| Suitability of your placement to your career interests | B | B |
| Direction and Supervision provided by your employer | B | B |
| Guidance on services available after your placement | C + | C + |
| Level of financial support while on placement | B - | B - |
| N varies: around 50-60 for JCP and 70-85 for TWS | ||
The objective of Self-Employment is to assist clients with viable business ideas to get their business up and running. Most Self-Employment clients (94 percent) were interested in starting a business before becoming involved in the SE component. Only percent had sought assistance from another source in an attempt to start their own business, though. One in five had been self-employed in the past.
Apparently most had a reasonable basis for starting the business: 77 percent had previous experience or training in the field of interest. About two-thirds had taken a training course or workshop to help them start a business.
About 95 percent of Self-Employment clients made an initial capital investment in their new business. The median contribution was $9,000.
Almost every Self-Employment participant (97 percent) started a business through the program. At the time of the survey, 68 percent were still employed in the business they developed (Chart 8.9), which is similar to the proportion reported by business development agency representatives interviewed. The typical business had been in operation for about a year and a half. Interestingly, the amount of the initial investment had no bearing on whether the business survived.
However these new businesses seem to be struggling judging by their mean monthly profit of $1,390 and the fact that 24 percent reported no profit in a typical month. The highest monthly profit was $10,000. Median monthly profit was only $600.
Seven in ten of the new businesses employed only the Self-Employment participant when the survey took place. One business had six full-time employees and two others had five full-time employees besides the owner. In total, Self-Employment businesses had generated 91 jobs — 51 of them full-time positions — besides providing employment for the entrepreneurs. This will understate the true job creation power of Self-Employment since the survey only reached 54 percent of Self-Employment clients. A good guess is the other 46 percent were not as successful since many had moved or were otherwise unreachable. Still it is likely that Self-Employment has produced over 100 direct jobs besides the owners' positions.
Sixty percent attributed their success in starting their business at least in part to Self-Employment while 40 percent thought they would have been successful in establishing their own business without the help of Self-Employment. Those who presumed they would have been successful without Self-Employment tended to operate more successful enterprises: all were still in business at the time of the survey, and they created most of the jobs.
About half of those who believed they could have started the business without Self-Employment said they were able to start it sooner because of Self-Employment.
Asked what the program did for them that they could not have done for themselves, Self-Employment clients most often mentioned the financial support they received from EI while they attempted to get the business going (Chart 8.10). Skills and knowledge to develop a business plan was the most frequently stated active intervention of Self-Employment.
The 32 percent of SE participants whose business had not been successful were asked how long their business lasted, why it failed and what they would do differently if they had it to do again. On average, the failed businesses lasted for 10 months, with a range of 2 to 19 months. Almost no one was able or willing to answer why the business was no longer operating. They were more forthcoming about what they would do differently if they had the chance to start again. About 14 percent reported they would change locations; another 14 percent would save more money before starting; 11 percent would do more research or better planning; 8 percent would take more business courses; 8 percent wouldn't start again; 14 percent didn't know; and the rest gave other lessons.
A priori, one would predict that those whose businesses succeeded would be more approving of the program than would those whose businesses foundered. That is most certainly the case as Table 8.9 demonstrates. Overall, the ratings for Self-Employment were the lowest of the EBSM components.
| Service | Business still operating | Business not operating | t -test |
| Information provided about the program | B | B - | p > 0.40 |
| Self-Employment business skills it taught | B - | C + | p < 0.05 |
| Help provided by a role model (mentor) | B - | C - | p < 0.05 |
| Size of loan | C + | C - | p > 0.20 |
| Availability of income assistance | B | B - | p > 0.20 |
Because the performance measures focus on the period immediately after completing the intervention, it is important to determine what clients did upon completion. Table 8.10 lists the responses of survey respondents. For the program as a whole, 65 percent were employed immediately after their participation, either because they continued working with their placement employer, continued self-employment, or found a job with- another employer. Another 25 percent began searching for work.
There were large differences across EBSMs, most of which are predictable given the nature of each EBSM component. Over three-quarters of Self-Employment participants were self-employed after participation. About 58 percent of Targeted Wage Subsidies and 42 percent of Job Creation Partnerships clients continued working for their placement employer. Over 43 percent of the Employment Assistance Services clients continued their job search. One finding was unexpected: 34 percent of Purchase of training clients continued working for their placement employer (these were apprentices). Earlier it was reported that 44 percent of the Purchase of training courses included a work placement component: with a few exceptions, these clients were apprentices. Apparently, many of these clients impressed their placement employer enough to gain a permanent job there.
| Activity | EAS | JCP | POT | SEA | TWS | EBSM |
| Continued employment with placement employer | 0.0% | 41.8% | 33.9% | 0.0% | 57.5% | 30.0% |
| Started working for another employer | 43.3 | 19.4 | 29.3 | 13.0 | 13.6 | 25.0 |
| Continue/start self-employment | 1.4 | 2.5 | 2.5 | 78.3 | 0.8 | 10.4 |
| Continued education | 3.4 | 1.5 | 2.5 | 1.7 | 0.9 | 2.0 |
| Took a job training program | 1.9 | 0.0 | 1.4 | 0.0 | 0.9 | 1.0 |
| Stayed at home with children | 1.0 | 1.0 | 0.0 | 0.9 | 0.9 | 0.7 |
| Stayed at home for other reasons | 2.4 | 1.0 | 1.4 | 0.9 | 2.6 | 1.7 |
| Started looking for a job | 43.3 | 25.0 | 25.1 | 3.5 | 19.7 | 25.1 |
| Other | 3.4 | 7.7 | 3.9 | 1.8 | 3.0 | 4.1 |
| x 2 = 90.6 df=44, p<0.001 | ||||||
Those who were not working or looking for work immediately after the program were asked if they had searched for a job since then. Almost 70 percent said yes. This ranged from 55 percent of Purchase of training clients to 85 percent of Job Creation Partnerships clients.
Of those who ended the program looking for work or who subsequently began looking for work, 68 percent had found paid employment by the time the survey took place. Factoring in those who found jobs immediately after EBSM, 11 percent of all EBSM participants who tried to find a job since finishing their intervention had thus far failed to get one.
As for non-participants, 68 percent got a job when their EI benefits ceased (in most cases it is likely that EI benefits ended because they found a job). About 14 percent continued looking for a job, and most of the rest withdrew from the labour market. Of this last group, 43 percent eventually began a job search. Close to two-thirds of those who looked for a job since EI ended had found one by the time of the survey. That means about 11 percent who wanted a job hadn't found one by the time of the survey — the same proportion as participants.
| Activity | All Non-participants |
| Started working | 67.4% |
| Self-employment | 1.0 |
| Continued education | 1.2 |
| Took a job training program | 0.0 |
| Stayed at home with children | 2.6 |
| Stayed at home for other reasons | 3.0 |
| Started looking for a job | 14.0 |
| Other | 10.8 |
| N = 157 | |
For those who did not have a job upon completing their EBSM intervention, but had found one subsequently, it took an average of 11 weeks to find that job. It took non-participants 12 weeks to find a job after their benefits ended (not a significant difference).
In total, 84 percent of participants and 71 percent of non-participants had been employed since completing their intervention/benefits.30 Participants had worked at more jobs post-program than had non-participants (Chart 8.11). On average, participants have had 1.2 jobs, non-participants 0.9 jobs since EBSM/EI.31
Of participants who left their first post-program job, 53 percent had secured another; the comparable figure for non-participants was 48 percent. About 39 percent of participants and 37 percent of non-participants who left their first post-program job immediately began working at a second (Chart 8.12). Most of the rest began looking for a new job. Most participants (65 percent) and non-participants (69 percent) in this position hadn't found a second job by the time of the survey: about four in ten of those who hadn't found a second job hadn't looked for one. Of participants who left their second post-program job, 59 percent found a third.
An important objective of the survey was to validate primary employment results indicators — how many EI clients have returned to work and the resulting unpaid EI benefits. Only 24 percent of EBSM clients were coded as employed according to the HRIB results data from the same period as the evaluation covered. The findings from the survey indicate this figure is far too low. About two-thirds of EBSM participants were employed immediately after their intervention ended.
The employment indicator is defined as EI claimants who become employed before the end of their benefit period and is calculated by looking at the EBSM clients who had 12 consecutive weeks of reduced benefits defined as 25 percent or less of entitlement. Survey results show that about seven in ten participants (68 percent) were employed for at least 12 consecutive weeks following participation in EBSM.32 The first job after EBSM accounted for almost all of this; 6 percent reached the 12 week plateau on their second post-program job. Since virtually all this occurred within one year of leaving EBSM, 67 percent of EBSM clients were employed for 12 consecutive weeks within one year of participating in EBSM. Chart 8.13 presents the proportion of participants with at least 12 consecutive weeks of employment by EBSM component. Self-Employment and Targeted Wage Subsidies clients were most successful in this regard.
On average, EBSM participants had completed participation in EBSM 58.9 weeks prior to the survey and had worked 43.1 weeks during that time. Employment Assistance Services, Job Creation Partnerships and Purchase of training clients averaged around 40 weeks of post-program employment; Targeted Wage Subsidies averaged 51 weeks, Self-Employment 45 weeks.33 The jobs were 40-hour positions for the most part.
A crucial test of success is what participants are now doing as compared to those who did not participate. Are participants more likely to be working than non-participants? Are they less likely to be on social assistance? Are they less likely to be on EI? Are Self-Employment clients still operating their own businesses? Does participation lead to a greater tendency to upgrade one's education or job training skills? Chart 8.14 provides preliminary answers to these questions. The next chapter will control for other factors that may have lead to these results for a more definitive picture.
Before correcting for possible outside influences, it appears that participation in EBSM led to a significantly higher probability of upgrading one's education and training.34 On the other hand, current labour market status is not so positive: EBSM participants were somewhat less likely than non-participants to be employed35; more likely to be unemployed and looking for work36; and more apt to be on EI.37
As Table 8.12 shows, there is preliminary evidence that Self-Employment was more effective than the other options at removing people from EI and at getting them into a paid job. About 68 percent of Self-Employment participants were self-employed at the time of the survey. Purchase of training clients were at the other end of the spectrum, being least likely to have a job and most prone to rely on EI; Purchase of training clients were also most likely to be in training or upgrading courses.
| Activity | EAS | JCP | POT | SEA | TWS | Sig. * |
| Working in a paid job | 60.3% | 59.2% | 58.5% | 10.3% | 69.3% | p<0.001 |
| Self-employed | 0.0 | 0.5 | 2.1 | 68.1 | 0.0 | p<0.001 |
| Looking for a job | 28.7 | 37.2 | 31.3 | 10.3 | 24.6 | p<0.001 |
| Upgrading education | 12.0 | 7.7 | 14.8 | 4.3 | 5.3 | p<0.01 |
| In a job training program | 2.9 | 2.0 | 8.1 | 0.9 | 2.6 | p<0.01 |
| On social assistance | 6.7 | 7.1 | 1.8 | 1.7 | 4.4 | p<0.02 |
| On EI | 14.8 | 23.5 | 32.4 | 4.3 | 18.4 | p<0.001 |
| N | 209 | 196 | 284 | 116 | 228 | — |
| x2 test of significance (df=4) between options. | ||||||
The reason participants offered most for being unemployed was that there were no jobs available (Chart 8.15).
|
This chapter presents the econometric analysis of the impacts of Employment Benefits and Support Measures in Nova Scotia. Because there are several outcomes of interest, the chapter proceeds in stages, examining outcome measures in the same sequence as in the previous chapter.
Note that the measures of impacts presented here should be considered preliminary only. More definitive measures will be presented in a summative evaluation when there will be more time following Employment Benefits and Support Measures (EBSM) participation to detect impacts.
This section analyzes the relationship between participating in EBSM and the time spent in three main activities: work, in school, and unemployment. The survey of participants and non-participants asked about time (in months) devoted during each of the years 1995 to 1998 to the following labour market activities: In School and Not Working, In School and Working, Working and Not in School, and Not Working and Not in School. Because only a small amount of time is devoted to "In School and Working," these four activities were combined into three: Working, In School, and Unemployed (Not in School/Not Working). None of the findings is altered by treating the four activities separately.
The focus of the analysis is the time devoted to these activities in 1998. Because some of the participants were still in EBSM during 1998, the analysis in this section is restricted to participants who completed or dropped out of the program during 1997. If this was not done, it would not be possible to separate the effects on time spent on various activities that are associated with participating in EBSM from the impacts of EBSM on these activities. The comparison group consists of regular Employment Insurance (EI) recipients who did not participate in EBSM during the sample period.38 The sample analyzed consists of 559 participants and 499 non-participants.
For each of the three principal activities three types of estimates of program impacts are obtained. The first are simple linear regression models with and without controls for observable demographic and individual characteristics (referred to in the tables as "demographic controls"). Explanatory variables used in these regressions are as follows: age, educational attainment, gender, marital status, single parent status, number of children under 6 years of age, number of children 6 to 18 years of age, visible minority status, limitation/disabled status, aboriginal status, and language (English, French, Other). Age is allowed to have a non-linear effect on the outcome of interest, as most research finds that the effect of Age on various outcomes displays diminishing returns.
These regressions compare participants and non-participants in the post-program year 1998. Although they control for the influence of observable factors that may differ between participants and non-participants, they do not account for possible non-observable factors that may influence selection into the program.
The other estimates reported utilize the longitudinal nature of the data on time spent in labour force activities in order to account for certain types of non-random selection into the program. In the presence of these types of non-random selection the simple linear regression models discussed above would yield biased estimates of program impact. Two types of longitudinal estimators are employed; both take advantage of pre-program information obtained in the survey on time spent in each activity in 1995 and 1996 as well as 1998. "Difference-in-differences" estimators provide unbiased estimates of program impact if selection into the program is based on unobserved person-specific and time-invariant factors that also influence the time spent on these activities by individuals. Linear regression estimates that control for pre-program levels of the activities in question are also reported. These longitudinal estimators are sometimes referred to as "unrestricted difference-in-differences" estimators because they do not restrict the coefficient on the pre-program level to be unity, as is the case with difference-indifferences estimators.
Table 9.1.1 reports the estimated program impacts on the three activities based on these alternative specifications. Estimates from a variety of specifications are reported in order to determine whether the estimated impacts are sensitive to alternative assumptions about the nature of selection into the program. All estimates use only the subset of the sample of participants who completed EBSM in 1997.
In 1995 and 1996 (i.e., prior to EBSM), those who became EBSM participants spent less time working, more time in school and more time unemployed than did those who did not participate in the program. These differences between participants and non-participants were approximately -1.1 to -1.4 months working per year, +0.6 to +0.7 months in school each year, and +0.4 to +0.7 months unemployed each year. EBSM clearly attracted or selected individuals who were more likely to be in school or unemployed during the year and less likely to be working than those who did not apply for or who were not selected into the program. These systematic differences in pre-program activities suggest the presence of non-random selection into the participant and non-participant groups.
The differences between participants and non-participants are much smaller after the completion of the program. In 1998, participants continued to spend less time working than non-participants, but the gap between the two groups had narrowed to -0.1 months, i.e., had almost disappeared. Similarly the gap in time spent in school narrowed to +0.2 months and the gap in time spent unemployed was actually reversed in sign to -0.2 months. In the post-program year 1998, only the difference in time spent in school is statistically significant (see equation 9.1.1). This narrowing of differences between participants and non-participants is principally due to participants increasing their time spent working relative to pre-program levels and decreasing time spent in school and unemployed relative to pre-program levels of these activities. These results suggest that the program may have had a positive impact on time spent working and negative impacts on time spent in school and unemployed.
| Equation | Working | In School | Unemployed | Model Specification |
| 9.1.1 | -0.1 | 0.2** | -0.2 | Linear regression, no controls |
| (0.3) | (0.1) | (0.2) | ||
| 9.1.2 | -0.4* | 0.2 | 0.3 | Linear regression, demographic controls |
| (0.3) | (0.1) | (0.2) | ||
| 9.1.3 | 1.0*** | -0.2 | -0.7*** | Difference-in-differences 1996 vs. 1998 |
| (0.3) | (0.1) | (0.3) | ||
| 9.1.4 | 0.7** | -0.2 | -0.5* | Difference-in-differences with demographic controls 1996 vs. 1998 |
| (0.2) | (0.2) | (0.3) | ||
| 9.1.5 | 0.7** | -0.3 | -0.4 | Difference-in-differences 1995 vs. 1998 |
| (0.3) | (0.2) | (0.3) | ||
| 9.1.6 | 0.4 | -0.3 | -0.1 | Difference-in-differences with demographic controls 1995 vs. 1998 |
| (0.3) | (0.2) | (0.3) | ||
| 9.1.7 | 0.3 | 0.1 | -0.4* | Linear regression, no demographic controls, 1996 activity controls |
| (0.2) | (0.1) | (0.2) | ||
| 9.1.8 | -0.0 | 0.1 | 0.1 | Linear regression, demographic controls, 1996 activity controls |
| (0.2) | (0.1) | (0.2) | ||
| 9.1.9 | 0.4 | 0.1 | -0.4* | Linear regression, no demographic controls, 1995 and 1996 activity controls |
| (0.2) | (0.1) | (0.2) | ||
| 9.1.10 | -0.0 | 0.1 | 0.1 | Linear regression, demographic controls, 1995 and 1996 activity controls |
| (0.2) | (0.1) | (0.2) |
The first two sets of estimates reported in Table 9.1.1 (equations 9.1.1 and 9.1.2) compare participants and non-participants in the post-program period, 1998. These indicate that participants spent less time working and more time in school than non-participants and, after controlling for demographic characteristics, more time unemployed than non-participants. None of the differences between the two groups is large, and only the difference in time spent working is statistically significant. However, these estimates do not take into account the clear evidence of non-random selection into the program. For this reason the longitudinal estimates are preferred (equations 9.1.3 to 9.1.10).
For the activity "In School," all the longitudinal estimates yield the same conclusion —that EBSM did not have a significant impact on the time spent in this activity. The estimated impacts are all small in magnitude and none is significantly different from zero. The evidence for the activities "Working" and "Unemployed" is more mixed, with some estimates indicating a positive impact on time spent working and a negative impact on time unemployed and other estimates suggesting no impact. In turn the evidence relating to each of these activities is discussed.
The difference-in-differences estimates (without demographic controls) suggest that the program may have had a positive impact on time spent working — estimates of +1.0 months using 1996 as the base year and +0.7 months using 1995 as the base year. However, these estimates become smaller once demographic controls are included and the estimate for the 1995 base year is no longer significantly different from zero. Because the specification with demographic controls is more general, these estimates are preferable to the simple difference-in-differences estimates without controls. These two estimates indicate that the program increased time spent working by 0.4 months using 1995 as the base year to 0.7 months using 1996 as the base year. This is a fairly narrow range, and the two estimates are not significantly different from each other in a statistical sense.40 This is an important finding because the assumptions under which the difference-in-differences methodology provides unbiased estimates of program impact imply that the estimated impacts should not be sensitive to the choice of base year. The estimates reported in equations 9.1.4 and 9.1.6 pass this specification test. Thus the difference-in-differences estimates suggest that the program may have had a small positive impact on time spent working (0.4 to 0.7 months per year); however, the 95 percent confidence interval on the lower estimate of 0.4 months includes zero, so this evidence is also consistent with the view that the program had no impact.
Somewhat in contrast, the regressions which control for pre-program activity levels (equations 9.1.8 to 9.1.10) indicate that the program did not have a statistically significant impact on time spent working. Choosing between these longitudinal estimates and the differences-in-differences estimates is not straightforward as each is based on a different set of assumptions about the nature of selection into the program. Given the clear evidence from the pre-program data of non-random selection into EBSM, the difference-in-differences estimates are preferred when, as is the case here, these pass the specification test of being invariant with respect to the choice of base year.
Turning to the final activity, there is weak evidence from the difference-in-differences estimates that the program may have had a small negative impact on time spent unemployed, with the estimates ranging from —0.5 months for 1996 as the base year to -0.1 months for 1995 as the base year. However, the former estimate is only marginally significant (10 percent level) and the latter is not significantly different from zero. Furthermore the estimates that control for pre-program activity levels as well as demographic factors are also not significantly different from zero. As a consequence, the evidence from the various longitudinal estimates suggests that EBSM did not significantly affect time spent unemployed.
In summary, EBSM does not appear to have had any impact on the time spent in school. With respect to time spent working and unemployed the evidence is more mixed. The differences between participants and non-participants in the time devoted to these activities narrowed following the program, and in the desired directions: toward more time working and less time unemployed. However, these changes were not large enough for EBSM to have a statistically significant impact on time spent unemployed, at least in the short term (first year following the program). The evidence relating to work activity is somewhat more positive, with the preferred estimates being in the 0.4 to 0.7 months range; nonetheless it cannot be claimed with confidence that this impact is significantly different from zero.
Tables 9.1.2 to 9.1.6 report the results of the analysis by program component. The sample of 499 non-participants is used throughout, and the sample of 559 participants is examined separately for each of the five program components: Self-Employment (Table 9.1.2), Job Creation Partnerships (Table 9.1.3), Targeted Wage Subsidies (Table 9.1.4), Employment Assistance Services (Table 9.1.5), and Purchase of training (Table 9.1.6).
For Self-Employment, each of the estimated impacts on time spent working is positive but none is statistically significant. The longitudinal estimates range from +1.1 to +2.1 months. Similarly, the estimated impacts on time spent unemployed are consistently negative, but also generally not statistically significant. The longitudinal estimates range from -1.2 to -2.8 months. The longitudinal estimates of the impact on time spent in school are consistently positive, small in magnitude (0.1 to 0.5 months) and not statistically significant. Overall these results suggest that Self-Employment may have had a positive impact on time spent working and a similarly desirable negative impact on time spent unemployed, but larger sample sizes would be needed to obtain more precise estimates which are significantly different from zero.
The results for Job Creation Partnerships are similar to those for EBSM as a whole. When participants and non-participants are compared in the post-program period alone (and after controlling for demographic factors), participants spent significantly less time working and significantly more time unemployed than non-participants. However, the longitudinal estimates indicate that these post-program differences should not be attributed to the program. Indeed, the simple difference-in-differences estimates suggest a positive impact of EBSM on time spent working. However, once demographic factors are taken into account the evidence is more mixed, with one estimate of +1.4 months (1996 base year) being significantly greater than zero and the other estimate of +1.1 months (1995 base year) not being significantly different from zero. Note that these estimates of +1.1 to +1.4 months do not differ significantly from each other, and thus pass the specification test discussed previously. This result increases confidence in the difference-in-differences estimates; nonetheless, because the 1995 base year estimate is not significantly different from zero it cannot be concluded that there is unambiguous evidence that the program increased time spent working. Turning to unemployment, although most of the longitudinal estimates are negative, none is statistically significant.
Thus there is no evidence that the program reduced the duration of unemployment. Finally, the estimated impacts on time in school are consistently small in magnitude and none are significantly different from zero so there is clear evidence that the program did not impact on this activity.
The results for the Targeted Wage Subsidies component provide the strongest evidence of positive program impacts. In particular, after controlling for demographic and individual characteristics, the difference-in-differences estimates indicate a positive impact of 1.2 to 2 months, depending on the base year. Each of these estimates is significantly greater than zero; they also pass the specification test of not being significantly different (at the 5 percent level) from each other for alternative choices of the base year. However, the longitudinal estimates which include pre-program activity levels as controls result in a different conclusion — that the program had a small positive (+0.6 months) but statistically insignificant impact on time spent working. It is not possible with the available data to determine which of these two sets of estimates is more credible. Because the difference-in-differences estimates pass the "base year" specification test these estimates are preferred. Thus the Targeted Wage Subsidies program component appears to have had a positive impact of approximately 1.2 to 2.0 months on time spent working during 1998, but this conclusion is not supported by all the evidence. The evidence relating to time spent unemployed is weaker, with all the longitudinal estimates being negative but most not being significantly different from zero. There is also some evidence that Targeted Wage Subsidies reduced time spent in school, but this again is mixed.
The results for the Employment Assistance Services and Purchase of training components are shown in Tables 9.1.5 and 9.1.6 respectively. Both of these components appear to have had no significant impact on time spent in the three labour market activities.
| Equation | Working | In School | Unemployed | Model Specification |
| 9.1.1 | 1.2 | 0.0 | -0.9 | Linear regression, no controls |
| (1.4) | (0.5) | (1.4) | ||
| 9.1.2 | 1.1 | 0.2 | -0.9 | Linear regression, demographic controls |
| (1.4) | (0.5) | (1.3) | ||
| 9.1.3 | 2.1 | 0.5 | -2.8* | Difference-in-differences 1996 vs. 1998 |
| (1.5) | (0.7) | (1.4) | ||
| 9.1.4 | 1.9 | 0.4 | -2.5 | Difference-in-differences with demographic controls 1996 vs. 1998 |
| (1.5) | (0.7) | (1.4) | ||
| 9.1.5 | 1.1 | 0.2 | -1.2 | Difference-in-differences 1995 vs. 1998 |
| (1.6) | (0.8) | (1.5) | ||
| 9.1.6 | 1.3 | 0.1 | -1.3 | Difference-in-differences with demographic controls 1995 vs. 1998 |
| (1.6) | (0.8) | (1.5) | ||
| 9.1.7 | 1.6 | 0.2 | -1.8 | Linear regression, no demographic controls, 1996 activity controls |
| (1.3) | (0.5) | (1.3) | ||
| 9.1.8 | 1.4 | 0.2 | -1.5 | Linear regression, demographic controls, 1996 activity controls |
| (1.2) | (0.5) | (1.2) | ||
| 9.1.9 | 1.5 | 0.2 | -1.7 | Linear regression, no demographic controls, 1995 and 1996 activity controls |
| (1.3) | (0.5) | (1.3) | ||
| 9.1.10 | 1.3 | 0.2 | -1.4 | |
| (1.2) | (0.5) | (1.2) |
| Equation | Working | In School | Unemployed | Model Specification |
| 9.1.1 | -0.4 | 0.1 | 0.3 | Linear regression, no controls |
| (0.6) | (0.2) | (0.6) | ||
| 9.1.2 | -1.0* | 0.0 | 0.9* | Linear regression, demographic controls |
| (0.6) | (0.2) | (0.6) | ||
| 9.1.3 | 1.6** | -0.4 | -0.8 | Difference-in-differences 1996 vs. 1998 |
| (0.6) | (0.3) | (0.6) | ||
| 9.1.4 | 1.4** | -0.4 | -0.5 | Difference-in-differences with demographic controls 1996 vs. 1998 |
| (0.6) | (0.3) | (0.6) | ||
| 9.1.5 | 1.7** | -0.3 | -0.8 | Difference-in-differences 1995 vs. 1998 |
| (0.7) | (0.3) | (0.6) | ||
| 9.1.6 | 1.1 | -0.4 | -0.3 | Difference-in-differences with demographic controls 1995 vs. 1998 |
| (0.7) | (0.3) | (0.6) | ||
| 9.1.7 | 0.5 | -0.0 | -0.2 | Linear regression, no demographic controls, 1996 activity controls |
| (0.6) | (0.2) | (0.5) | ||
| 9.1.8 | -0.1 | -0.1 | 0.4 | Linear regression, demographic controls, 1996 activity controls |
| (0.5) | (0.2) | (0.5) | ||
| 9.1.9 | 0.6 | -0.1 | -0.2 | Linear regression, no demographic controls, 1995 and 1996 activity controls |
| (0.6) | (0.2) | (0.5) | ||
| 9.1.10 | 0.0 | -0.1 | 0.3 | |
| (0.5) | (0.2) | (0.5) |
| Equation | Working | In School | Unemployed | Model Specification |
| 9.1.1 | 0.5 | 0.5** | -0.5 | Linear regression, no controls |
| (0.5) | (0.2) | (0.5) | ||
| 9.1.2 | -0.1 | 0.4* | 0.2 | Linear regression, demographic controls |
| (0.5) | (0.2) | (0.5) | ||
| 9.1.3 | 2.4*** | -0.8*** | -1.2** | Difference-in-differences 1996 vs. 1998 |
| (0.5) | (0.3) | (0.5) | ||
| 9.1.4 | 2.0*** | -0.7** | -0.9* | Difference-in-differences with demographic controls 1996 vs. 1998 |
| (0.6) | (0.3) | (0.5) | ||
| 9.1.5 | 1.8*** | -0.7** | -0.8 | Difference-in-differences 1995 vs. 1998 |
| (0.6) | (0.3) | (0.5) | ||
| 9.1.6 | 1.2** | -0.6* | -0.3 | Difference-in-differences with demographic controls 1995 vs. 1998 |
| (0.6) | (0.3) | (0.5) | ||
| 9.1.7 | 1.3*** | 0.2 | -0.8* | Linear regression, no demographic controls, 1996 activity controls |
| (0.5) | (0.2) | (0.4) | ||
| 9.1.8 | 0.6 | 0.1 | -0.1 | Linear regression, demographic controls, 1996 activity controls |
| (0.5) | (0.2) | (0.4) | ||
| 9.1.9 | 1.3*** | 0.1 | -0.7* | Linear regression, no demographic controls, 1995 and 1996 activity controls |
| (0.5) | (0.2) | (0.4) | ||
| 9.1.10 | 0.6 | 0.1 | -0.1 | |
| (0.4) | (0.2) | (0.4) |
| Equation | Working | In School | Unemployed | Model Specification |
| 9.1.1 | -0.3 | 0.4 | -0.1 | Linear regression, no controls |
| (0.6) | (0.2) | (0.6) | ||
| 9.1.2 | -0.4 | 0.2 | 0.3 | Linear regression, demographic controls |
| (0.6) | (0.2) | (0.5) | ||
| 9.1.3 | 0.4 | 0.2 | -0.5 | Difference-in-differences 1996 vs. 1998 |
| (0.6) | (0.3) | (0.6) | ||
| 9.1.4 | 0.4 | 0.1 | -0.2 | Difference-in-differences with demographic controls 1996 vs. 1998 |
| (0.6) | (0.3) | (0.6) | ||
| 9.1.5 | 0.2 | -0.4 | 0.2 | Difference-in-differences 1995 vs. 1998 |
| (0.7) | (0.4) | (0.6) | ||
| 9.1.6 | 0.1 | -0.5 | 0.4 | Difference-in-differences with demographic controls 1995 vs. 1998 |
| (0.7) | (0.4) | (0.6) | ||
| 9.1.7 | 0.0 | 0.3 | -0.3 | Linear regression, no demographic controls, 1996 activity controls |
| (0.5) | (0.2) | (0.5) | ||
| 9.1.8 | 0.1 | 0.2 | 0.1 | Linear regression, demographic controls, 1996 activity controls |
| (0.5) | (0.2) | (0.5) | ||
| 9.1.9 | 0.0 | 0.3 | -0.2 | Linear regression, no demographic controls, 1995 and 1996 activity controls |
| (0.5) | (0.2) | (0.5) | ||
| 9.1.10 | -0.1 | 0.2 | 0.2 | |
| (0.5) | (0.2) | (0.5) |
| Equation | Working | In School | Unemployed | Model Specification |
| 9.1.1 | -0.2 | 0.2 | -0.2 | Linear regression, no controls |
| (0.4) | (0.1) | (0.3) | ||
| 9.1.2 | -0.5 | 0.1 | 0.3 | Linear regression, demographic controls |
| (0.4) | (0.1) | (0.3) | ||
| 9.1.3 | 0.6 | -0.1 | -0.6* | Difference-in-differences 1996 vs. 1998 |
| (0.4) | (0.2) | (0.3) | ||
| 9.1.4 | 0.2 | -0.1 | -0.3 | Difference-in-differences with demographic controls 1996 vs. 1998 |
| (0.4) | (0.2) | (0.4) | ||
| 9.1.5 | 0.3 | -0.2 | -0.3 | Difference-in-differences 1995 vs. 1998 |
| (0.4) | (0.2) | (0.4) | ||
| 9.1.6 | -0.0 | -0.2 | -0.0 | Difference-in-differences with demographic controls 1995 vs. 1998 |
| (0.4) | (0.2) | (0.4) | ||
| 9.1.7 | 0.2 | 0.1 | -0.4 | Linear regression, no demographic controls, 1996 activity controls |
| (0.3) | (0.1) | (0.3) | ||
| 9.1.8 | -0.2 | 0.1 | 0.1 | Linear regression, demographic controls, 1996 activity controls |
| (0.3) | (0.1) | (0.3) | ||
| 9.1.9 | 0.2 | 0.1 | -0.4 | Linear regression, no demographic controls, 1995 and 1996 activity controls |
| (0.3) | (0.1) | (0.3) | ||
| 9.1.10 | -0.2 | 0.1 | 0.1 | |
| (0.3) | (0.1) | (0.3) |
This section analyzes the impact of EBSM on earnings, using both survey-based data on post-program earnings and administrative data on pre-program earnings. The survey asked respondents how much money they earned before deductions from all jobs in 1998, and this is used as the measure of post-program earnings for EBSM participants and non-participants. The administrative data provide information on gross earnings for the preprogram years 1992 through 1996. Because the survey-based and administrative data on earnings may differ for a variety of reasons some caution is required in interpreting the results of the analysis of earnings. However, as noted in the previous section, combining survey-based and administrative information will not bias the results as long as any differences between the two sources of data affect both the participants and non-participants to a similar extent. In these circumstances, the differences associated with the alternative earnings measures are removed by having a comparison group of non-participants.
In order to separate the earnings behavior associated with EBSM participation from the possible impact of the program on subsequent earnings, the focus is on those participants who completed EBSM in 1997. After removing those individuals who did not respond to the earnings question in the survey, this leaves a sample of 311 participants and 284 non-participants. The results reported in Table 9.2.1 are based on this sample and subsets of the participant sample for the various program components.
Earnings is the most commonly used measure of success associated with employment and training programs because it combines both the effects of employability (weeks worked per year, hours worked per week) and the "price" of labour (weekly or hourly wage rate). For this reason the analysis includes those who report zero earnings. Thus results are for all the observations available on participants and non-participants, and are not conditional on the subset who report positive earnings.
Average annual earnings of EBSM participants was about $ 670 higher than that of non-participants in 1998, although the difference is not statistically significant (mean difference of $ 669; see equation 9.2.1). However, once observed demographic and individual characteristics are controlled, the regression-adjusted mean difference is negative and approximately $1,500 smaller (see equation 9.2.2). This indicates that EBSM selected (or was selected by) individuals with observable characteristics that made them likely to earn less than the comparison group of non-participants. This indication is confirmed by inspection of the administrative data on pre-program earnings: those who became enrolled in the ESBM program consistently earned less during the years 1992 through 1996 than did those who did not participate in EBSM. Thus there is evidence of non-random selection into the program. EBSM participants have observable (and possibly also unobservable) characteristics that result in lower earnings than non-participants even in the absence of the program. Estimates of program impact that do not take into account this non-random selection — such as the mean difference in post-program earnings (equation 9.2.1) and the regression-adjusted mean difference in post-program earnings (equation 9.2.2) — are likely to be biased.
Equations 9.2.3 to 9.2.12 report difference-in-differences estimates of program impact for alternative choices of the base year (1992 to 1996). The simple difference-in-differences estimates are appropriate when selection into the program is influenced by unobserved factors that also influence earnings. These unobserved factors may vary across individuals but are assumed to be constant over time. In these circumstances, taking pre-versus post-program differences removes the influence of these unobserved factors, thus providing estimates that are free from selection bias. The difference-in-differences estimates with demographic controls take account of both observed and unobserved differences between participants and non-participants. These regression-adjusted difference-in-differences estimates are thus more general and the analysis will therefore focus on these.
For EBSM as a whole, the regression-adjusted difference-in-differences estimates are all positive but none is significantly different from zero. The magnitude of the estimates ranges from $1,084 using 1996 as the base year to $2,403 using 1995 as base year. Note that apart from the 1996 base year estimate, the range of estimates is reasonably narrow: $1,826 (1992 base year) to $2,403 (1995 base year). A common finding in evaluating employment and training programs is that the earnings of program participants "dip" in the year prior to entry into the program (i.e., 1996 in the case of EBSM). This decline in earnings in the period prior to the program is not surprising; it simply indicates that those who apply for or who are selected for employment and training programs enter these programs because they have unusually low earnings due to job loss or other employment related adverse outcomes. For this reason, earnings in the period immediately prior to entry into the program are not representative of the "normal earnings" of these individuals. Accordingly, longitudinal estimates of program impact generally avoid using the data from the period just prior to program entry. When the comparison group is chosen from large nationally representative surveys, the earnings of the non-participants generally do not display this decline in the period immediately prior to the program. This outcome is to be expected because the sample of non-participants is not drawn from a group that has experienced a poor employment-related outcome. In the analysis of EBSM, it is fortunate that the comparison group is drawn from regular EI recipients, many of whom, like the participant group, have entered EI because of an adverse outcome such as job loss. Thus the earnings of comparison group members are expected to also be unusually low relative to their historical pattern in the period just prior to the EI spell. This is in fact the case: the earnings of both the participant and non-participant groups decline noticeably in 1996 relative to their average levels over the 1992-1995 period. Because there would generally be a tendency for earnings of both groups to return to their usual levels even in the absence of the program, it is advisable to place less weight on the difference-in-differences estimates for the pre-program year 1996 than for earlier years 1992 to 1995. As a consequence, the principal focus will be on the estimates using the base years 1992 to 1995.
As discussed previously, an important test of the key assumption which underlies the difference-in-differences estimates — that the unobserved factors which influence both participation in the program and the earnings of participants are constant over time — is that the estimates for different choices of the base year should not be statistically different from each other. Both the simple difference-in-differences estimates for the base years 1992 to 1995 and the regression-adjusted difference-in-differences estimates for the same years pass this specification test. The more general regression-adjusted estimates range from $1,826 (1992 base year) to $2,403 (1995 base year), and these estimates are all within one standard error of each other (see the standard errors reported in equations 9.2.6, 9.2.8, 9.2.10, and 9.2.12 in Table 9.2.1). That the difference-in-differences estimates pass this specification test increases confidence in these estimates of program impact.
The longitudinal estimates that include pre-program earnings as controls (equations 9.2.13 to 9.2.15) are consistently negative and fall in the range of -$283 to -$467. However, none of these estimates is significantly different from zero, so they are consistent with the regression-adjusted difference-in-differences estimates that also find no significant impact of EBSM on earnings. Note that although these estimates control for observed differences in pre-program earnings between participants and non-participants, they do not take account of unobserved factors that influence both participation and earnings and will generally be biased estimates of program impact in the presence of such factors. The fact that the difference-in-differences estimates pass the specification test of being invariant with respect to the base year suggests the presence of such unobserved influences on participation and earnings. For this reason, the evidence provides more support for the difference-in-differences specification than for the specification using pre-program earnings as controls.
In summary, both sets of longitudinal estimates — those based on the regression-adjusted difference-in-differences specification and those based on the specification including preprogram earnings as controls — support the conclusion that EBSM did not have a significant impact on earnings in the year following program completion. The difference-in-differences estimates are consistently positive and indicate an impact of approximately + $2,000, but these impacts are imprecisely estimated and one cannot reject the hypothesis that they are not significantly different from zero. Similarly the regressions which include pre-program earnings as controls are consistently negative but do not differ significantly from zero.
The results reported in Table 9.2.1 provide an impact estimate that is common to all participants in the program. In order to investigate whether the intervention was more effective for some types of participants than for others, the analysis also estimated each of the equations reported in Table 9.2.1 allowing the estimated EBSM impacts to differ by gender, visible minority status and disability status. The results suggest that there are not significant differences in program impact by these characteristics. Specifically, estimated impacts are generally (but not always) higher for females than for males, but the differences between females and males are never statistically significant. Estimated impacts are consistently higher for the disabled than for non-disabled, but none of these differences is statistically significant. With respect to visible minorities, the estimated impacts are smaller than for the other participants, and in the majority of cases these differences are not statistically significant. Thus it is concluded that there is no clear evidence that the impacts of the program on earnings (which are themselves not significantly different from zero) differ by gender, visible minority status or disability status.
Table 9.2.1 also reports estimates of the impact of the program on earnings for the various program components. There are large differences across the program components in the earnings of participants in the year following program completion (see equation 9.2.1). Those in the Self-Employment, Job Creation Partnerships, Targeted Wage Subsidies and Employment Assistance Services components earned less than their counterparts in the non-participant group during 1998, with the largest gap being Job Creation Partnerships participants whose earnings averaged approximately $7,500 below those of non-participants. In contrast, the Purchase of training participants earned $3,186 more than non-participants in 1998. These summary statistics suggest that there may have been large differences across the program components in the impact of EBSM on earnings.
In order to determine whether these differences in average post-program outcomes can be attributed to the program, the analysis turns to the longitudinal estimates of program impact. For reasons discussed previously, the researchers have the most confidence in the regression-adjusted difference-in-differences estimates for the base years 1992 to 1995 (equations 9.2.6, 9.2.8, 9.2.10, and 9.2.12). For each of the components Self-Employment, Job Creation Partnerships, Targeted Wage Subsidies and Employment Assistance Services, the estimated impacts based on the longitudinal estimators are generally negative; however, because the sample sizes are quite small most of the estimates are very imprecise, with standard errors often larger than the estimated impact coefficients. Thus the only statement that can be made with confidence about the impact of these four program components is that they do not appear to have significantly reduced earnings of participants relative to what their earnings would have been in the absence of the program.
The one exception is the Purchase of training component. The regression-adjusted difference-in-differences estimates are consistently positive and fall in the range of $3,206 (1994 base year) to $3,673 (1992 base year). None of the estimates for the base years 1992 to 1995 inclusive is statistically significantly different from each other, and three of the four estimates are significantly different from zero at the 10 percent level of significance, with the fourth being on the margin of significance at that level of significance. Thus the difference-in-differences estimates for the Purchase of training component pass the specification test of the assumptions underlying that type of longitudinal estimator. As was the case for the estimates for EBSM as a whole, the longitudinal estimates which simply include pre-program earnings as controls are smaller in magnitude (ranging from $483 to $699) and are not significantly different from zero. For the reasons discussed previously with respect to the EBSM estimates, the difference-in-differences estimates are superior. Accordingly there is some evidence that the Purchase of training option raised the earnings of Purchase of training participants by approximately $3,500 in the year following the program.
As a final step in the analysis of earnings, the evaluators examined whether those who contributed to the cost of their intervention did better in terms of post-program earnings than those who didn't. In the case of the Purchase of training component, the analysis tested whether those who paid for some of their training had estimated earnings impacts that were different than those who did not contribute to the cost of their training. The difference-in-differences estimates (with demographic controls) for the base years 1992 to 1995 are consistently larger for those who contributed to the cost of their training than for Purchase of training participants who did not so contribute. However, none of these estimates is significant at the 5 percent level, and only two (those for the 1995 and 1994 base years) are significant at the 10 percent level. Although the estimates are all positive, they are too imprecise to reach a firm conclusion about the impact of contributing to the cost of training.
Similarly, for the Self-Employment component the evaluators investigated whether those who contributed to the initial capital investment in their business did better in terms of subsequent earnings.41 Because all the Self-Employment participants who completed the program during 1997 contributed a positive amount to the business, the analysis examined whether there was any difference between those who contributed less than the median amount of $9,000 and those who contributed more than the median amount. No systematic differences were found between these two subsets of the Self-Employment component. The estimated earnings impacts are sometimes smaller for those contributing above the median amount and sometimes larger, and the differences between those with above and below median contributions were not significant for any of the choices of base year.
In this section impact on use of Employment Insurance is investigated, using administrative data on paid weeks of EI benefits and total EI benefits received. These data are available on an annual basis from 1992 to 1998. In order to distinguish between any effects of participation in the program on EI receipt during the program itself and any impacts of the program on subsequent behavior, the analysis focuses on those participants who completed EBSM during 1997. For these individuals, the administrative data for 1998 provide some early indication of the possible impacts of the program on reliance on EI.
Tables 9.3.1 and 9.3.2 provide estimates of the impact of EBSM and its components on two measures of EI receipt: (1) Paid Weeks of EI Benefits and (2) Total EI Benefits. In the post-program year 1998, EBSM participants received 0.6 weeks more EI benefits than did non-participants (equation 9.3.1) although the difference was not statistically significant. However, those who participated in EBSM received, on average, EI benefits for two to three weeks more than did non-participants during the period prior to EBSM (1992 to 1996). Accordingly, most of the longitudinal estimates of program impact, which take account of pre-program differences in weeks of EI benefits, suggest that EBSM may have reduced weeks of EI benefit receipt. The regression-adjusted difference-in-differences estimates are, however, not fully conclusive because they range from -0.3 weeks (and not significantly different from zero) using 1992 as the base year to -2.5 weeks using 1994 as the base year. The difference-in-differences estimates do not pass the specification test of being invariant with respect to the choice of base year. Thus there is some evidence that EBSM may have reduced weeks of EI receipt, but one could also interpret this evidence as suggesting that the program had no significant impact on weeks of EI benefits.
The estimated impacts on total EI benefits are similarly inconclusive. The regression-adjusted difference-in-differences estimates range from + $13 (and not significantly different from zero) using 1992 as the base year to - $530 using 1994 as the base year. Again these estimates do not pass the base year specification test. Some of the longitudinal estimates suggest that EBSM reduced EI benefits received by $400 to $500 per year, while others suggest that the program did not have a significant impact on EI benefits. It is not possible on the basis of the available information to choose between these alternative conclusions.
However, one clear conclusion does follow from the evidence in Tables 9.3.1 and 9.3.2: EBSM does not appear to have increased reliance on EI, as measured by weeks of EI benefits or total EI benefits received. This finding is important because the survey-based evidence on post-program EI use at a point in time (see section 9.5) suggests that EBSM may have increased reliance on EI. Because the analysis in this section is based on EI use by participants and non-participants before and after the program, these results are more robust than in those obtained in that section.
As indicated in Tables 9.3.1 and 9.3.2, there are large differences across program components in the estimated impacts of the program on EI use. The Self-Employment component appears to have been particularly effective in reducing EI weeks and total benefits received. All the estimated impacts on weeks of EI are negative and fall in the range of 7 to 14 weeks. Similarly, the estimated impacts on total EI benefits are consistently negative and lie in the range of $2,000 to $4,000. Thus there is considerable evidence that the Self-Employment component had a substantial impact on EI use.
There is also some, albeit less conclusive, evidence that the Job Creation Partnerships component reduced reliance on EI during 1998. Keep in mind, though, that work under Job Creation Partnerships is not insurable, and this alone could account for decreased post-program use of EI. The regression-adjusted difference-in-differences estimates of the impact of Job Creation Partnerships on weeks of EI benefits are consistently negative and fall in the range of —3.0 to —7.1 weeks; however, not all these estimates are significantly different from zero. Similarly, all the longitudinal estimates of the impact of Job Creation Partnerships on total EI benefits are negative, although not all are significantly different from zero.
Most of the estimated impacts of the Purchase of training component on EI use are also negative, suggesting that if anything this program component acted to reduce EI weeks and benefits. However, these estimates are smaller than those obtained for Self-Employment and Job Creation Partnerships and several are not significantly different from zero. Thus the Purchase of training component may have reduced EI use to a modest degree, but the evidence supporting this finding is not conclusive. One could also conclude from the evidence in Tables 9.3.1 and 9.3.2 that the Purchase of training component had no significant impact on reliance on EI.
Finally, the Targeted Wage Subsidies and Employment Assistance Services components appear to have had no impact on EI use in the year following the completion of the program.
In summary, there is some (weak) evidence that EBSM reduced EI use, measured in terms of both weeks of benefits and total benefits received. To the extent that this reduction occurred, it came principally from the Self-Employment component that had a large impact on the use of EI, and to a lesser extent from the Job Creation Partnerships component. The Purchase of training component may also have contributed to a reduction in EI receipt. The Targeted Wage Subsidies and Employment Assistance Services components appear to have had no impact on this outcome.
This section examines the impact of the program on social assistance receipt to determine whether there is evidence that the program helped reduce reliance on social assistance. The survey asked participants and non-participants whether they received social assistance during 1998, as well as the number of months of social assistance receipt. The administrative data provide information on income from social assistance for each year from 1992 to 1997. Combining the administrative and survey data yields a variable that measures whether or not the individual received social assistance benefits sometime during the year; this variable is constructed from the administrative data for the years 1992 to 1996 and from the survey data for 1998. Because the analysis combines administrative and survey data, some caution is required in interpreting the results. In particular, it is generally found to be the case that survey respondents tend to under-report both whether or not they received social assistance and the amount of social assistance income received. However, this tendency will not bias the results providing that program participants and non-participants under-report social assistance receipt to the same degree.
For some of the survey respondents, their extent of reliance on social assistance during 1998 combines the effects of participating in the program with any post-program impacts of EBSM. As in the previous section, in order to obtain estimates of EBSM impacts that are not contaminated in this way the analysis is restricted to those participants who completed EBSM in 1997. This yields a sample of 557 participants and 499 non-participants. The estimates for this group thus provide evidence on reliance on social assistance during the year after program completion.
EBSM participants were more likely to receive social assistance following the program than were the non-participants (11.3 percent versus 5.4 percent, a difference of 5.9 percentage points). However, EBSM participants were also more likely to receive social assistance benefits prior to the program. For example, the difference in social assistance recipiency rates was 5.7 percentage points in 1996 and 4.3 percentage points in 1995. Clearly there is evidence that the EBSM tended to select participants who had a much higher likelihood of needing income support in the form of social assistance than was the case for non-participants.42
Table 9.4.1 reports estimates of program impact for EBSM as a whole as well as for the five program components. Because the dependent variable is an indicator or limited dependent variable (SAR98 = 1 if individual received social assistance in 1998; SAR98 = 0 otherwise) a probit specification is employed.43 Because the parameter estimates themselves are difficult to interpret, the table reports the marginal probabilities. These can be interpreted as the estimated effect of changing from a value of 0 for the participation dummy (i.e., a non-participant) to a value of 1 for this dummy variable (i.e., a participant). Thus, for example, in the simplest specification of a probit model with no controls, the estimated marginal probability of 0.05 corresponds to an increase in the likelihood of social assistance receipt of 5 percentage points, similar to the unweighted mean difference between participants and non-participants of 5.9 percentage points (see above). Note, however, that the estimated marginal probability (difference between participants and non-participants) declines to 4 percentage points once individual and demographic factors are controlled (equation 9.4.2).
Most of the estimated program impacts become essentially zero once pre-program levels of social assistance receipt are controlled. With the exception of the estimates using 1992 as the base year, the difference-in-differences estimates are close to zero in magnitude and none are significantly different from zero. Note also that the difference-in-differences estimates from the years 1993 to 1996 pass the specification test of being invariant with respect to the choice of base year. Accordingly the researchers have more confidence in these estimates than in those based on the 1992 base year or those using probit models which control for pre-program levels of social assistance receipt. Thus EBSM had no impact on social assistance receipt. In this context, it is worth noting that a simple comparison of social assistance usage in the post-program year 1998 would show the EBSM participants with a level of SA receipt more than double that of the comparison group, a difference that might be attributed in part to the program. The analysis suggests that such an attribution would be a mistake.
The results in Table 9.4.1 indicate that participation in EBSM did not reduce reliance on social assistance. This conclusion also holds for most of the components. For Self-Employment, Targeted Wage Subsidies, and Employment Assistance Services none of the impacts based on longitudinal estimates is statistically significant. In the case of Job Creation Partnerships, the cross-sectional estimates using 1998 data alone (equations 9.4.1 and 9.4.2) do indicate a positive and significant effect associated with participation in Job Creation Partnerships (i.e., suggesting that participation in Job Creation Partnerships increased reliance on social assistance). However, this result disappears once pre-program levels of social assistance receipt are controlled for using the difference-indifferences methodology. These difference-in-differences estimates also pass the base year specification tests, and accordingly are preferable to the probit model estimates that control for pre-program levels of social assistance receipt. Thus Job Creation Partnerships — like most of the other EBSM components — did not increase (or decrease) reliance on social assistance.
The one component that may have resulted in an increase in social assistance receipt is Purchase of training. The longitudinal estimates of program impact are consistently positive, and range from an impact of 0.01 (i.e., one percentage point) which is not significantly different from zero to 0.08 and significantly greater than zero at the 1 percent level of significance. The difference-in-differences estimates do not pass the base year specification tests in this case. The evidence is clearly mixed, and one could conclude either that Purchase of training had no impact on social assistance receipt (i.e., using the information from the most recent pre-program years, 1994, 1995 and 1996) or that it had an impact of raising social assistance receipt by 6 to 8 percentage points (i.e., using the base year estimates from 1992 and 1993).
The survey also asked respondents about their current activities, including whether or not the individual is currently on social assistance. Information regarding current activities is analyzed next.
The survey asked participants and non-participants about their current activities, including working on a paid job, self-employed, looking for a job, upgrading one's education, in a job training program, on social assistance and on employment insurance. The responses to these questions are examined in this section. Because the information relates to post-program activities for all participants and non-participants, the analysis includes the responses for all individuals in the sample, rather than for those who completed EBSM in 1997 as in the previous sections in this chapter. Thus for this section there are 1,033 observations on EBSM participants and 499 observations on non-participants. Although being able to examine the current activities of all the survey respondents is a desirable feature of this section, an offsetting disadvantage is that there is no comparable information on pre-program activities. Specifically, the survey question on current activities refers to activities at a point in time (the date of the survey), whereas the information available for the period prior to the program relates to activities during a calendar year. Administrative data are used as much as possible in this section in order to take account of observable pre-program differences between the participants and non-participants. However, in the absence of point-in-time measures of activity levels prior to the program, difference-in-differences estimators of program impact are not possible. Accordingly, the range of model specifications that can be investigated in this section is more limited than that available in previous sections.
The analysis combines the various current activities into five main groups: EMPLOYED (includes working in a paid job and self-employed), UNEMPLOYED (looking for a job), EDUCATION/TRAINING (upgrading one's education or in a job training program), ON SOCIAL ASSISTANCE, and ON EI. For the Self-Employment component the researchers also investigate the impact of the program on the likelihood of being self-employed. Each of the dependent variables is defined in indicator (or "dummy") form; that is, EMPLOYED = 1 if the individual is currently employed in a paid job or self-employed and EMPLOYED = 0 otherwise, and similarly for SELF-EMPLOYED, UNEMPLOYED, EDUCATION/TRAINING, ON SOCIAL ASSISTANCE, and ON EI.
Because each of the dependent variable is an indicator or (0,1) variable, all the models use the probit specification. All the estimated impact effects have been converted to marginal probabilities for ease of interpretation.
The results are presented in Tables 9.5.1 to 9.5.6. In each table, the first two equations use only the post-program cross-sectional data and the remaining equations control for pre-program activity levels using the administrative data. As discussed below, some of these pre-program controls are quite crude and thus the associated results should be treated with caution.
Equation 9.5.1 indicates that, at the survey date, EBSM participants were approximately six percentage points less likely to be employed than non-participants. This difference of 6 to 7 percentage points remains when individual and demographic characteristics as well as for pre-program employment levels are controlled. The pre-program controls, however, are relatively crude. Information on gross earnings for the years 1992 to 1996 is used to construct an indicator variable for those who were employed sometime during the year (i.e., reporting positive gross earnings) versus not employed during the year (gross earnings of zero). As the results in Table 9.5.1 indicate, adding these pre-program employment indicator controls essentially leaves the cross-sectional estimates unchanged. Clearly the EBSM participants are somewhat less likely to be employed than the non-participants. These differences appear to be principally driven by the Purchase of training component, with differences of a similar magnitude but not statistically significant being found for the Job Creation Partnerships and Employment Assistance Services components.
| Equation | EBSM | JCP | TWS | EAS | POT | Model Specification |
| 9.5.1 | -0.060** | -0.077 | 0.023 | -0.066 | -0.077** | Probit regression, no controls |
| (0.025) | (0.056) | (0.045) | (0.047) | (0.034) | ||
| 9.5.2 | -0.074*** | -0.130** | -0.001 | -0.067 | -0.085** | Probit regression, demographic controls |
| (0.027) | (0.060) | (0.049) | (0.050) | (0.037) | ||
| 9.5.3 | -0.072*** | -0.093 | 0.037 | -0.072 | -0.082** | Probit regression, demographic controls, 1996 activity controls |
| (0.028) | (0.063) | (0.050) | (0.053) | (0.038) | ||
| 9.5.4 | -0.073*** | -0.094 | 0.039 | -0.065 | -0.083** | Probit regression, demographic controls, 1996 and 1995 activity controls |
| (0.028) | (0.064) | (0.050) | (0.055) | (0.038) | ||
| 9.5.5 | -0.072*** | -0.073 | 0.047 | -0.059 | -0.086** | Probit regression, demographic controls, 1996, 1995 and 1994 activity controls |
| (0.029) | (0.065) | (0.050) | (0.056) | (0.039) | ||
| 9.5.6 | -0.070** | -0.078 | 0.053 | -0.081 | -0.083** | Probit regression, demographic controls, 1996, 1995, 1994 and 1993 activity controls |
| (0.029) | (0.067) | (0.051) | (0.059) | (0.040) | ||
| 9.5.7 | -0.067** | -0.081 | 0.058 | -0.101* | -0.077* | Probit regression, demographic controls, 1996, 1995, 1994 and 1993 and 1992 activity controls |
| (0.030) | (0.069) | (0.052) | (0.061) | (0.041) |
Although the results in Table 9.5.1 are consistent across the various specifications, they do not provide convincing evidence that the program reduced the likelihood of employment among EBSM participants. The differences between the two groups could reflect unobserved factors that affect both EBSM participation and the likelihood of employment.
Table 9.5.2 reports comparable results for the impact of the Self-Employment component on the likelihood of being self-employed as of the survey date. There is an enormous difference between the incidence of self-employment among Self-Employment participants and that among non-participants. This difference of 66 to 69 percentage points remains when demographic factors and for pre-program levels of self-employment activity are controlled. The administrative data reports self-employment income for each of the years 1992 to 1996, and this information is used to construct a variable indicating whether the individual was self-employed (i.e. received positive self-employment income) during the year in question. During the years 1992-1996, approximately 6 to 7 percent of those who became EBSM participants were self-employed by this measure; this self-employment rate was quite stable over the 1992-96 period. The self-employment rate among non-participants was in the 8 to 10 percent range, and was also quite stable over time. By these indicators there is no evidence that the Self-Employment component tended to select (or be selected by) individuals who were more likely to be self-employed even in the absence of the program. Indeed, the non-participants appear to be slightly more likely to be self-employed in the absence of the program than do the Self-Employment participants. Thus there is some evidence to suggest that at least some of the large differences between Self-Employment participants and non-participants can be attributed to the program. Indeed, given the pre-program incidence of self-employment among the two groups, it would not be unreasonable to conclude that much of the 66 - 69 percent differential in self-employment rates after the program can be attributed to the intervention.
| Equation | SEA | Model Specification |
| 9.5.1 | 0.660*** (0.088) |
Probit regression, no controls |
| 9.5.2 | 0.691*** (0.096) |
Probit regression, demographic controls |
Table 9.5.3 presents results on unemployment status at the time of the survey. After controlling for demographic factors, EBSM participants were about 10 percentage points more likely to be looking for a job than non-participants. This difference between participants and non-participants declines slightly (to about 9 percentage points) when pre-program levels of unemployment are controlled for using weeks on EI as a proxy for weeks of unemployment. Clearly there is no evidence that EBSM helped to reduce the incidence of unemployment among participants. Because there is no good measure of job search activity for the pre-program period, a firm conclusion from this evidence is not possible. The results suggest that if the program had any effect on unemployment it was to increase job search activity rather than decrease it. The program components which are associated with a significantly higher level of job search activity among participants than non-participants are Job Creation Partnerships, Employment Assistance Services and Purchase of training.
For the activities of "Upgrading your education" and "In a job training program" there are no proxy measures for pre-program activity. Thus Table 9.5.4 reports results using the post-program cross-sectional data alone. After controlling for individual and demographic factors, the probability that EBSM are engaged in education or training is about 4 percentage points higher than is the case for non-participants. This difference arises from participants in the Purchase of training component; for the other program components there are no significant differences between the two groups. This greater propensity to be engaged in education or training among the Purchase of training participants could be an impact of the program or could reflect unobserved factors that influence both the likelihood of being engaged in education or training and the likelihood of participating in the Purchase of training program.
| Equation | EBSM | SEA | JCP | TWS | EAS | POT | Model Specification |
| 9.5.1 | 0.065*** | -0.023 | 0.010 | -0.009 | 0.058** | 0.085*** | Probit regression, no controls |
| (0.015) | (0.040) | (0.030) | (0.023) | (0.031) | (0.021) | ||
| 9.5.2 | 0.042*** | -0.002 | 0.006 | -0.017 | 0.022 | 0.063*** | Probit regression, demographic controls |
| (0.015) | (0.043) | (0.026) | (0.018) | (0.025) | (0.022) | ||
| 9.5.3 | 0.042*** | -0.002 | 0.006 | -0.017 | 0.022 | 0.063*** | Probit regression, demographic controls, 1996 activity controls |
| (0.015) | (0.043) | (0.026) | (0.018) | (0.025) | (0.022) | ||
| 9.5.4 | 0.042*** | -0.002 | 0.006 | -0.017 | 0.022 | 0.063*** | Probit regression, demographic controls, 1996 and 1995 activity controls |
| (0.015) | (0.043) | (0.026) | (0.018) | (0.025) | (0.022) | ||
| 9.5.5 | 0.042*** | -0.002 | 0.006 | -0.017 | 0.022 | 0.063*** | Probit regression, demographic controls, 1996, 1995 and 1994 activity controls |
| (0.015) | (0.043) | (0.026) | (0.018) | (0.025) | (0.022) | ||
| 9.5.6 | 0.042*** | -0.002 | 0.006 | -0.017 | 0.022 | 0.063*** | Probit regression, demographic controls, 1996, 1995, 1994 and 1993 activity controls |
| (0.015) | (0.043) | (0.026) | (0.018) | (0.025) | (0.022) | ||
| 9.5.7 | 0.042*** | -0.002 | 0.006 | -0.017 | 0.022 | 0.063*** | Probit regression, demographic controls, 1996, 1995, 1994 and 1993 and 1992 activity controls |
| (0.015) | (0.043) | (0.026) | (0.018) | (0.025) | (0.022) |
For the final two activities (social assistance and employment insurance receipt) there are good measures of pre-program activity levels from the administrative data. Still difference-in-differences estimates are not possible because the pre-program information relates to social assistance and EI receipt during the calendar year whereas the post-program measure is based on the time of the survey. However, the analysis is able to control for observed differences in social assistance and EI use in the 1992 to 1996 period in addition to taking account of demographic and individual characteristics. Turning first to social assistance receipt, the EBSM participants were slightly more likely to be on social assistance at the time of the survey, and this difference (of approximately one percentage point) remains after controlling for individual characteristics. However, once pre-program levels of social assistance receipt are controlled, the differences between EBSM participants and non-participants become essentially zero and statistically insignificant. This result also holds for each of the Self-Employment, Targeted Wage Subsidies, Employment Assistance Services and Purchase of training components. However, participants in the Job Creation Partnerships component remain about two percentage points more likely to be on social assistance than non-participants after taking account of pre-program levels of social assistance receipt.
In summary, there is no evidence from the information on activities at the time of the survey that EBSM reduced the reliance on social assistance. This finding supports the conclusion reached in section 9.4 which was based on social assistance receipt during the calendar year 1998. The results for EBSM as a whole also hold for the Self-Employment, Targeted Wage Subsidies, Employment Assistance Services and Purchase of training components. For the Job Creation Partnerships component the program may have increased the likelihood of being on social assistance by a modest amount (raising the probability by approximately 0.02). This result is not consistent with that found earlier (in section 9.4) where it was concluded that the Job Creation Partnerships component — as was the case for other program components — did not alter the likelihood of relying on social assistance. Thus there is mixed evidence on this issue. Nonetheless it should be noted that the differences in findings are not large, with the results in section 9.4 indicating a zero impact on social assistance receipt and those in this section indicating a small positive impact. The difference in results is probably attributable to the differences in pre-program information, with that in section 9.4 being better than that available for the analysis in this section.
| Equation | EBSM | SEA | JCP | TWS | EAS | POT | Model Specification |
| 9.5.1 | 0.014* | 0.003 | 0.057*** | 0.029** | 0.052*** | 0.003 | Probit regression, no controls |
| (0.007) | (0.025) | (0.028) | (0.019) | (0.022) | (0.008) | ||
| 9.5.2 | 0.007** | 0.008 | 0.063*** | 0.018** | 0.027** | 0.003 | Probit regression, demographic controls |
| (0.003) | (0.021) | (0.033) | (0.015) | (0.020) | (0.004) | ||
| 9.5.3 | 0.004 | 0.012 | 0.049*** | 0.017* | 0.006 | 0.002 | Probit regression, demographic controls, 1996 activity controls |
| (0.003) | (0.028) | (0.030) | (0.015) | (0.011) | (0.003) | ||
| 9.5.4 | 0.003 | 0.008 | 0.034*** | 0.010 | 0.002 | 0.001 | Probit regression, demographic controls, 1996 and 1995 activity controls |
| (0.003) | (0.019) | (0.025) | (0.011) | (0.008) | (0.003) | ||
| 9.5.5 | 0.001 | 0.012 | 0.024*** | 0.007 | 0.001 | -0.000 | Probit regression, demographic controls, 1996, 1995 and 1994 activity controls |
| (0.002) | (0.028) | (0.021) | (0.010) | (0.007) | (0.002) | ||
| 9.5.6 | 0.001 | 0.013 | 0.018* | 0.004 | 0.000 | -0.000 | Probit regression, demographic controls, 1996, 1995, 1994 and 1993 activity controls |
| (0.002) | (0.028) | (0.022) | (0.008) | (0.007) | (0.002) | ||
| 9.5.7 | 0.001 | 0.019 | 0.022* | 0.006 | 0.002 | 0.000 | Probit regression, demographic controls, 1996, 1995, 1994 and 1993 and 1992 activity controls |
| (0.002) | (0.038) | (0.027) | (0.010) | (0.008) | (0.002) |
Although EBSM does not appear to have had any impact on social assistance receipt, the results in Table 9.5.6 indicate that the program may have increased reliance on EI. The differences in EI receipt as of the date of the survey between participants and non-participants are moderately large (about 10 percentage points) and these decline only slightly after controlling for individual characteristics and pre-program levels of EI incidence. This result is being driven principally by the Purchase of training component, with the estimated impacts of the other components being small and not statistically significant. Thus the Purchase of training component appears to have increased EI incidence by approximately 12 percentage points, while the other components had no significant impact on reliance on EI. The point-in-time estimates are at odds with the longitudinal findings that EBSM may have modestly reduced EI dependency (section 9.3). Estimates based on the longitudinal data are preferred because they control for preprogram differences.
| Equation | EBSM | SEA | JCP | TWS | EAS | POT | Model Specification |
| 9.5.1 | 0.097*** | -0.139** | 0.052 | 0.001 | 0.001 | 0.141*** | Probit regression, no controls |
| (0.022) | (0.041) | (0.048) | (0.038) | (0.038) | (0.030) | ||
| 9.5.2 | 0.096*** | -0.098 | 0.075* | -0.001 | -0.001 | 0.133*** | Probit regression, demographic controls |
| (0.023) | (0.050) | (0.051) | (0.037) | (0.037) | (0.032) | ||
| 9.5.3 | 0.107*** | -0.091 | 0.072 | 0.002 | 0.002 | 0.143*** | Probit regression, demographic controls, 1996 activity controls |
| (0.023) | (0.049) | (0.051) | (0.038) | (0.038) | (0.033) | ||
| 9.5.4 | 0.096*** | -0.093 | 0.066 | -0.003 | -0.003 | 0.128*** | Probit regression, demographic controls, 1996 and 1995 activity controls |
| (0.024) | (0.051) | (0.052) | (0.038) | (0.038) | (0.033) | ||
| 9.5.5 | 0.093*** | -0.095 | 0.054 | -0.011 | -0.011 | 0.124*** | Probit regression, demographic controls, 1996, 1995 and 1994 activity controls |
| (0.024) | (0.048) | (0.051) | (0.038) | (0.038) | (0.034) | ||
| 9.5.6 | 0.091*** | -0.095 | 0.052 | -0.015 | -0.015 | 0.122*** | Probit regression, demographic controls, 1996, 1995, 1994 and 1993 activity controls |
| (0.025) | (0.047) | (0.052) | (0.038) | (0.038) | (0.035) | ||
| 9.5.7 | 0.090*** | -0.096 | 0.062 | -0.032 | -0.032 | 0.122*** | Probit regression, demographic controls, 1996, 1995, 1994 and 1993 and 1992 activity controls |
| (0.025) | (0.046) | (0.055) | (0.037) | (0.037) | (0.036) |
It should be noted that the impact results presented here are preliminary only and that more definitive results will be presented in a summative evaluation when more time after program completion will have elapsed. The impact analysis showed the following for those participants who completed the program in 1997:
Lack of pre-program data make the following conclusions less robust because the differences between the two groups could reflect unobserved factors that affect both EBSM participation and the outcome:
|
By way of summary and conclusion, this closing chapter will present a capsulized response to each evaluation question.52 Chapter/section numbers from this report are provided for the reader who wants more specifics.
1. To what extent are the Employment Benefits and Support Measures (EBSM) delivered under the Labour Market Development Agreement (LMDA) consistent with the EI legislation? With the priorities of Human Resources Development Canada (HRDC) and Nova Scotia (NS)? What are the strengths and weaknesses of the strategic partnership approach?
EBSM is consistent with the EI legislation, however, program guidelines such as targeting, selection, and case management are not always closely adhered to, as discussed in Chapters Four and Six.
HRDC Nova Scotia is satisfied that it had ample input into the design of the program for Nova Scotia. Informants felt that EBSM is consistent with the priorities of HRDC.
Although the LMDA is experiencing some challenges, this is to be expected at the early stages of any partnership. Concerns raised by key informants include a cumbersome committee structure, committee members being on multiple committees, and communication problems. LMDA partners are addressing most of the concerns uncovered in the evaluation by streamlining the committee structure (section 5.2).
2. To what extent are the EBSMs relevant to the employment needs of individuals? To the labour market needs of employers? To the needs of communities? Have adjustments been made to make the EBSMs relevant to these needs?
Qualitative evidence suggests that the EBSM are meeting client needs. Most EBSM participants (78 percent) thought the program was good or excellent. Few gave the program a failing grade (4 percent) or a below average grade (5 percent). Clients in every EBSM component gave a mean overall grade of B, except for Employment Assistance Services clients who gave a B-.
Although clients were disappointed with the program's help with finding them a permanent job, 88 percent felt fairly or very well prepared for a new job as a result of their intervention. The C grade for finding a permanent job may therefore be a comment on client perception of the job market rather than the program.
Every employer interviewed for the case studies was enthused with the Targeted Wage Subsidies option (section 6.5). They also asserted that it helped them meet their labour market needs. HRCC staff were divided on whether employer needs were met. In areas where Targeted Wage Subsidies is employer driven, employer needs were said to be met. Where Targeted Wage Subsidies was client-driven, staff were not sure if employer needs were met. They felt options for on-the-job training were limited.
Because of local level flexibility it is fair to conclude that EBSM has good potential to be relevant to the needs of the community. Certainly every community agency representative interviewed in the three communities where case studies took place was pleased with the program and its new emphasis on community capacity building.
3. Do the design and delivery of Part II initiatives reflect service delivery principles and guidelines of the EI Act and LMDA?
a)To what extent have areas of joint cooperation minimized unnecessary overlap or duplication within and between governments?
Most informants were of the opinion that joint cooperation had minimized unnecessary overlap or duplication between governments, especially for Targeted Wage Subsidies and Self-Employment. A few, however, questioned to what extent harmonization has really occurred. In support of the latter position, there are still various programs to serve the self-employed in Nova Scotia.
b)To what extent have Part II initiatives been designed and implemented in partnership with others in the community? Have they led to development or enhancement of partnerships among various levels of government, employers and community groups?
Not surprisingly, given the flexibility of the measures, partnerships depend a lot on the particular community. In the Yarmouth area, the HRCC has been very proactive in developing community partnerships with community groups and employers. In other areas such as Antigonish and Sydney, HRCCs seem to be less proactive. HRCC staff in Antigonish do not go out and actively search out potential projects, but they do represent HRDC on a variety of community boards and committees where ideas for projects are often broached (section 6.3).
All three communities had examples of successful partnerships with employers and the community. HRCC Antigonish and Yarmouth have Regional Development Authorities that play important intermediary roles in the Local Labour Market Partnerships option. These organizations work with HRDC to identify partners that need help.
c)To what extent is there flexibility to allow significant implementation decision-making at the local level? To what extent does the use of Part II initiatives reflect the needs of the community?
A recurring theme throughout this evaluation has been the unprecedented level of flexibility at the local level. The strengths and weaknesses of the flexibility have a strong bearing on the strengths and weaknesses of the program. The flexibility enables each HRCC to respond to the needs of its community.
d) Are the Part II initiatives accessible in the official language of choice where demand warrants? Were those receiving services in a minority language satisfied?
This is not really an issue in Nova Scotia. Survey findings show that only 0.3 percent of EBSM clients spoke French only. Participants most comfortable speaking French were asked in the survey if they received services in the language of their choice. Eleven percent said no (but note this represents only three respondents).
4. What administrative measures have been put in place to collect information on participants, initiatives and third party providers? Is this information adequate for measuring project impacts? If not what improvements are necessary?
Overall, key informants were unsatisfied with the monitoring function of EBSM. Regional office and zone offices were worried about the lack of monitoring by HRCCs. HRCC managers were concerned about lack of monitoring of clients on work experience projects. HRCCs were also worried about their own ability to monitor external agencies under contract (section 4.2.4).
HRCC staff note that downsizing may have contributed to their inability to monitor agencies and clients. HRCC staff lack the time to visit employer or project sites. Some staff expressed concern about lack of time, and in some cases expertise, to provide financial oversight (to ensure money is spent appropriately).
According to virtually all interviewees in the zones and HRCCs, management information tools were inadequate. There is little timely information available about interventions or contracts. This complexity, along with the aforementioned lack of resources dedicated to monitoring, including data monitoring, may well be causing data quality problems.
5. a) To what extent do HRCCs have client targeting and selection? How are targeting and selection criteria determined? b) Are participants representative of these groups? c) To what extent were targets met? d) Are the criteria addressing the needs of the community? Does it result in gaps in services?
In most offices targeting and selection criteria exist but staff at some offices do not always adhere to the criteria in their effort to help all clients. It seems that the targeting and selection process is being reviewed and adjusted in some offices, while essentially being omitted in others (section 4.2.5).
Concerning gaps in service, most everyone was worried about those who are not eligible for EBSM because they have had no recent EI eligibility. Excluding people who are not on EI attached has large implications for the province. "The decision by the Federal government to get out of training and to narrow the legislation has been hard on Nova Scotia... Many Nova Scotians who are not eligible for EI will end up on social assistance."
Among the groups mentioned as "falling between the cracks" by informants: the social assistance client after three years; youth with low education levels; women returning to the labour market; the Black and disabled communities; and employers. Some offices have been able to provide some services through the Labour Market Partnership measure. But these groups cannot be funded for training programs.
6. Is Labour Market Information (LMI) available and accessible to support Part II initiatives and to provide information to clients? To what extent were clients satisfied?
Everyone interviewed thought LMI is available and accessible. A few interviewees mentioned that access is a problem for those with reading problems. And several warned that the information becomes outdated quickly, and there must be constant product development.
Clients surveyed gave LMI an average grade of B- for both written and computerized material (section 8.2).
7. What are the strengths and weaknesses of the EBSM delivery structure? Are the roles and responsibilities of various partners clear for planning, management, delivery and budgeting?
On the basis of interviews and focus groups, the evaluators concluded that flexibility is at once the primary strength and weakness of the delivery structure. HRCCs are happy with their ability to make local decisions. External service providers are also pleased with the discretion local HRCCs have to make decisions without waiting for signatures at every level. There is little doubt that flexibility makes the HRCCs more responsive to local needs.
But flexibility can also have its drawbacks. Rules help to ensure the intervention is delivered as designed. Without them, lack of consistency may be introduced between and even within zones. Some HRCC managers and staff confirmed that the lack of consistency among the HRCCs had contributed to some problems.
Many of those interviewed for this project indicated that the flexibility of EBSM and devolution of power to local staff could be a challenge to new workers and that even experienced workers need some parameters. (section 4.2.2).
9. How satisfied are participants with the benefits and measures provided under Part II initiatives? To what extent did the participants discontinue before the anticipated completion date? What were the main reasons for not completing the action plan?
Most participants (78 percent) thought the program was good or excellent. Few gave the program a failing grade (4 percent) or a D (5 percent). The mean overall grade was B (good). Clients in every EBSM component gave a mean overall grade of B, except for Employment Assistance Services clients who gave a B- (section 8.3).
Administrative data showed that about a quarter of EBSM clients had no Action Plan at any time, but most of those who did completed it. According to the data, 93 percent of EBSM clients successfully completed their intervention. No single reason predominated for not completing the intervention. Discontinuation was not a major issue in Nova Scotia.
Survey findings bolster these results. The vast majority of participants on job placements stayed on the job for the entire period of the placement: 84 percent for Targeted Wage Subsidies and 86 percent for Job Creation Partnerships. Only 11 percent of the Purchase of training trainees failed to finish their course with the reason given most often being they were still in the course (section 8.2).
10. To what extent does EBSM encourage participants to assume more responsibility in the development and implementation of their action plan? Did participants become involved in decisions concerning interventions? To what extent did they contribute to the cost of the intervention? Does it result in more commitment from participants?
According to HRCC staff, clients who develop action plans — normally high need clients who require upgrading and/or training — generally take the onus for developing and implementing their action plan. Many come into counseling with a good idea of what they want and, judging by the completion rate, follow through with their plans. Counselors ensure clients know relevant information about future employment prospects and help clients to identify their strengths and weaknesses, but otherwise participants have the primary role in deciding what their needs are.
The survey showed that 43 percent of Purchase of training respondents were determined to be able to contribute to the cost of their training: 40 percent actually did so. All but 5 percent of Self-Employment clients made an initial capital investment in their new business (section 8.2).
Nearly all informants reported that the client is much more committed to the plan when it is not being entirely financed by HRDC. They have found clients to be more motivated to succeed if they contribute to their intervention. Econometric results suggest that those who contributed to the cost of their intervention did better in terms of earnings than those who did not contribute.
11. To what extent have the Part II initiatives helped participants to reduce their dependency on EI benefits and/or social assistance? Obtain or keep employment? Increase their earnings? Create their own jobs? Have the primary targets in terms of employment been met?
The econometric analysis in Chapter 9 suggested that EBSM had no impact on reducing social assistance dependency; however there is some evidence that EBSM modestly reduced EI use, measured in terms of both weeks of benefits and total benefits received Participation in EBSM appears to have slightly increased time spent working — Targeted Wage Subsidies was most effective showing an increase of approximately 1-2 months in time spent working.
EBSM was shown to have no significant impact on earnings on most EBSM components.
About 70 percent of participants were employed for at least 12 consecutive weeks following participation in EBSM. Self-Employment and Targeted Wage Subsidies clients were most successful in this regard.
12. To what extent did the EBSM assist individuals to become self-reliant in career decision-making, skill enhancement, employment maintenance and job search?
Some 23 percent of EBSM participants had gone back to school, college or university since taking part in the program. By comparison, 24 percent of non-participants had taken further education or training since the start of 1997 (section 8.1). When one considers that the question covered a two-year period for all non-participants but 59 weeks for the typical participant, it appears that participants were more inclined to upgrade their skills. The impact analysis attributed a small increment — about 4 percent — to the program in the probability of being engaged in education or training at the time of the survey.
EBSM clients surveyed gave a B to the program for improving job skills, a B + for upgrading educational skills, a B- for improving their job search skills, and a B for helping them to develop a career action plan (section 6.8).
13. What EBSM characteristics/program components are associated with improved employability and earnings, further education, or self-employment? What activities/interventions were most effective? For what type of participant? For completers/non-completers? Why do some participants remain unemployed and on income support after the project?
The previous chapter presented the impact EBSM as a whole and for each component in terms of employability and earnings, further education, or self-employment (see highlights section of Chapter 9 for a synopsis).
The analysis looked at program impact on annual earnings by sex, visible minority status and disability status, and found nothing significant.
The reason most frequently offered for being unemployed was that there were no jobs available (see Chart 8.15 for the rest of the reasons).
16. For what reasons do employers participate? Do funding arrangements offer sufficient incentives to participate? What do employers like/not like about the program?
Employers who were interviewed indicated they participated for two main reasons: because they need extra help for a period of time; and to subsidize employees that need training on the job to carry out specific duties.
Employers were pleased with EBSM (section 6.5). The only area for improvement, in one employer's opinion, is better screening prior to sending the potential employee out.
19. What is the survival rate of Self-Employment (SE) sponsored businesses?
The survey found that 68 percent of SE supported enterprises were still in business at the time of the survey. Business development agencies interviewed reported a 76 percent success rate (section 6.7 and 8.2).
20. To what extent does SE result in direct job creation in addition to the self-employment of the participant?
About 70 percent of the new businesses employed only the Self-Employment participant when the survey took place. In total, Self-Employment businesses had generated 91 jobs — 51 of them full-time positions — besides providing employment for the entrepreneurs. This may understate the true job creation power of Self-Employment since the survey only reached 54 percent of Self-Employment clients. However, it could be assumed that the remainder of participants were not as successful since many had moved or were otherwise unreachable. Still it is likely that Self-Employment has produced over 100 direct jobs besides the owners' positions.
21. Who is the target group for Targeted Wage Subsidies? Do participants remain employed with employers after the subsidy has ended?
In theory, Targeted Wage Subsidies aims to provide job experience for those who have been unemployed for a long time or face special barriers to employment. HRCCs normally consider Targeted Wage Subsidies for moderate need clients who need more than Employment Assistance Services to get a job. The survey found that 62 percent of Targeted Wage Subsidies participants were offered a job after the placement: almost all accepted. However, only 35 percent of these clients were still with the subsidized employer at the time of the survey (section 8.2).
22. Is activity under Targeted Wage Subsidies incremental?
There is no direct evidence on whether employers would have hired someone in the absence of the subsidy.
23. What are the level, scope and type of activities taking place under Job Creation Partnerships?
With a start date on or after January 1, 1997 and an end date on or before June 30, 1998, there were 294 Job Creation Partnerships in Nova Scotia. Chart 8.8 indicates what types of occupations Job Creation Partnerships clients were working at while being subsidized. Job Creation Partnerships jobs were chiefly in construction, services or administration. This conforms to the type of projects funded through Job Creation Partnerships. Section 6.4 of this report gives several examples of Job Creation Partnerships projects.
24. To what extent have Job Creation Partnerships been successful in fostering partnerships? Who are the local community partners involved? What type of contributions are they making?
There is only limited, anecdotal information on the success of Job Creation Partnerships in fostering partnerships. The examples related in section 6.4 demonstrate that partners made contributions to some partnerships, perhaps an indicator of success. However, many employers were community organizations with limited resources for contributing financially (or keeping employees after the subsidy expires). Some 44 percent of the Job Creation Partnerships clients surveyed worked for community organizations. Another 24 percent of Job Creation Partnerships employers were public sector organizations. But 31 percent were private sector employers who may well have topped up the wages.
Approximately 38 percent of Job Creation Partnerships jobs survived the end of funding (section 8.2).
27. What occupations are training Purchases clients being trained for? How are training needs identified?
Chart 8.6 presents the occupation for which Purchase of training clients were training. Sciences, engineering and health, as well as construction were the most popular fields.
Clients often identify their own training needs according to HRCC counselors. Otherwise the counselor and/or various workshops help clients identify their needs.
28. Are clients finding employment in related occupations?
Asked how closely related their current job is to the job training they received, 58 percent of Purchase of training participants said very closely related, 17 percent said somewhat related and 26 percent said not at all related (section 8.2).
29. To what extent is Negotiated Financial Assistance used at the local level? How is it used? Are staff adequately prepared to use it?
Negotiated financial assistance is used across the province for clients who go on training interventions. The extent of the contribution is very much influenced by the individual's situation, with some not being able to contribute at all. The main reason given for being unable to contribute: they do not have the resources available. Offices vary in their approach to financial negotiations with some offices having a starting point on the negotiations (e.g. client can contribute 40 percent unless they can prove otherwise) while other offices do a financial assessment on the client to determine their ability to contribute to their training.
Although most staff who were interviewed had received training on financial negotiations, they acknowledge that there is considerable variation amongst staff on what is expected from the client.
30. To what extent have Purchase of training participants contributed to the cost of their intervention? Was the contribution appropriate to their circumstance?
Some 40 percent of Purchase of training participants contributed to the cost of their training. The average course contribution was $1,171; but this figure is skewed by nine clients who paid $4,000 or more for their course. The median contribution was $400 (section 8.2). Counselors would argue that the contribution is always negotiated and depends on the individual circumstances of the client.
32. To what extent are third-party organizations used to deliver employment services to clients? How are HRCCs managing Employment Assistance Services?
According to survey data, 67 percent of Employment Assistance Services clients received Employment Assistance Services from an HRCC, 18 percent from a third party, and 15 percent from both (section 8.2). This seems to be at odds with interview and focus group findings that most HRCCs contract out Employment Assistance Services to third party organizations.
33. Does the use of external service deliverers address issues related to capacity at the local level? Has Employment Assistance Services increased access? Has Employment Assistance Services resulted in contributions from other agencies?
The main reason given for using external service deliverers was the reduced capacity of HRCCs to carry out their tasks due to less staff (section 4.2.6).
Employment Assistance Services is a component mostly delivered by external agencies, in particular services such as teaching job search skills and resume writing. Whether it has increased access is an open question: it has helped maintain access that otherwise might have been forfeited in the face of reductions in resources and staff.
The case study reported examples of contributions from external agencies as a result of an Employment Assistance Services contract: mostly in-kind contributions like office space and equipment. A good example is the Human Resource Services Centre at Burridge Campus of the Nova Scotia Community College which delivers Employment Assistance Services to clients in the Yarmouth area. The Centre does all the information group sessions, and Needs Determination interviews with the clients. They do some of the counseling and case management as well. The campus provides the facilities and access to other college resources such as the learning centre.
34. What are the level, type, and scope of activities taking place under Local Labour Market Partnerships? Who are the partners involved?
There were 221 Local Labour Market Partnerships in Nova Scotia with a start date on or after January 1, 1997 and an end date on or before June 30, 1998. Local Labour Market Partnerships money has been used to fund a wide variety of projects including: hiring coordinators for groups serving the EBSM target groups, financial support for career fairs, creating community economic development advertising materials, funding research to support community development, and financial support for other regional development activities. Partners include Regional Development Authorities, community agencies, employers, chambers of commerce, and governments. Section 6.3 includes examples of Local Labour Market Partnerships used at case study sites.
35. What are the objectives of Local Labour Market Partnerships? How are they established? To what extent do they meet local labour force priorities?
The objectives of Local Labour Market Partnerships are to support community efforts to identify labour market needs, identify the capacity of the community to meet those needs, work with partners in building "community capacity" to address local labour market needs, and minimize duplication and gaps in service. As such, Local Labour Market Partnerships fund research, community planning, and innumerable other activities in the name of community economic development.
Antigonish and Yarmouth have Regional Development Authorities that play important intermediary roles in the Local Labour Market Partnerships option. These organizations work with HRDC to identify partners that need help.
36. How are sharing of costs, mandate, mutual responsibilities and accountability for results determined? Are partners satisfied?
Local Labour Market Partnerships are negotiated with the partners. Local Labour Market Partnerships are usually handled by the project officers in the HRCCs. Third parties are generally required to submit quarterly reports to the HRCC for monitoring purposes. Partners were satisfied with the Local Labour Market Partnerships and with the HRCC (section 6.3).
|
|
| Last Modified: 2002-11-29 | Important Notices |
![[To Top]](/web/20060127000311im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}