Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Labour Market Development Agreements

Formative Evaluation of
Provincial Benefits and Measures
under the Canada/New Brunswick
Labour Market Development Agreement

Final Report

Canada/New Brunswick LMDA
Joint Evaluation Committee

July 1999

SP-AH111-07-99E
(également disponible en français)

The Formative Evaluation of the Canada/New Brunswick Labour Market Development Agreement was conducted by Ekos Research Associates Inc. and managed by a Joint Evaluation Committee composed of representatives of Human Resources Development Canada (HRDC), the New Brunswick Department of Labour (formerly the New Brunswick Department of Advanced Education and Labour (AE&L) and Human Resources Development New Brunswick (HRDC).

The evaluation team would like to thank all those who contributed to the study, especially officials and employees in both the federal and provincial governments who gave of their time and experience to assist the evaluation team.

We would also like to thank the many employment insurance clients, employers, unions and service providers who generously shared information about their experience with the Provincial Benefits and Measures.

On December 13, 1996 the Canada/New Brunswick Labour Market Development Agreement (LMDA) was signed, thus enabling the government of New Brunswick to assume responsibility for the design and delivery of active employment benefits and measures identified in Part II of the Employment Insurance Act. The Agreement also provided for the establishment of Canada/New Brunswick Human Resource Service Centres through which New Brunswickers have access to a range of federal and provincial labour market programs and human resource development services.

Provincial Benefits and Measures (PBMs) provided under the LMDA are supported by Employment Insurance funds and are available to EI clients within the province. The specific benefits that New Brunswick is providing include: Partners, a wage subsidy program; the Entrepreneur Program, which provides assistance to EI clients who wish to start their own businesses; Job Action, work experience for EI recipients in support of obtaining long-term employment; Skills Loans and Grants (SLG), a program to support EI clients pursuing training or education; and the Earnings Supplement Program, based on findings from ongoing national pilots and which is not yet available.

Provincial measures provided under the LMDA include: the Adjustment Services Initiative, which delivers labour market partnership activities which support research, planning and other activities by community partners or industry to address human resource development challenges and opportunities; Employment Assistance Services, which is provided to all unemployed individuals and provided by third party contracts; and Research and Innovation, whereby the province will conduct activities, experimentation and research to address labour market development, policy and design issues (the Rural Experience program is currently funded under Research and Innovation).

The objectives of the overall evaluation of the LMDA are as follows:

• to measure the extent to which PBMs and the supporting infrastructure have been successful in achieving the objectives of the LMDA;
• to provide information to managers, policy makers and program designers on a number of program issues;
• to estimate the cost-effectiveness of the interventions; and
• to demonstrate "what works best" and "lessons learned".

The evaluation of the LMDA consists of two major components, a formative and summative evaluation. The summative evaluations are designed to measure outcomes, impacts and cost-effectiveness of programs, projects and interventions. The purpose of the formative evaluation, the subject of this report, is to supply information indicating what improvements, if any, are required to the LMDA design, delivery and supporting infrastructure that would permit it to better meet its objectives.

Rationale

The PBMs' focus on moving clients toward self-sufficiency means that they are generally compatible with the intent of the EI Act and the LMDA.

• There is a need to improve the co-ordination of programming delivered by HRD-NB and the Department of Labour in order to better link workers with employers.
• There do not appear to be any major problems of PBMs and other existing provincial and federal programs duplicating services or working at cross-purposes with one another.

The PBMs have sufficient flexibility to be adapted to client needs, they are providing helpful assistance to clients, and they are focused on helping people get back to work.

• Respondents expressed some concerns about clients in need who "fall through the cracks", including youth, persons with disabilities, people who are under-employed or who have had very little steady employment, and small- and medium-sized businesses in the midst of cut-backs.
• Many feel that it is the EI Act (and the definition of "EI clients"), rather than the LMDA, which renders some of these people ineligible for assistance.

Design and Delivery

There is still some confusion and discomfort over the new roles and responsibilities of the various players, such as a lack of a shared understanding of the new roles and responsibilities among staff and different levels of government. Much of this confusion appears to be due to inadequate communications both within and between levels of government.

Several aspects of the LMDA implementation were regarded as successful, including:

• the smooth transfer of funds to the Province;
• the transfer of HRDC staff to the Province (including staff communications, sound change management and careful attention to transferred staff's package in terms of money and protection);
• good co-operation among the three players and two levels of government;
• a willingness to work together in the field, through local implementation teams and regional workshops, which greatly facilitated implementation; and the maintenance of good program delivery and client service throughout the implementation process.

Less successful aspects of the implementation process include:

• disagreement between federal and provincial governments with respect to resources for implementing the LMDA;
• confusion and frustration among the partners due to unresolved details of the LMDA resulting from the speed with which the Agreement was signed;
• the lack of a dedicated project manager for LMDA planning, co-ordination and implementation to oversee and co-ordinate the overall LMDA implementation, troubleshoot and facilitate communications between the three players;
• inadequate information and monitoring systems and poor integration of the systems operated by the three players, which have impeded proper planning and management of the LMDA; and
• confusion for the public due to the fact that three distinct events happened simultaneously — the passing of the EI Act, the reduction of the federal government's consolidated revenue fund (thereby significantly reducing the funds available for non-EI clients), and the devolution of programs to the Province through the LMDA.

Although impeded by these factors, joint implementation planning has been attempted at both central and local offices, and this has been helpful in terms of sorting out roles and responsibilities. Further, given the complexity of the task of implementing the PBMs and the LMDA, there has been remarkably little disruption to client service in the view of many respondents, with the exception of the delay in implementing the loan component of Skills Loans and Grants and delays in co-location. However, the LMDA implementation process has been somewhat disruptive for some staff. For instance, federal employees who were transferred to the province feel isolated and confused, and are having difficulty adapting to their new environment and many staff continue to feel the need for clear communications regarding the PBMs and their new roles and responsibilities.

In terms of partnership and co-operation, the LMDA has provided an opportunity for the partnerships among HRDC, HRD-NB and DOL to become stronger. Further, consultations have been made with the New Brunswick Labour Force Development Board (regarding LMDA implementation), and with community organisations, though some respondents in the evaluation felt that these could be improved.

With respect to adapting information systems for the monitoring of program delivery and results, there appears to be a need for:

• clarification on accountability requirements and who is responsible for what results;
• a clear definition of valid results measures;
• better integration of the (currently incompatible) information systems operated by the three players;
• a resolution of client privacy issues so that client information can be shared among the players; and
• a resolution of issues involving shared access to LMI and Electronic Labour Exchange systems among the three partner departments.

Turning to co-location, the impacts to date can be summarized as follows:

• Service delivery has either improved or stayed the same. Improvements include: greater convenience of one-stop service; better resource centres for the provision of Labour Market Information; faster processes in approving some DOL programs; and better service for social assistance clients.
• Clients are generally satisfied with the one fully co-located site and like the convenience, but they do not perceive a significant difference in the service they receive.
• At partially and fully co-located sites, most staff did not feel that client service has been altered, but some felt that it had declined due to poorly conceived reception services, impeded client flows, and an increase in the number of required client contacts as well as conflicting departmental operations.

Co-location has not led to poorer service compared to when the federal government delivered these types of services, but neither has it created integrated information services and seamless service delivery. Staff have been frustrated and disappointed because their expectations that co-location would increase interdepartmental communication and increase access to information have not materialized.

In order to improve the delivery of PBM services through a co-located approach, the three partner departments will need to address the following issues identified in the evaluation:

• the harmonization of departmental operations and service methods;
• the provision of reception services that serve all three departments as well as the different target client groups;
• shared responsibility for administrative costs;
• re-examination of the information needs of each department and improved information exchange;
• improved interdepartmental communications; and
• the establishment of a streamlined management structure that can resolve problems quickly and that represents the interests of an integrated HRSC.

Considering service delivery, the qualitative evidence suggests that the programs are believed to be meeting clients' needs and to have sufficient flexibility and capability to adapt to local needs. The high degree of local flexibility in the delivery of SLG has however raised some concerns about consistency between case managers and regions. In addition, the promotion of all PBMs could be improved.

The results of the employer and participant surveys were consistent with the qualitative components of the evaluation. These results show that, overall, respondents who have participated in LMDA programs tend to be fairly satisfied with the programs, although there is room for improvement in several program areas. Based on employer and participant suggestions, the largest improvements to the programs could be made through better promotion and advertising, increased relevance to the needs of employers and clients, broader eligibility for programs, and more flexible regulations associated with their use. Employers also found that the quality of program participants, the programs themselves, and service received from the government have improved slightly since the LMDA was implemented.

Success to Date

Target Attainment

One measure of success is the extent to which results targets set for the LMDA for the 1997/98 fiscal year were met.

• First, 1998 EI Monitoring and Assessment Report submitted to HRDC found that only 70 per cent of the target of 7,947 PBM participants being returned to work was attained. This figure, which is based on administrative data, is believed to underestimate actual returns to work.¹
• Second, the 1998 EI Monitoring and Assessment Report found that roughly only 50 per cent of targeted savings $25.9 million from unpaid EI claims, due to early returns to work, were achieved. Our computations indicated that there were $11.3-14.1 million in unpaid EI benefits for the 1997/98 and 1998/99 fiscal years associated with the interventions under study.
• Third, active EI claimants account for 56 per cent of individuals who have participated in the benefits under study during the 1997/98 fiscal year. The percentage for participants during the part of 1998-99 considered in this evaluation was 41 per cent and overall for the two years was 47 per cent. Though this figure cannot be compared to the targeted proportion which is set for the full set of benefits offered, there is reason to believe that active claimants served is below target.

Client Outcomes

Evidence gathered in this evaluation enabled us to assess the success to date, from the perspective of clients, employers and communities. As noted active EI claimants account for less than half of the individuals who have participated in the benefits and measures. The shortfall was most apparent with respect to Entrepreneur participants. It should also be noted that SLG participants represent more than half of all individuals who have accessed benefits and measures.

Finally, note that the measures of impacts discussed here should be considered preliminary only and that more definitive measures will be presented in a summative evaluation when there will be more time available to detect impacts, particularly for interventions for which the expected time horizon until employment tends to be long.

Employment

The study findings show that, overall, LMDA programs particularly Entrepreneur and Partners do benefit from positive employment outcomes, absolutely and relative to the comparison group.

• Over one-half of participants with jobs at the time of the survey stated that their participation was important to attaining those jobs.
• Over 60 per cent of participants were employed at some time following their intervention. Particularly high percentages were reported for Entrepreneur and Partners. Claimants reported higher percentages than reachbacks and the comparison groups.
• Stability of post-intervention jobs was related to program type (Entrepreneur and Partners tended to be more stable), age of worker (older workers were the most likely to occupy longer lasting jobs), and claimant status (EI claimants tended to occupy longer lasting jobs than reachbacks).
• Claimants were more likely to occupy jobs lasting 12 consecutive weeks than the comparison group.
• Entrepreneur and Partners participants were most likely to be employed both in the week after the intervention and at the time of the survey.
• Job Action and Rural Experience participants experienced the highest levels of unemployment in the post-program period, at about 50 per cent.
• Generally, gains were made over the post-program period (i.e., between the first week and the time of the survey) and participants in most programs experienced increases in employment incidence and school attendance and declines in unemployment incidence.
• While comparison group members were more likely to be currently employed than EI claimant participants (which was the only employment measure by which the comparison group exceeded EI claimants), the incidence of full-time employment and the proportion going to school was higher in the latter.
• One-quarter of those unemployed and those not in the labour force before the intervention found full-time jobs and close to 10 per cent found part-time jobs after the intervention. One-fifth of pre-intervention part-timers changed their status to full-time.

Job Quality

On most job quality measures, EI claimants did somewhat better than the comparison group.

• The vast majority (almost 90 per cent) of employed participants were working more than 30 hours a week in their current job. Weekly hours worked were highest for Entrepreneurs.
• About half of participants occupied year-round jobs. The incidence of casual, contract and seasonal employment was highest among Job Action, Rural Experience, and EAS participants.
• Comparison group members were more likely to be in year-round jobs and less likely to be in casual/contract employment than EI claimants.
• Weekly pay was highest for SLG participants and lowest for Job Action participants.
• EI claimants earned somewhat more than reachbacks, reflecting the relative work experience of the two groups.
• Participants earned somewhat more than the comparison group, implying some advantage for those making use of PBMs compared to other non-LMDA employment assistance.

Retention

Just under one-half of wage subsidy participants were retained by their host employer, with Partners participants particularly likely to be hired on, as expected.

• Notable reasons for not hiring workers after the wage subsidy included a lack of resources and the temporary or seasonal nature of the work.
• One-third of employers "lost" hires before the end of the wage subsidy. The most frequently mentioned reasons for losing hires were, in order, that they quit before the end of the subsidy, a lack of amenity to training, poor attitude, incompetence, personal problems and finding another job.

Incrementality

Over 70 per cent of wage-subsidy employers said they would have left the position vacant had there been no wage subsidy (complete incrementality).

Joblessness and Job Search

Participants on average experienced joblessness for about 30 per cent of the time following their intervention.

• The mean was greatest for Rural Experience and Job Action participants (36 and 43 per cent) and lowest for Entrepreneur participants (about eight per cent).
• EI claimants experienced less joblessness than reachbacks, reflecting work history, and than the comparison group, implying some advantage for provincial benefits and measures.

Entrepreneur participants looked for work for the least amount of time while unemployed, though they were unemployed for the lowest percentage of time following their intervention. At the other extreme were Job Action and Rural Experience participants who were most likely to look for work while unemployed.

• Reachbacks looked for a somewhat longer period of time than EI claimants.
• Comparison group members tended to look for work for an even longer period of time while unemployed (relative to EI claimants) implying some advantage in favour of the PBMs.

The type of job search activity pursued varied by program participant.

• EAS participants were most likely to send out resumes and rely on newspapers than participants in other PBMs.
• Rural Experience and Job Action participants tended to rely on word of mouth to find jobs.

Utilization of Income Support

About one-quarter of participants entered a new spell of EI benefits following their intervention.

• Entrepreneur participants and, to a somewhat lesser extent, participants in SLG and EAS did best in this respect, in terms of both incidence of EI receipt and per cent of post-intervention weeks receiving EI.
• Job Action and Rural Experience participants fared the poorest in this respect.
• Claimants were less likely than reachbacks to be on EI.
• However, non-participants were only a little more likely to be on EI than participants.

Only about one-tenth of participants received SA benefits after their intervention.

• The highest incidence being reported among Job Action and EAS participants, who also were on it for the longest periods of time.
• Reachbacks were more likely to have been on SA than EI claimants, but comparison group members were less likely.

Attitudes and Skills

Qualitative and quantitative evidence indicates that participants' experience in the PBMs increased their skills and enhanced their attitudes to work and learning.

• Specific psychological outcomes identified included confidence, a feeling of accomplishment and dignity, and an appreciation of the benefits of contributing to one's own human capital development.
• From a labour market perspective, participants benefited from increased labour-market intelligence, a strengthened ability to set career goals, and increased job-specific job-search skills.
• Between 75 and 89 per cent of wage-subsidy employers believed that participation in the programs benefited clients with respect to their attitude to learning and training, job-specific skills and preparedness and job readiness.
• Six in 10 employers thought that participants were job ready following the end of the wage subsidy.

About eight in 10 employers provided job-specific and orientation training to PBM participants, which is true of participants in all wage-subsidy programs.

• About one-half provided task rotation, personal skills training, mentoring and job shadowing to their hires.
• Computer training and career counselling were the least frequently provided (particularly in Rural Experience) as was job-search advice (particularly in Partners).

Employer Outcomes

The focus groups and case studies indicated that employers benefited from their participation in the wage subsidy programs.

• Increased sales resulted from the two Adjustment Service Initiative (ASI) up-side adjustment situations studied for this evaluation.
• Other benefits included increased management and marketing knowledge, increased confidence in the company's direction, a business plan and marketing strategy, and increased awareness of business-support services available.

Results from the employer survey indicated that participation in the wage-subsidy programs benefited their organizations.

• Over eight in 10 employers thought the wage subsidies had a positive impact on their organisation overall.
• The widest gap in reported impacts occurred between Partners and Rural Experience employers in terms of the organisation's ability to evaluate new employees (85 versus 62 per cent).
• The extent that positive impacts were perceived declined with organisation size.

The areas where participating employers reported they had incurred significant costs were in supervision/training and compensation (39 and 31 per cent of employers) and benefits paid (particularly Partners employers).

• Wages were most frequently mentioned as a significant cost by private sector employers.
• Over 40 per cent mentioned no significant costs, with non-profit organisations most likely not to do so.

Community Outcomes

There was varying opinion on the extent to which the benefits and measures had benefited the community. Key informants were not able to detect many impacts.

• Some employers participating in the focus groups thought that the programs resulted in enhanced infrastructure for the community, which will benefit it in the long run.

In the two case studies of ASI community development adjustment experiences, it was found that communities did benefit from the ASI's committee's work.

• In both communities, a community action or strategic plan was implemented and morale was increased as a result of the projects.
• In one, the plan resulted in real benefits for the community (i.e., purchase of land for an industrial park and negotiations with prospective employers).
• In the other, the plan strengthened the sponsoring community organisation's capacity to be more financially self-sufficient and thus to place greater numbers from the community into jobs.

Modelling Results-Client Outcomes

Econometric modelling was carried out for three sets of client outcome measures: employment, earnings and income support use. In this analysis, the outcome measures were explained in terms of the interventions while controlling for the time since the intervention, sociodemographic and employment history characteristics, and service-delivery use. Separate analyses were run for specific sex, age and claimant status segments. The latter include active EI claimants and "near-reachbacks" who are former EI claimants whose claim ended within six months of the intervention in question.

The results of the interventions should be considered in the context of the formative evaluation in which they were measured. The interventions address differing needs and may be expected to have differing "gestation" periods, i.e., periods of time over which impacts would be detected. Entrepreneur and Partners, which in many cases lead to immediate employment, would be expected to have more favourable employment results in a formative evaluation than other interventions which do not have immediate employment outcomes and which deliver training and other assistance to persons whose needs are such that the expected time of entry into the labour market would be further down the road.

• Partners — Controlling for other factors, this program had a significant positive impact on all employment outcomes, overall; this was generally true for all individual segments but active EI claimants and except for percentage of weeks working, where no segment was affected. Further, it had no impact on the weeks looking for work as a percentage of the weeks since the intervention, overall and in any segment. Partners contributed positively to current weekly earnings levels and their percentage growth from before to after the intervention, overall and for all segments but older and EI claimants. As for absolute earnings growth, Partners had no effect overall on it, but did increase it for just males. Finally, it reduced weeks of post-intervention EI receipt, overall but only for females, younger participants and near-reachbacks, and reduced the likelihood of receiving SA in the post-intervention period only for younger participants.
• Entrepreneur — Controlling for other factors, this program had a significant positive impact on all employment outcomes, overall and for every segment except for claimants in the case of 12 consecutive of weeks of employment. It reduced the length of job search in all segments but near-reachbacks. As well, Entrepreneur increased all three earnings measures, overall but generally only for males, younger participants and near-reachbacks. Further, it had a significant negative impact on the relative duration of EI receipt, overall and in all segments but EI claimant participants, but had no impact on SA receipt.
• Job Action — Controlling for other factors, the only employment impact this program had was a significant positive impact on being full-time employed just for older participants. It had no impact on relative post-intervention job search except to increase it among younger and claimant participants. There were no earnings or income-support dependence impacts found.
• SLG — Controlling for other factors, among all participants and for every segment, this program had no significant impact on all employment and job search outcomes, except for a positive impact on weeks working as a percentage of weeks since the intervention for females only. SLG positively affected all earnings measures, overall and for all segments but active EI claimants. SLG reduced post-intervention EI receipt, overall and for all segments but males and claimants. It had no impact on SA use.
• EAS — Controlling for other factors, this program had a significant positive impact on just two employment outcomes: currently employed (overall but only for the younger and older segments) and currently full-time employed (overall and only for the older segment). No significant impact was detected on the other two employment outcomes (except for a negative impact on weeks working for males only), the job search outcome, and the earnings outcomes. EAS was found to reduce post-intervention weeks of EI receipt, overall and for all segments but active EI claimants. No SA impact was detected overall, but EAS did increase the chances of SA receipt for males.
• Rural Experience — Controlling for other factors, this program had no significant effect on any employment outcomes. No impact was detected for the job search and earnings outcomes. Finally, Rural Experience increased post-intervention use of EI overall, and in all segments except for females and near-reachbacks. No impact was detected for SA use. Further, it is worth noting that the program's negative effects on currently full-time employed and change in current weekly earnings and its positive effect on SA receipt (with just program variables in the model) were nullified by the introduction of background controls, implying that the program's negative impacts may be compensated for by the characteristics of participants.

Lessons Learned

The following is a preliminary overview of the major lessons that were learned over the course of the formative evaluation. The major lessons were:

• The evidence from most data sources in the study suggests that the success of the LMDA is highly dependent on the degree to which activities were planned and co-ordinated prior to their implementation. Specific areas in which planning and co-ordination of effort were considered to be most important include:
  • Estimation of the schedule and costs of co-location in consultation with local authorities;
  • Plans for interim arrangements for service delivery prior to co-location;
  • The co-ordination of service delivery efforts within and among government departments;
  • Consistency in the application of programs;
  • Central project management to oversee and co-ordinate all aspects of the LMDA implementation; and
  • Proper attention to change management for government staff, as many felt isolated and confused with the change in roles and responsibilities resulting from the LMDA.
• A major issue for many key informants involved in the delivery of services under the LMDA concerned a lack of communication and team building within and between departments.
• Overall, programs and services were working quite well to meet the needs of individuals, employers and the community and clients tended to be satisfied with the services received.
• All programs and services are being delivered in both official languages.
• Much evidence suggests that many people involved in the design, delivery and receipt of programs and services believe the eligibility criteria should be broader.
• Existing data systems are not adequate to properly track and monitor the impacts of LMDA programs, and the attainment of targets.

Recommendations

A review of findings from the formative evaluation suggests that a number of concrete steps can be taken at this point in the implementation of the LMDA to improve the effectiveness of the overall LMDA infrastructure. These recommendations are as follow:

• An overall internal communications strategy should be developed to address intra and inter-departmental concerns. Such a strategy would need to encompass issues related to clarifying the respective roles and responsibilities of both staff and LMDA partner departments, as well as changes in service delivery that have resulted from the implementation of the LMDA.
• Decisions around implementation issues of mutual concern to all partner departments will need to be made. These decisions would address issues related to the promotion of LMDA programs and services, as well as other service delivery issues, such as reception and appropriate signage in HRSCs.
• The responsibility for serving the Aboriginal clientele, especially those living on reserves, needs to be clarified between federal and provincial partners.
• With respect to promotion, an external communications strategy needs to be developed to raise awareness among active EI claimants, whose participation in the benefits and measures was well below target.
• Further qualitative research needs to be conducted into the reasons for the high uptake among reachbacks, particularly in the Entrepreneur program. The research will identify whether the high uptake can be attributed to actions on the part of client service officers, to a larger than expected demand for assistance among reachbacks, or to some other factor.
• Changes are required to ensure that programs are properly targeted to clients, as evidenced by the smaller than expected proportion of EI claimants who participated in LMDA programs. Related to this, some evidence suggests that EI clients were not being referred to PBMs. All this suggests that the lack of results target attainment may be related to issues of service delivery, inter-departmental communication and inappropriate targeting.
• Greater integration of information systems maintained by the three government departments would allow for proper monitoring and results tracking, thus greatly facilitating the management of program and service delivery, as well as the subsequent determination of impacts at the provincial and local levels.
• Greater co-ordination of program delivery among all three government departments would enhance service delivery and improve the reach of LMDA programs. These efforts would cover such areas as hours of operation and client referrals between departments.
• The appointment of a dedicated project manager to oversee the LMDA implementation (including overseeing action on these recommendations) would minimize further difficulties that may arise throughout the remainder of the implementation process. Among the potential benefits of this appointment, such a manager could facilitate communications between the three departments, troubleshoot further difficulties that may arise and work to establish clear guidelines for many of the implementation issues that were not resolved prior to the signing of the LMDA (i.e., the establishment of respective responsibilities for administrative costs).
• To address concerns over inconsistency across service delivery sites, the different sites should be encouraged to exchange views and experiences through e-mail, a special constructed Internet site or some other means.
• Further research will be required, particularly in the summative evaluation, in order to make a more definitive assessment of the relative impacts of different programs on employment and other outcomes. It would be premature to make recommendations about specific programs based on the short-term results of this formative evaluation. The main reason is that the PBM programs target different employability needs and would thus be expected to require different lengths of time before the full impacts are felt. For instance, the observed positive outcomes for the Partners and Entrepreneur programs in this formative evaluation are not surprising given that both these programs target career decision-making using a job placement strategy typically leading to immediate employment outcomes. Conversely, the limited impact of other interventions such as SLG may be due to the fact that they target an employability need (e.g., skill enhancement) necessarily requiring a longer time horizon to realize and detect labour-market impacts.

Recommendation 1

An overall internal communications strategy should be developed to address intra and inter-departmental concerns. Such a strategy would need to encompass issues related to clarifying the respective roles and responsibilities of both staff and LMDA partner departments, as well as changes in service delivery that have resulted from the implementation of the LMDA. One possibility may be to develop a single, automated, user-friendly information system on new programs to assist front-line staff.

Response

A Communications Committee consisting of representatives for all departments was established at the direction of the Implementation Committee. A draft strategy was completed. The three departments recognize the importance of having an overall internal communications strategy, and will take the necessary steps to finalize and implement (e.g. by using existing technologies). Furthermore, we are holding a workshop for field Directors in October 1999, which will focus on improving communications and client service. This will also aid in further clarification of respective roles and responsibilities.

Recommendation 2

Decisions around implementation issues of mutual concern to all partner departments will need to be made. These decisions would address issues related to the promotion of LMDA programs and services, as well as other service delivery issues, such as reception and appropriate signage in HRSCs.

Response

Decisions regarding issues of mutual concern are resolved at the local level. Those issues, which cannot be resolved by the Local Implementation Committees, are forwarded to the Senior Implementation Committee. Decisions have been made with respect to reception and signage.

Recommendation 3

The responsibility for serving the Aboriginal clientele, especially those living on reserves, needs to be clarified between federal and provincial partners.

Response

Service delivery issues for Aboriginal clientele, including those living on reserve, have been dealt with in a cooperative manner by provincial and federal representatives, and by the aboriginal community. All partners have made efforts towards clarifying roles and providing service in a coordinated manner. For example, the Aboriginal Human Resource

Development Strategy and the Aboriginal Employment Strategy have been implemented to address the needs of aboriginal clientele.

Recommendation 4

With respect to promotion, an external communications strategy needs to be developed to raise awareness among active EI claimants, whose participation in the benefits and measures was well below target.

Response

A marketing plan has been developed which is specifically aimed at active EI claimants, to raise awareness of employment services and to promote the importance of early intervention. This will be implemented in Autumn 1999.

Recommendation 5

Further qualitative research needs to be conducted into the reasons for the high uptake among reachbacks, particularly in the Entrepreneur program. The research will identify whether the high uptake can be attributed to actions on the part of client service officers to a larger than expected demand for assistance among reachbacks, or to some other factor.

Response

Clients typically receive assistance during the latter part of an EI claim. As stated in response 4 above, we are making efforts to assist clients earlier in their claim. The nature and time required in the Entrepreneur approval process may contribute to the observation of a higher percentage of reachback clients. An analysis of the group seeking service and the reasons for doing so, may be beneficial in delivering service.

Recommendation 6

Greater integration of information systems maintained by the three government departments would allow for proper monitoring and results tracking, thus greatly facilitating the management of program and service delivery, as well as the subsequent determination of impacts at the provincial and local levels. Related to this, there is a need for clarification of accountability requirements and who is responsible for what results, for a clear definition of valid results measures, as well as for a resolution of client privacy issues to improve information sharing among the LMDA partners.

Response

The conceptual design of case management for EI and reachback clients is complete. The province expects to have a single case management system (NBCASE) by the Spring of 2000. The system integration will allow for a streamlined and comprehensive management of client case/action plans, regardless of income source. Work is also being done in the development of the EDTS (Employment Development Tracking System).

This will provide an opportunity to better link client needs with program activity. This was also identified in our recent policy review, and we are in agreement to explore further. The three partners have committed to and have undertaken activities to improve our understanding of the accountability requirements. During fiscal year 1998/1999 improvements were recorded. A letter of understanding regarding the sharing of EI client information has been finalized and this will result in a formal change to the agreement. This has resulted in more effective targeting of EI clients.

Recommendation 7

Changes are required to ensure that programs are properly targeted to clients, as evidenced by the smaller than expected proportion of EI claimants who participated in LMDA programs. Related to this, some evidence suggests that EI clients were not being referred to PBMs. All this suggests that the lack of results in target attainment may be related to issues of service delivery, inter-departmental communication and inappropriate targeting.

Response

As stated in response 4 above, a marketing plan is being developed to inform EI clients of the benefits of seeking assistance earlier in their claim. A letter of understanding regarding the sharing of EI client information has been finalized and this will result in a formal change to the agreement. This has resulted in more effective targeting of EI clients. A number of initiatives in field offices are in place to ensure EI clients are informed of available services.

Recommendation 8

Greater coordination of program delivery among all three government departments would enhance service delivery and improve the reach of LMDA programs. These efforts would cover such areas as hours of operation and client referrals between departments.

Response

An inter-departmental task group was established in January 1999 to examine issues regarding LMDA program delivery. A document entitled "Policy Issues Action Plan" was drafted and supported by all three departments. This resulted in the establishment of a number of working groups which were charged with examining program issues such as, programs for disabled, youth, and students; a review of training and wage subsidy programs; and a review of our respective planning processes. Work has been completed in some areas, in particular in the area of a collaborative planning process.

Recommendation 9

The appointment of a dedicated project manager to oversee the LMDA implementation (including overseeing action on these recommendations) would minimize further difficulties that may arise throughout the remainder of the implementation process. Among the potential benefits of this appointment, such a manager could facilitate communications between the three departments, troubleshoot further difficulties that may arise, and work to establish clear guidelines for many of the implementation issues that were not resolved prior to the signing of the LMDA (i.e., the establishment of respective responsibilities for administrative costs).

Response

Key people from each department have been identified to oversee and manage the LMDA implementation and provide the links between field staff and senior officials.

Recommendation 10

To address concerns over inconsistency across service delivery sites, the different sites should be encouraged to exchange views and experiences through e-mail, a special constructed Internet site or some other means.

Response

We agree that inconsistencies across Service Delivery Sites are of concern. However, we believe the collaborative efforts of many joint working groups, such as Local Implementation Committees, SLG Consultation group and the upcoming Directors workshop, will address the concerns. This was also identified in our recent policy review, and we are in agreement to further explore.

Recommendation 11

Further research will be required, particularly in the summative evaluation, in order to make a more definitive assessment of the relative impacts of different programs on employment and other outcomes. It would be premature to make recommendations about specific programs based on the short-term results of this formative evaluation. The main reason is that the PBM programs target different employability needs and would thus be expected to require different lengths of time before the full impacts are felt. For instance, the observed positive outcomes for the Partners and Entrepreneur programs in this formative evaluation are not surprising given that both these programs target career decision-making using a job placement strategy typically leading to immediate employment outcomes. Conversely, the limited impact of other interventions such as SLG may be due to the fact that they target on employability need (e.g., skill enhancement) necessarily requiring a longer time horizon to realize and detect labour-market impacts.

• Based on the observed short-term impacts from this formative evaluation, it is possible to articulate certain hypotheses regarding possible modification to certain interventions that may improve their effectiveness but which would require further research to substantiate. For instance:

  • the finding that self-serve products led to negative outcomes from men and younger participants may simply reflect the fact that the use of such products distracted these groups from seeking more intensive employment assistance.

    Further analyses may be able to shed light on the true impact of these products and suggest ways to improve the promotion of programs and/or referrals among LMDA partners; and

  • the poor showing the Rural Experience and Job Action relative to Entrepreneur and Partners in terms of employment outcomes is not surprising given that the former are designed to provide only short-term employment experience. Nonetheless, the short-term nature of these job placements may encourage recurring use of EI. Further research on longer-term impacts may be able to shed light on the extent to which this may be occurring.

Response

The Joint Evaluation Committee is responsible for establishing the statement of work for the summative evaluation, and will take these points into consideration.

This document presents the results of the formative evaluation of the Canada/New Brunswick Labour Market Development Agreement. Chapter One provides information on the context for the evaluation. Chapter Two presents an overview of the methodology utilized in undertaking this evaluation and the remaining chapters provide the key results.

1.1 Background

The Employment Insurance Act

The Employment Insurance (EI) Act was implemented in July 1996. Part I of the EI Act provides for EI benefits for people temporarily out of work, and Part II of the EI Act involves active employment benefits and measures to help the unemployed return to work. Together, these two parts form a balanced and integrated framework designed to promote employment growth.

Part I of the EI Act maintains the national system of temporary income support for EI claimants while they look for a job. The Government of Canada, through Human Resources Development Canada (HRDC), will continue to be responsible for providing Employment Insurance income support and for delivering labour market development programs consistent with national interests.

Part II of the EI Act provides for a range of active employment benefits and measures that assist people in returning to work as quickly and efficiently as possible. These measures, which can be tailored to the needs of individuals and local labour market realities, are intended to provide unemployed Canadians with opportunities to obtain and maintain employment, and to be productive participants in the labour force. In accordance with the Canada/New Brunswick Labour Market Development Agreement (Cda/NB LMDA), the Province is responsible for designing and delivering programs that have goals and objectives similar to the active employment benefits and measures identified in the Employment Insurance Act. These programs — called Provincial Benefits and Measures (PBMs) in New Brunswick — include targeted wage subsidies, targeted earnings supplements, self-employment assistance, job creation and labour market partnerships, research and innovation, and a skills loans and grants program. The Province of New Brunswick also provides EI clients with a variety of services including assistance in developing career action plans, employment counselling and job placement services.

LMDA Development

New Brunswick has seen a long history of co-operation between the federal and provincial governments, at both regional and local levels, which has resulted in innovative partnerships such as NB Job Corps and NB Works. The Canada/New Brunswick agreement was built on this tradition of co-operation.

In late 1995, and during 1996 several actions by the federal government in the area of Employment Insurance and Labour Market Development were announced that would enable New Brunswick to take an expanded role in delivering human resource development services.

On November 27, 1995 federal withdrawal from labour market training was promised by the Prime Minister recognizing that labour market training relates to provincial responsibility for education.

In July 1996, Bill C-12 was enacted, bringing together in a single statute under the name "Employment Insurance" provisions for income support and employment assistance for eligible unemployed persons. The EI Act commits the federal government to work more closely with the provinces in labour market activities and also authorizes the Government of Canada to enter into agreements on the design and delivery of new active employment benefits and measures.

On May 30, 1996 a labour market proposal was made to the provinces and territories by the Government of Canada and on December 13, 1996, the Canada/New Brunswick Labour Market Development Agreement was signed enabling New Brunswick to assume responsibility for the design and delivery of active benefits and measures, supported by Employment Insurance funds. The Agreement also provides for the establishment of Canada/New Brunswick Human Resource Service Centres (HRSCs) through which New Brunswickers will have access to a full range of federal and provincial labour market programs and human resource development services. During the first three years of the agreement (1997-2000), the Government of Canada will contribute more than $228 million from the EI account to the Government of New Brunswick to support active employment benefits and measures. Subsequent to the Cda/NB LMDA, a Letter of Understanding has been signed between the parties to address a number of transitional and operational issues resulting from the agreement.

1.2 Eligibility Criteria

The eligibility criteria for participation in the Provincial Benefits and Measures (PBMs) vary with the program. Detailed eligibility criteria by program are provided in the descriptions of the PBMs presented below.

In order to meet the objective of reducing "unnecessary overlap and duplication" of services, programs such as Partners, the Entrepreneur Program, Job Action and Employment Assistance Services encompass the provision of services for target groups above and beyond those covered by the Cda/NB LMDA. Services for the "non-LMDA covered" participants are funded from a separate allocation of provincial dollars. The scope of this evaluation includes only those participants and services that are funded under the Cda/NB LMDA.

Pursuant to the Cda/NB LMDA² , "EI client" means:

• "an unemployed person who, when requesting assistance under a provincial benefit or measure,
  1. is an active claimant, or
  2. had a benefit period that ended within the previous 36 months, or
  3. had a benefit period established for him/her within the previous 60 months and
     1. was paid parental or maternity benefits under the Employment Insurance Act or the former Unemployment Insurance Act,
     2. subsequently withdrew from the labour force to care for one or more of their new-born children or one or more children placed with them for the purpose of adoption, and
     3. is seeking to re-enter the labour force".

1.3 Provincial Benefits and Measures

Annex 1 of the Canada/New Brunswick LMDA describes the Provincial Benefits and Measures (PBMs) to be provided by New Brunswick. Section 63 of the EI Act requires that contributions from the EI account only be made toward the cost of programs which are "similar" to those established by the Commission under Section 59 of the EI Act and which are consistent with the purposes and guidelines set out in Part II of the EI Act.

The subsections which follow provide detailed descriptions of the PBMs that are delivered by New Brunswick under the Cda/NB LMDA. It is important to keep in mind that year one of this agreement is transitional and, therefore, the service delivery infrastructure is at a developmental stage.

For the evaluation of the Cda/NB LMDA, only those clients that are funded under the agreement will be evaluated — that is, the focus is on funds used to provide services to EI clients, both claimants and reachback.³

Provincial Benefits

Partners

(Implementation Date: April 1, 1997)

The objective of Partners is to provide wage subsidies to eligible private sector employers and non-profit organizations in order to assist in the establishment of permanent employment or annually recurring seasonal jobs in New Brunswick. Employers applying to this program must demonstrate the "potential for permanence" of any job for which funding is requested.

Participating employers are funded by "wage reimbursement". Currently, wages are reimbursed at a rate of 50 per cent for permanent full-time jobs, 30 per cent for seasonally recurring jobs and 60 per cent for jobs for recent post-secondary graduates⁴. In all cases, the maximum provincial contribution is currently set at $6.00 per hour for a maximum of 40 hours per week.

Entrepreneur Program

(Implementation Date: April 1, 1997)

The objective of this initiative is to help unemployed individuals create jobs for themselves and others through self-employment.

The Entrepreneur Program has two distinct components:

• Under the Loan Guarantee component an unemployed New Brunswicker may receive a loan currently of up to $10,000. This loan is guaranteed by the Province of New Brunswick for two years and is interest free for one year.
• The Self-Employment Benefit provides various types of support during the business start-up period for EI clients only. Clients in receipt of EI benefits can continue to receive their benefits until their claim terminates. The income support may continue at a provincially determined rate to a maximum of 45 weeks, contingent upon the approval of a project consultant from the Province.

Application may be made for either or for both components. Loans under the Loan Guarantee and benefits under the Self-Employment component are judged on the business plan and the applicant's ability to implement the plan.

Job Action

(Implementation Date: April 1, 1997)

Job Action is designed to give the unemployed a meaningful work experience which will support them in obtaining long-term employment. The program provides participating employers in municipalities, non-profit organizations or government departments or agencies with incentives to help create short-term job opportunities.

EI claimants are eligible for a non-insurable supplement to their benefits while they participate in the program.

Skills Loans and Grants

(Implementation Dates: Grants portion — October 1, 1997
Loans portion — (not yet implemented)

The Skills Loans and Grants (SLG) program will provide financial support to EI clients to pursue education or training. Subsequent to the EI Act in 1995, HRDC phased out of the direct purchase of education and training programs, training allowances, and "fee payer" arrangements previously funded under the Unemployment Insurance and National Training Acts. As opposed to a general entitlement, support through the SLG program is considered to be a selective investment in an individual client, based on a case management process.

Earnings Supplement Programming

Programming in this benefit area is intended to be targeted towards EI clients who are displaced workers and others with high wage expectations, frequent users of EI and possibly Social Assistance (SA) recipients. New Brunswick is currently not offering programming in this area. It is anticipated that a program will be designed based on findings from ongoing national pilot projects.

Provincial Measures

Adjustment Service Initiative

(Implementation Date: April 1, 1997)

The Adjustment Service Initiative (ASI) is the term used to describe the Province's approach to the adjustment process. Activities under ASI are in place to meet objectives in two areas, Employment Adjustment and Development and Community Partnership.

• Employment Adjustment and Development: The objectives of this component are to assist private sector employers and non-profit organizations in becoming more effective and efficient, resulting in stable employment and/or the creation of new jobs. This component may also be used to help those persons affected by lay-offs reintegrate into the work force.
• Community Partnerships: The objective of this component is to encourage communities to take responsibility for their own employment related needs by building on local strengths and existing infrastructure. By developing positive relationships between community organizations, complementary human resource and economic development strategies can be advanced.

Employment Assistance Services

(Implementation Date: April 1, 1997)

The Employment Assistance Services (EAS) program is intended to help unemployed individuals obtain and maintain employment. EAS is targeted to those individuals who face barriers to employment which cannot be addressed fully by the services offered by the Canada/New Brunswick Human Resource Service Centres.

EAS is delivered in partnership with non-profit, private and/or public organizations through third party contracts. Services are purchased in accordance with the NB Public Purchasing Act.

Projects may include the following: individualized employment counselling — including skill assessment, development of action plans, case management, job search, job development, job placement, and follow-up; group "Job Search" activities; referral services — linkages with employers, educational institutions and government, and coordination of support services; general education and awareness activities — for employers and the community at large; and/or marketing of clients and client groups.

Project contracts include negotiated benchmarks for results, which include, at a minimum: number of employment placements; number of placements resulting in long-term labour force attachment; and savings resulting from decreased dependency on income support (EI, SA, etc.).

Research and Innovation

(Implementation Date: April 1, 1997)

According to Annex 2 of the agreement, under the Research and Innovation measure, New Brunswick will conduct activities, experimentation and research to address labour market development, policy and design issues. Currently, the only activity being carried out in this area is Rural Experience.

Rural Experience is a joint federal/provincial job creation initiative designated specifically for rural New Brunswick. Through this initiative, funds have been made available to employers wishing to create new jobs primarily in the agricultural, tourism, silviculture and environmental areas. In 1997-98, eligible participants for this program included social assistance recipients and EI clients. Participants must come from rural areas of the province. Rural New Brunswick has been defined as areas outside the three largest urban centres of Fredericton, Saint John and Moncton.

1.4 Evaluation Objectives

The objectives of the overall evaluation of the Cda/NB LMDA (hereafter referred to as the LMDA) are as follows:

• to measure the extent to which the Provincial benefits and measures and the supporting "infrastructure" — which includes counselling, needs determination, information management, marketing and co-location — have been successful in achieving the objectives of the LMDA in terms of assisting persons to obtain and retain employment;
• to provide information to managers, policy makers and program designers on a number of program issues, including design, implementation, delivery, flows, experiences and data needs, so that optimum use of resources is possible within each local labour market;
• to estimate the cost-effectiveness of the interventions; and
• to demonstrate "what works best" and "lessons learned".

The evaluation of the LMDA consists of two major components, a formative evaluation and a summative evaluation. A formative evaluation is normally conducted during the first 12-18 months of implementation of a program/project/initiative. Its focus is on improving the design, delivery and supporting infrastructure of a program/project/ initiative during the implementation period. Summative evaluations are designed to measure the outcomes, impacts and cost-effectiveness of programs, projects and interventions. An initial summative evaluation is conducted 3 to 5 years after start-up with follow-up work done every 3-5 years.

The purpose of the formative evaluation of the LMDA, the subject of this report, is to supply information indicating what improvements, if any, are required to the LMDA design, delivery and supporting infrastructure that would permit it to better meet its objectives. Specifically, the objectives of this formative evaluation are:

• to collect baseline data to permit tracking of impacts over time;
• to identify and contribute to the resolution of unanticipated problems;
• to ensure that participants are progressing toward intended outcomes;
• to describe the strengths and weaknesses of the LMDA, which would contribute to innovative changes;
• to allow for adjustments to be made to the LMDA;
• to facilitate adjustment of the LMDA by supplying information on administration, management and operations during the first year of the LMDA; and
• to produce information on "what works best" and "lessons learned" to date.

1.5 Purpose of this Document

This report presents the methodology and findings from the Formative Evaluation and Baseline Data Collection for the Canada/New Brunswick Labour Market Development Agreement.

The original design of the formative evaluation of the LMDA consisted of one stream of several methodological components: a documentation and administrative data review; 35 key informant interviews; eight focus groups; five case studies; six mini-case studies of the Adjustment Service Initiative (ASI); surveys of participants and a comparison group; and a survey of employers and unions. During the development of the research design, it became apparent that certain aspects of the delivery structure (i.e., co-location) as well as the programming (i.e., the loans portion of Skills Loans and Grants) were not yet in place. Hence, it was decided that it would be prudent to delay several components of the research to provide additional time for these activities to occur.

Thus it was decided by the Joint Evaluation Committee⁵ that the evaluation would be divided into two phases. Phase I would be conducted immediately in order to provide information for an interim report in October 1998. Phase II would be undertaken following the submission of this first report in October, culminating in a final report to be submitted by the end of March 1999. This document presents the integrated findings of these two phases.

The formative evaluation was conducted through the use of a variety of data collection methodologies including key informant interviews, focus groups, documentation and administrative data review, surveys, case studies and mini-case studies. Each methodology is briefly described in this chapter.

2.1 Key Informant Interviews

Phase I

Four groups of all potential key informants were formed based on the different perspectives that individuals would bring to the interview (i.e., program design and management; program delivery; systems and infrastructure; and the New Brunswick Labour Force Development Board (NBLFDB)). From a master interview guide, four distinct interview guides were developed, one for each group. The Phase I key informant interviews were with individuals from three of these groups (i.e., program design and management, program delivery, and systems and infrastructure groups).⁶

There were 25 key informant interviews conducted for Phase I. These interviews involved a total of 33 respondents (14 officials with the Department of Labour, nine with HRD-NB, and ten with HRDC).

Attempts were made to conduct as many of these interviews in person as possible. Six interviews, however, were conducted by telephone with individuals who were located outside of Fredericton, Moncton and Bathurst (or who were unavailable at the time of our visit to these centres). A series of interviews with those in Fredericton who preferred to be interviewed in English was conducted from June 23 to 25, 1998. A second series of interviews with individuals located in Bathurst and Moncton, as well as individuals in Fredericton who preferred to be interviewed in French, and other outstanding English interviews in Fredericton was conducted during the week of July 6, 1998. Most of the telephone interviews were conducted during the remainder of July 1998.

Phase II

The remaining interviews from the initial list of potential key informants were conducted in Phase II. The Phase II key informant interviews were with individuals from three of the four groups described earlier (i.e., program design and management, program delivery, and NBLFDB).⁷

There were nine key informant interviews conducted for Phase II. The breakdown consisted of the following:

• four interviews with representatives from HRD-NB;
• one interview with a representative from the Department of Labour;
• two interviews with representatives from HRDC; and
• two interviews with representatives of the NBLFDB.

Attempts were made to conduct as many of these interviews in person as possible. Three interviews were done in person during visits to the focus group locations (March 1 to March 3, 1999), while the remaining six interviews were conducted by telephone. Telephone interviews were conducted during March 1999.

2.2 Focus Groups

Phase I

Three focus groups were held during the week of July 6, 1998 with staff located in Bathurst and Moncton who were involved in the delivery of the programs. The groups included both transferred and non-transferred staff from HRDC, HRD-NB and the Department of Labour. Participants were asked to express their views on the relevance of the Provincial Benefits and Measures, their roles, the implementation of the LMDA, co-location, service delivery, the impacts of the LMDA, and suggestions for the future.

In each location, staff were asked if they preferred to participate in an English or a French focus group. Two groups were held in Moncton, one in English and one in French. The one group in Bathurst was in French.

Phase II

Four focus groups were held with clients and employers located in Bathurst and Fredericton. One client group and one employer group was held in each centre. Both group discussions in Bathurst were conducted in French, while both groups in Fredericton were in English. The focus groups took place during the week of March 1, 1999.

2.3 Documentation and Administrative Data Review

Documentation that was provided on various aspects of the LMDA included the following:

• Local Implementation Committee Meetings Minutes and Federal/Provincial Working Group Status Reports;
• Joint Service Delivery Framework for the Set up and Development of Canada/New Brunswick Human Resource Service Centres, January 1998;
• Communications file (e.g., press releases, internal memos);
• Accommodations file (e.g., memos on financial arrangements, Financial and Administrative Services, LMDA Implementation Working Group meeting minutes);
• Documentation on Employee Transfer Process;
• Documentation on Negotiation;
• Policy Framework for the Implementation of Employment Benefits and Measures;
• Joint Federal Provincial Accountability Framework;
• Documentation on Consultation on Programs and Services;
• Documentation on Planning, Accountability and Evaluation;
• Documentation on Systems;
• Documentation on Service Delivery;
• Documentation on "Stocktaking on Implementation";
• Final Report: Best Practices Review of IAS/ASI Initiatives; and
• Evaluation of Rural Experience.

This documentation was reviewed by the team at the beginning of the project to ensure a common and thorough understanding of the context for the evaluation. This review of the documentation also contributed to the design of all research instruments.

The administrative data analysis enabled us to profile PBM clients and provide information on program intake. As well, the administrative data permitted a profile of the target population for PBMs to be developed, and formed the basis of the selection of an appropriate comparison group for the participant surveys.

2.4 Surveys

Participant Survey

A total of 1,600 interviews were completed with participants from January 15, 1999 to February 9 , 1999. The overall margin of error for the survey is ±2.3 per cent. That is, the overall survey results are accurate within ±2.3 percentage points, 19 times out of 20. The overall response rate was 53.5 per cent and the overall refusal rate was 7.1 per cent. Further details on the methodology for the survey are presented in Appendix A.

Comparison Group Survey

A total of 800 interviews were completed with non-participants from February 9, 1999 to February 13, 1999. The overall margin of error is ±3.5 per cent, meaning the overall survey results are accurate within ±3.5 percentage points, 19 times out of 20. The response rate for the survey was 48.4 per cent and the refusal rate was 8.8 per cent. Further details on the methodology for the survey are presented in Appendix A.

It should be noted that the current comparison group survey results are based solely on feedback from current claimants and do not include reachback clients. Preparations for a partial replication of the comparison group survey are underway in order to solicit feedback from reachback clients as well.

2.5 Case Studies

A total of five case studies were conducted as part of Phase II of the evaluation. Two types of case studies were conducted. The first type focused on specific benefits and measures, and the second on the evolution of service delivery at different sites. Two case studies of the first type were conducted (one of a specific benefit [Skills Loans and Grants], one of a specific measure [Employment Assistance Services]) and three case studies of the second type were conducted (at three sites — Miramichi, Edmunston and Fredericton).

Each case study included a review of documentation and administrative data, and from seven to 12 key informant interviews. The case studies were conducted on-site.

2.6 Survey of Employers and Unions

Employers

A total of 300 interviews with employers who had participated in the Partners, Job Action and/or Rural Experience programs were completed from February 12, 1999 to February 24, 1999. The overall margin of error is ±4.4 per cent. That is, the overall survey results are accurate within ±4.4 percentage points, 19 times out of 20. The refusal rate for all employer groups was very low and ranged from 3.9 per cent for Rural Experience employers to 6.8 per cent for Partners program employers, with an overall refusal rate of 5.3 per cent. The response rate for all employer groups was equally satisfactory, ranging from 41.6 per cent among Partners program employers to 54.7 per cent for Rural Experience employers. The overall response rate was 47.9 per cent. Further details on the methodology are presented in Appendix A.

Unions

The original study design called for a survey of union representatives from companies who had participated in wage subsidy programs funded under the LMDA since April 1, 1997. The sample for the union survey was to be developed on the basis of referrals from respondents to the employer survey. At the completion of the employer survey, however, only 13 names and phone numbers of union representatives had been given by employer survey respondents. On the basis of the low number of union names that were obtained, the approach to the union survey was modified somewhat. Given the low number of responses that would be obtained with a sample of only 13 individuals, it was decided to implement the survey, but to treat the data gathered from the survey qualitatively.

Of the 13 union representatives for whom we had contact information, the results of the union survey yielded only two completed interviews. Six of the 13 union contacts denied any knowledge of the LMDA employment programs, four of the telephone numbers that were given were not in service, and one union contact was unable to do the survey because of other reasons or illness. The results of the two completed union surveys are incorporated qualitatively in this report.

It is important to note that since few employers who participated in programs have unions, this data source was less relevant to the overall evaluation findings than originally anticipated. Thus, the poor response to the union survey should have very little impact on the applicability of evaluation findings.

2.7 ASI Mini-Case Studies

A total of six mini-case studies of ASI were conducted in Phase II: two of community committees; two upside firm-based; and two downside firm-based. Each mini-case study consisted of an interview with the committee chair and an interview with one other individual who was involved from the beginning and knew about the processes and history of the committee.

To develop a profile of LMDA program and service participants, this chapter outlines sociodemographic data, for both participants overall and for users of individual programs, based on the most part on administrative data supplied by Human Resources Development Canada (HRDC). Comparisons are also drawn between active EI claimants on the one hand and reachback and comparison groups on the other.

3.1 Comprehensiveness of LMDA Participant Data

As shown in Exhibit 3.1, HRDC administrative data provided to Ekos for purposes of this evaluation were relatively complete for participants' gender and age, and to a lesser extent, spoken language. There was far less information available (information for only about one in four individual participants) regarding those who may have been disabled, a visible minority, a social assistance recipient, or Aboriginal. Education data were available for only one of every seven participants. Because of concerns over measurement bias, profile results for variables other than age, sex, language and income (see down) will not be reported based on the administrative data.

3.2 PBM Activity

By a significant margin, the intervention most likely to be used was Skills Loans and Grants (SLG). A total of 54.5 per cent of LMDA participants were involved in SLG, compared to 12 per cent for Partners (Exhibit 3.2). Other benefits and measures were used much less frequently: six per cent participated in Job Action; three per cent participated in Employment Assistance Services; and two per cent participated in the Entrepreneur benefit. Twenty-two per cent participated in Rural Experience, which although it is not an LMDA Provincial Benefit or Measure, is included here because it is partly funded under the Research and Innovations component of the LMDA. Results on activity by type of intervention are presented in Appendix B.

Most (about nine in 10) LMDA participants engaged in only one intervention (Exhibit 3.3). The vast majority of others (9.4 per cent of all LMDA participants) participated in two interventions, meaning that only one per cent of all participants had three or more interventions.

3.3 Profile of LMDA Participants Overall

About twice as many men than women participated in Cda/New Brunswick LMDA benefits and measures (Exhibit 3.4). Almost two-thirds (64.1 per cent) of participants were men, compared to one third (35.9 per cent) who were women. It is important to note that the proportion of women accessing benefits and measures is similar to the overall proportion of women who made new claims for EI in 1997 and 1998⁸ (36 and 40 per cent, respectively).

Participants tended to be relatively young — under one in five (17.2 per cent) were 45 or older. Moreover, a similar number of participants (41.2 per cent) were between 30 and 44 as were under 30. The youth participants were just as likely to be under 25 (21.8 per cent) as between 25 and 29 (19.8 per cent) years. A solid majority (72.1 per cent) of participants identified themselves as English-speaking. Just over one quarter (27.8 per cent) indicated that they were French-speaking.

The mean total income of participants decreased in the period leading up to the intervention (Exhibit 3.5).⁹ The average value of Employment Insurance (EI)period benefits and social assistance benefits received fluctuated, and generally declined, during that time period. EI benefits were an average of $650 lower in 1996 than in 1994, while social assistance benefits were an average of $430 lower over the same time period.

3.4 Profile of Participants by PBM

Sociodemographic differences between participants by program type are detailed in Exhibit 3.6 based on HRDC administrative data. Information is presented with these two variables because of deficiencies in the other variables as noted in Exhibit 3.1. The total column of this exhibit repeats some of the results presented in Exhibit 3.4. A more comprehensive profile of participants in each benefit and measure, based on the survey data, is presented in Chapter Six.

A description of participation in each benefit and measure incorporating these results follows.

• Partners participants were more likely to be male (59.6 per cent versus 40.4 per cent female), although somewhat less so than the overall mean (Exhibit 3.6). Just under half of program participants (46.3 per cent) were between 30 and 44 years of age.
• As shown in Exhibit 3.6, Entrepreneur participants tended to be older (three quarters were 30 years of age or older) than the average LMDA participant. Participants' gender mix was, however, close to the overall distribution (61.1 per cent male, versus 38.9 per cent female).
• Job Action recorded the lowest male participation (46.8 per cent male versus 53.2 per cent female) (Exhibit 3.6). Participants' mean age tended to be near the overall mean age.
• The most widely used program, Skills Loans and Grants (SLG), had almost twice as many male participants as female (65.7 per cent male versus 34.3 per cent female). This program had the highest proportion under 25 years of age (24 per cent).
• Employment Assistance Services interventions were used almost equally by men and women (51.5 per cent men versus 48.5 per cent women) (Exhibit 3.4). They tended to be older than the overall population of participants.
• Rural Experience, although it is not an LMDA Provincial Benefit or Measure, is included here because it is partly funded under the Research and Innovations component of the LMDA. The program had the highest proportion of male participants (69.9 per cent male versus 30.1 per cent female) (Exhibit 3.4). This clientele was slightly older than the average age of other participants.

View EXHIBIT 3.6

3.5 Profile of EI Claimant, Reachback, and Comparison Groups

As mentioned in the program description, there are two client groups eligible for PBMs: individuals who are current EI claimants and reachback clients (those who have claimed regular EI within the past three years or claimed maternity/paternal benefits within the last five years). The administrative data file could not provide a reliable indicator of EI claimant/reachback status of participants and so an indicator was constructed based on participants' EI profile.

In Exhibit 3.7, we observe the extent to which the reachback share of participants varies across programs, based on the administrative data for the full population. This exhibit also presents the distribution of reachbacks according to the length of time between when the EI claim ended and the current intervention began. Note that excluded are those whose claimant status is unknown. The results in row one indicate, first, that reachbacks represent over one-half (56 per cent) of all participants. Second, the vast majority (87 per cent) of Entrepreneur participants is represented by reachbacks. Third, there is little variation in reachback share for the other interventions, with the share being the smallest for SLG participants (47 per cent). Fourth, the largest share of reachbacks participated in their intervention within three months of the end of their EI claim, with over one-half (56 per cent) reachbacks being in this group.

Exhibit 3.8 presents an overview of the sociodemographic characteristics of the claimant, reachback, and comparison groups based on survey data.¹⁰ Note that the comparison group is made up of only active EI claimants (who did not participate in the PBMs). The data suggest, first, that females represent a greater percentage of reachback participants than EI claimants. Women represent an even larger share of the comparison group (49 per cent). As for the age mix, the results indicate that, at 15 per cent, youth (under 25 years of age) represent a greater share of EI claimant participants than of reachbacks (10 per cent) and the comparison group (six per cent). The language profile of EI claimants, reachbacks and comparison group members is very similar, with over 70 per cent of all three groups being anglophone. The educational profile of the three groups is also similar, though EI claimants are somewhat more likely to have completed a college diploma than their counterparts in the reachback group, while members of the comparison group were somewhat more likely to have a university degree.

As for marital status, reachbacks were more likely than EI claimants to have once been married (13 versus eight per cent), while comparison group members were considerably more likely to be married or in a common-law relationship 67 versus 52 per cent). Finally, the representation of equity group members is similar among the three groups.

Exhibit 3.9 presents historical data on EI claimants' and reachback clients' income and earnings in the three years prior to their intervention and also their use of EI and social assistance during this time period. These figures are based on administrative data. It should be noted that EI claimant and reachback results are based on the full population of program participants to obtain an accurate picture of these individuals' earnings and income history. However, for the comparison group members, the results are based on just survey respondents because of difficulties in establishing the comparative reference period for the full population of non-participants. (Recall that individuals were selected into the comparison group on the basis of having an EI claim that coincided with participants in the PBMs). It should be further noted that, because of this, comparisons between participants and non-participants should be made in terms of patterns over time rather than absolute levels of earnings and benefits.

It should also be pointed out that in Exhibit 3.9 the EI figures for EI claimant and reachback participant groups were computed somewhat differently from the other figures presented in the table. The former were based on a 12-month period basis rather than a calendar year period on which the other figures were based (including the EI usage figures for the comparison group). This approach yielded a more accurate representation of EI use based on monthly usage, particularly for reachbacks. For example, on a calendar year basis, a reachback participant whose EI claim ended in March 1996 and whose LMDA intervention began in December 1997 would have been counted as having a claim one year before the intervention. This is despite the fact that the person would have been classified as a "21-month reachback", i.e., whose EI claim ended 21 months prior to the intervention. Since EI monthly EI data were available, we were able to compute EI usage rates and levels for 12-month periods and this person would have been more accurately counted as receiving EI benefits in the second "year" (12-month period) prior to the intervention but no EI benefits one "year" before. However, since, as noted above, it was difficult to establish (hypothetical) intervention dates for the full population of non-participants, we did not perform this calculation for the comparison group. And, because only annual and not monthly data were available for gross earnings and social assistance (SA), the same calculation could not be performed for the earnings and SA. Once again, as noted above, because of the different methodologies, comparisons between the groups should be made according to patterns not levels.

Exhibit 3.9 indicates, first, that, while in the third year prior to program entry EI claimants and reachbacks were in similar earnings brackets, the pattern in the intervening years indicates rising earnings among the former and declining earnings among the latter. For comparison group members, earnings too were declining over the pre-intervention period. As for employment insurance (EI) benefits, panel 2 of Exhibit 3.9 indicates that, as expected, EI claimants were less likely than reachbacks to have received EI benefits in the period leading up to the intervention and somewhat less likely than the comparison group in the 12 month period prior to an intervention. For comparison group members, EI incidence appears to have been rising over time at levels similar to participants. As for levels of EI benefits (panel 3), we observe more or less stationary annual levels for EI claimants and rising levels for reachbacks leading up to program entry. (Note that the means and medians are for the entire population of participants including those with zero benefits in the respective year.) For the comparison group, mean EI benefit levels declined in the last two years prior to (hypothetical) program entry, while the median rose in the last three years.

Finally, the social assistance (SA) incidence results (panel 4 of Exhibit 3.9) indicate, first, that SA incidence was similar in each of the years prior to program entry for EI claimants but rising for reachbacks. By the last year prior to program participation, SA incidence was more than twice as high for reachbacks as for EI claimants (19 versus eight per cent). For the comparison group, SA incidence was even lower (two per cent). As for SA levels (panel 5 of Exhibit 3.9), we observe that mean SA benefit levels were falling for EI claimant participants but rising for reachback participants. Mean SA levels were negligible for comparison group members and falling over time.

4.1 Compatibility of PBMs with EI Act and LMDA

Key informants observed that the PBMs and supporting infrastructure are compatible with the programs that previously existed in New Brunswick (i.e., former provincial programs and former federal services delivered by the National Employment Service (NES), such as service needs determination, case management, employment counselling, and labour market adjustment). The PBMs are simply the current expression of the previous programs, with some minor modifications. Moreover, as the PBMs focus on moving clients toward self-sufficiency, they are generally compatible with the intent of the EI Act and the LMDA.

In the view of some key informants, the principles for operating programs/services at the provincial level are somewhat different from those at the federal level and from some specified in the EI Act. As reflected in the EI Act, active benefits and measures are intended to be both client— and labour market-centred (matching the two sides), with the worker as the client. Consistent with this focus, HRDC has traditionally focused on matching the worker client with employers. With the LMDA and PBMs, on the other hand, such matching may not be so readily coordinated because HRD-NB deals with the worker/social assistance client whereas the Department of Labour delivers active benefits and measures with the employer as the client.

4.2 Complementarity and Overlap of Programs/Services

On the basis of evidence from the interviews, focus groups and case studies, there may be some cases where the PBMs overlap to a degree with other existing federal or provincial programs. For instance, under its pan-Canadian activities, the federal government offers programs for youth, Aboriginals and persons with disabilities. Although the Province does not have any programs specifically targeted at these client groups under the LMDA, these types of clients can apply for other generic provincial programs. In addition, key informants identified the following areas of confusion and duplication: HRD-NB and the Department of Labour are still sorting out who "owns" the client at various stages of case management/service delivery, which may cause some confusion and competition among programs; other provincial departments (e.g., Economic Development and Tourism) have their own initiatives related to job creation, which may involve some duplication; the federal Electronic Labour Exchange (ELE) duplicates the provincial NB JobNet; the division between federal and provincial responsibilities has not been completely settled in the area of Research and Innovation; the Entrepreneur Program — Loan Guarantee has the same target clientele as the Self-Start program (offered by the provincial Department of Economic Development), though slightly different eligibility criteria; and there is some overlap of Skills Loans and Grants with Student Aid (Canada Student Loan and New Brunswick Student Loan).

Most respondents did not feel, however, that there are any major problems of duplication or programs working at cross-purposes with one another. In the view of many key informants, any minor problems in this respect may be viewed as "growing pains" which will probably be sorted out as the LMDA implementation proceeds. Moreover, co-location should ultimately serve to facilitate the coordinated delivery of complementary programming.

4.3 Relevance of PBMs to Clients and Communities

On balance, interview, focus group and case study respondents felt that the PBMs are generally relevant to the needs of clients and communities. For example, interview and case study results indicate that PBMs such as Skills Loans and Grants (SLG), Employment Assistance Services (EAS) and the Adjustment Service Initiative (ASI) have sufficient flexibility to be adapted to client needs and have been providing helpful assistance to clients. In addition, the general focus of the PBMs on helping people get back to work is clearly relevant to the needs of clients and communities.

Some reservations were expressed, however. For example, in interviews with HRDC officials it was observed that, at the local level, there is no evaluation information (i.e., no detailed monitoring of client needs and the extent to which they are being met by existing programs and services), so it is difficult to say whether or not the PBMs are relevant to the needs of individuals, employers and communities. It was also noted that participants in the PBMs are not representative of the individuals who most require these services because the target group is defined as EI clients. In the past, much of the needed counselling was done with youth and re-entrants into the labour market — groups which need help because they have little attachment to the labour market, but no longer qualify for EI. However, this problem is due more to the EI Act than to the LMDA.

Similarly, most provincial officials felt that the PBMs are mostly relevant but that they only partially reach people in the community who most require these services. In other words, there are people in need who "fall through the cracks". Key concerns were as follows:

• There are people with a genuine need (e.g., people who are under-employed or who lack steady employment), but who do not fall under the definition of EI client and hence are not eligible for EI benefits or for assistance under the PBMs. These types of clients are often automatically screened out by the EI agents, so that the case managers never see them. There may be a need for a better system whereby these types of clients would be referred to case managers who could then refer them to programs for which they are eligible, such as Employment Assistance Services and resource centres.
• The PBMs reflect the working world of the 1970s and 1980s (when full-time work was the norm) more so than that of the 1990s. There is a need to adopt a more flexible definition of "work" and allow case managers more flexibility to negotiate with employers, rather than relying exclusively on rigid formulas.
• In rural areas, the SLG program could be better adapted to meet local community needs. In particular, there is limited choice of training programs in rural areas due to the individualized focus of SLG, whereby each client is responsible for finding a course suitable to his/her needs. With the previous approach, whereby a significant number of seats in a training program could be purchased (making it worthwhile for a local college to offer the course), clients in rural areas had much better access to relevant programs. It would be helpful if SLG were allowed more flexibility to use this purchase of training approach in rural areas.
• Small and medium-sized businesses that are in the midst of cut-backs could benefit from wage subsidies under the Partners program, but do not meet the eligibility criterion of providing incremental employment opportunities.

In focus groups with staff, participants noted that in some respects the programs, services and supporting infrastructure at HRD-NB meet client needs better than before, principally because the training programs are more flexible (for people living in larger urban centres). One limitation, however, is the perceived focus of many programs on short-term results ("band-aid solutions"). At the Department of Labour, programs are employer-driven so employers typically have already recruited most of their workers by the time of their application for wage subsidies or other programs. Some participants observed that the DOL programs are continuing to meet employers' needs, even though there are fewer programs available and somewhat less flexibility because the requirements are more strict. Respondents found it difficult to say if Labour programs are meeting the needs of worker clients, however, because staff have very little direct contact with workers. Under the previous federal programs, there was more interaction between the staff serving workers and those serving employers. Ideally, DOL staff should inform HRD-NB staff about specific programs and try to establish linkages between employers and workers, but this is not happening at this stage.

Finally, in the case studies some respondents identified factors that limit the relevance of PBMs. In particular, the centralized decision-making associated with some provincial programs, a lack of coordination and information exchange between HRD-NB and DOL, and low awareness of the PBMs among both worker and employer clients have been problems.

5.1 Roles and Responsibilities

Clarity of Roles and Responsibilities

Mixed opinions were expressed regarding the clarity of partners' roles and responsibilities in the LMDA. Although the majority of respondents in interviews, focus groups and case studies felt that their LMDA roles and responsibilities are basically well understood and satisfactory, many pointed to areas of confusion. For example, former federal staff who were transferred to the Province are still adjusting to their new role and environment/culture, while some provincial staff feel that the former staff are still in "federal mode". Also, there is not a clear understanding of federal versus provincial roles and responsibilities for the reception function and for the service delivery model at co-located sites. These and other areas of confusion appear to be due largely to inadequate communications both within and between levels of government.

Some HRDC officials expressed concern about a lack of clarity around roles and responsibilities across the board. For instance, one key informant observed that an overall structure and process for implementation does not exist, but that this is necessary for the success of something as complex as the LMDA. Another argued that there is not yet a clear understanding of how the federal government will manage its business in this new environment and what role it will play with regards to the LMDA (e.g., "We don't necessarily want to be policing the Province, but at the same time, there are a number of issues and concerns that we have in terms of legislation"). It was also perceived that HRD-NB employees in regional sub-offices may be unclear about their accountability requirements.

On the provincial side, officials noted that there has been some confusion regarding the respective roles and responsibilities of HRD-NB and the Department of Labour for Skills Loans and Grants (SLG). Provincial key informants also observed that there is some confusion over LMDA accountability responsibilities, and over federal responsibilities in program areas perceived to overlap somewhat with provincial programs (e.g., the federal pan-Canadian programs) as well as which level of government should first serve youth, persons with disabilities and Aboriginals.

Another related issue noted in interviews and case studies is the fact that some former HRDC employees transferred to the Province did not have their seniority recognized by the Canadian Union of Public Employees (CUPE), the union representing the Province. To make matters worse, these staff sometimes work alongside other employees whose seniority was recognized. This state of affairs has had a negative impact on their morale, and they are concerned about their job security.

Roles for Additional Players

The consensus was that the current players/partners — HRDC, HRD-NB and the DOL — are probably sufficient at this stage, and that the LMDA is complicated enough at present. Later, after the LMDA is fully implemented, there may be room for additional players to play a bigger role in LMDA consultation, planning and design. These partners might include private sector employers, the not-for-profit sector, local community organizations, and economic development commissions.

5.2 Implementation of the LMDA

Most Successful Aspects of Implementation

Several aspects of the LMDA implementation were regarded as successful by key informants, focus group participants and case study respondents. In particular, there has been good cooperation between the three players and two levels of government, and a willingness to work together in the field through the local implementation teams and regional working groups (which included the Canada Employment and Immigration Union (CEIU), a component of the Union of the Public Service Alliance of Canada (PSAC)). In the view of some respondents, having control over the programs and having field staff in communities has improved local flexibility and the pursuit of regional economic development priorities. At the outset, the transfer of funds through the federal authorities went quite smoothly as did the transfer of federal staff. There were good communications, sound change management and careful attention to HRDC staff being transferred to the Province who received a good package in terms of money and protection. Moreover, program delivery and good client service were considered to have been maintained throughout the process. The fact that the LMDA has become functional, in spite of the technical complexity of the task (three departments with different information technologies), was viewed as an achievement in itself.

Least Successful Aspects of Implementation

Respondents from key informant interviews, focus groups and case studies also identified numerous aspects of the LMDA implementation which, in their view, were not so successful. Their observations are summarized in this section.

Both federal and provincial officials commented on the negotiation process. Due to pressures on both the federal and provincial sides to finish negotiations and get the Agreement signed, the LMDA was signed hurriedly, with the understanding that many of the details (which were presented in general, management terms/concepts rather than in practical terms) would be worked out at the implementation stage. However, many of these details remain to be worked out, creating confusion and frustration for all players. A provincial official also cited the exclusion of the Canada Employment and Immigration Union (CEIU), a component of PSAC, in the negotiations as a weakness.¹¹ More broadly, there have been some problems and confusion for the public associated with the fact that three distinct events happened simultaneously — the passing of the EI Act and its changes, the reduction of the federal government's consolidated revenue fund (thereby significantly reducing the funds available for non-EI clients), and the devolution of programs to the Province through the LMDA.

Some disruption was created for staff because of an unanticipated delay — due to negotiations between HRD-NB and CUPE — between the first wave (in July-August 1997) and second wave (in February 1998) of federal staff transfer to HRD-NB. This delay caused problems for affected staff, who felt anxious, stressed, and as though they were in a "no man's land" with no direction because they were still federal employees with federal managers but were working with provincial programs. Moreover, this interrupted some good momentum which had existed at the time. In addition, the delay in negotiating an agreement on transferred staff's pension (i.e., negotiation between the federal Treasury Board and the provincial Department of Finance) was another weakness — this has not been completed yet, but is a big (emotional) concern for staff.

Additional problems related to the staff transfer were identified by study respondents. It was perceived that the Province probably was not totally prepared for the changes (i.e., in terms of having the proper systems and training in place). Similarly, staff did not feel adequately prepared and trained for their new role and the PBMs. In addition, provincial employees felt that federal employees being transferred were receiving special treatment (good compensation package, etc.), while some provincial managers did not feel that they got the best (former) federal employees. There has been "culture shock" for both federal and provincial employees and difficulty adapting to a new environment for transferred staff. Finally, the issue of the lack of provincial recognition of the seniority of former federal staff has not been properly resolved, and there is some remaining uncertainty and insecurity with respect to what will happen at the end of staff's three-year job security agreement.

Resources have also been a source of disagreement in the LMDA implementation. On the one hand, some federal officials perceived that the Province is not contributing enough effort/resources in the partnership, considering that HRDC has transferred so much to the Province. On the other hand, HRD-NB officials complained about the lack of resources to cope with numerous costs associated with co-location and the addition of 35 per cent more staff to their department — costs which were not fully thought through at the time of signing the LMDA. There were disagreements over some of the financial interpretations that the federal authorities made under the LMDA (e.g., number of positions to be transferred and salaries). These concerns were echoed by DOL officials.

Some problems particular to SLG were identified. The transfer of programs over to the Province to produce seamless service and one-stop shopping has not yet been successful in the case of SLG, and will not be achieved until the co-located sites are all in place. Clients are still confused about "what door" they need to go to for SLG funding. In addition, the recruitment of qualified staff for SLG has been disruptive. In the view of provincial officials, the program was not described accurately and positively, and little interest was shown from suitably skilled (former) federal employees. It has been an "arduous" process attempting to fill positions with transferred federal employees, and recruitment has been very difficult. Related to this, case study respondents noted that case managers could have benefited from more training in financial assessments and the negotiation of clients' financial contribution (which are perceived to be done inconsistently), and that the services of an expert in the delivery of large, complex programs (like SLG) would have facilitated implementation. Also, the delay in implementing the loan component of SLG has caused difficulties for client service.

More generally, inadequate communications and coordination within and between levels of government have impeded implementation. In addition, communications to worker and employer clients about the changes (they need to know about) have not yet been done. There has been a less than optimal connection between worker clients' needs and the delivery of active benefits and measures (e.g., clients may be on waiting lists, and may be confused as to who should be helping them). Also, the new case managers come from a variety of backgrounds, for example, a hard-nosed employment counselling focus versus a social counselling focus where a supportive, accommodating approach is the norm. Because these case managers are not a uniform group, it is unreasonable to expect consistency in their approach and decisions at this early stage of the LMDA.

Implementation Plans

Joint implementation planning was done by the senior management Joint Implementation Committee, and this was helpful in terms of sorting out roles and responsibilities. This planning also involved: the design of a collaborative planning framework for the local level (though this was not distributed due to delays in co-location); the setting of results targets, using national benchmarks initially with the understanding that these could be refined later if necessary; and the development of a monthly reporting format for the Province. In addition, jointly developed implementation plans were prepared by other committees at more junior management levels and by local implementation committees, and local joint planning meetings were held for a while. There have, however, been many delays in implementing plans (e.g., delays related to the transfer of federal employees, problems related to resources and incompatible information systems, delays in co-location). Still, the delivery of the PBMs has been proceeding since the beginning.

Some joint planning was attempted regarding the information technology aspects of the LMDA, with the assistance of a consultant, but this was not implemented as designed. What has been required is a series of revised plans and weekly meetings to track progress. Systems officials with all partners identified the lack of a dedicated project manager - to oversee and coordinate the overall LMDA implementation process, facilitate communications among the three players, and deal with the various problems which arise - as a barrier to successful planning and implementation.

Further problems with the planning process were identified. For instance, key informants felt that there has been poor communications and management of the planning process — that it has not been "process mapped" or "project managed" - and that there has been no clear step-by-step plan and schedule for implementation. Also, there have been difficulties in integrating the planning processes of the three players, and insufficient infrastructure for proper planning for LMDA implementation.

Disruption to Staff and Clients

Given the complexity of the task of implementing the LMDA and PBMs, there has been remarkably little disruption to client service in the view of many respondents. Reasonably good service has been maintained and there have been very few serious complaints. One exception has been the disruption to client service due to the delay in implementing the loan component of Skills Loans and Grants. Clients have had to keep calling back to ask when the full SLG program and the loan component would be available, but have not been given a definite answer, which interferes with their educational planning for the coming year (because they are unsure if they will have sufficient funds for the duration of their program). In addition, delays in co-location have created some confusion for clients, who may be sent to many different places for assistance (e.g., the HRCC, HRD-NB and job finding clubs).

The LMDA implementation process has been somewhat disruptive for staff. Although the initial transfer of federal employees apparently went quite well (e.g., in terms of communications, change management and the financial package), some staff now working for the Province feel isolated and confused, and are having difficulty adapting to their new environment and corporate culture. In addition, there are concerns about job security and the recognition of seniority for transferred federal staff, as discussed earlier. Also, many staff continue to feel the need for clear communications regarding the PBMs and their new roles and responsibilities. For instance, a single, automated, user-friendly information system on the new programs would be very helpful for front-line staff.

A number of issues for HRD-NB staff were raised in focus group discussions. It was noted that HRD-NB staff only have partial access to EI information and this presents difficulties for HRD-NB counsellors, who cannot follow up on an EI applicant's file because they now have fewer organizational and communication links with HRDC staff responsible for EI files. Another perceived problem is that HRD-NB does not fully understand the EI clientele, which is different from the social assistance clientele. For example, many EI clients are seasonal workers, so their need for employment programs is concentrated in certain periods of the year. HRD-NB staff did not realize that many EI clients do not require employment programs during the summer because they typically find employment independently in that season, yet require assistance during the winter. In addition, in the case study of Employment Assistance Services (EAS), HRD-NB staff expressed concerns about their lack of experience (and staff) for their new responsibilities, which include program administration and the management of contracts with third-party organizations.

The HRD-NB counsellors found that they were not reaching their target clientele because the counselling services they offer are voluntary and EI clients were not seeking counselling unless they were referred by HRDC staff. A functional referral system had yet to be implemented. Furthermore, the HRDC staff working in EI are behind in their work and do not have time to coordinate with HRD-NB. Another issue pertains to case loads. HRD-NB tends to judge the performance of the transferred personnel (who are still working in HRCCs) by comparing their case load with that of the HRD-NB staff who take care of social assistance. A problem with this comparison, however, is that they are not the same clients — clients on EI come much more often. So the transferred staff see a lot more people per day than the HRD-NB staff who take care of social assistance recipients.

Cooperation from Partners and New Partnerships

There has been a history of good cooperation between the federal and provincial governments in New Brunswick, and the LMDA has provided an opportunity to strengthen the partnerships among HRDC, HRD-NB and DOL. The New Brunswick Labour Force Development Board (with labour and management representatives) was also consulted regarding the LMDA implementation. In addition, consultations have been done with community organizations, though some respondents in the evaluation felt that these could be improved.

Views were mixed on the degree to which the LMDA has helped to create new partnerships up to this point. Some key informants and focus group participants perceived that new partnerships have been developed, including partnerships between: provincial departments and regional development associations and industry associations (e.g., trucking); SLG officials and industry/trade groups and the community colleges; and the DOL and Department of Economic Development, Tourism and Culture. In addition, case study respondents noted that they have received good cooperation and in some cases developed partnerships with various community and third-party delivery organizations. On the other hand, many respondents in the evaluation did not think that significant new partnerships have been developed at this stage.

Adequacy of Monitoring Measures

The evaluation results indicate that there are some serious problems related to the adequacy of current information systems for the monitoring of program delivery and results. There appears to be a need for clarification on accountability requirements, for a clear definition of valid results measures, and for better integration of the (currently incompatible) information systems operated by the three players. There are also related client privacy issues because client information will need to be shared among the players. Systems officials feel the need for direction on these points from program management.

In interviews, some HRDC officials expressed concern about the adequacy of the monitoring done by the Province, noting that accountability does not appear to be a primary concern of provincial authorities. For instance, some respondents had not received the quarterly reports on results which the Province is required to produce. Monitoring is also a problem for the Department of Labour in that it does not do case management from the perspective of workers, but rather only deals with employers. In addition, at the local level, planning is a crucial role for staff. In order to do this properly, however, good local-level information is needed but not currently available. This problem is compounded by the fact that the Province does less local-level planning than HRDC does, and the timing of planning differs between the two levels of government.

HRD-NB officials argued that there is a need for one integrated information system (or at least compatible systems) that all players can agree on, clear information requirements and agreement on how this information is to be used. A concern was also expressed about the lack of comparison or benchmark measures for results monitoring, given that the PBMs (in particular, SLG) are sufficiently different from the previous programs to make comparisons difficult. Skills Loans and Grants staff are able to track their own clients through their internal financial data system (which is useful for process evaluation), and SLG client monitoring is done primarily through the Department of Labour system and NESS, but also through the HRD-NB case management system.

Department of Labour officials complained that the Department never received an adequate explanation of the rationale/basis for the LMDA results targets. Accountability regarding LMDA results and financial information remains unclear. In the view of these officials, adapting the existing information systems to collect valid results measures, and related issues such as gaining security access to the federal systems, will be an ongoing problem for some time.

DOL respondents also observed that existing systems cannot currently provide accurate reports of activities at the local level. Moreover, there is a time lag between an employer hiring employees and entry of this information into the system, and results information can get lost due to the incompatibility between systems (e.g., the information might be entered into one system, but a report generated from another incompatible system). With respect to results, the employers are really "in the driver's seat" because they decide who they will take on as employees. Consequently, field staff with the DOL feel that they have little control over these results.

Like the key informants, focus group and case study respondents identified some problems with information monitoring systems. For example, HRD-NB staff have lost some screens to which they had previously had access, even though they have remained working at the same desk. Also, former federal employees now at HRD-NB are experiencing stress because they are told to keep the funds allocated to clients to a maximum amount per client, but cannot properly keep track of this in the absence of regular, accurate monitoring of client expenditures. In the case studies, several respondents noted that there is low awareness among program staff of the results targets and/or how these are being monitored. On a positive note, however, third-party monitoring of EAS clients was viewed as effective due partly to the fact that contractors are held accountable for the impact of their services on clients.

Remaining Issues for Implementation

The major outstanding issues for implementation of the LMDA were discussed earlier. These issues include: resolving the resource issues (e.g., for co-location); adapting and integrating the information and monitoring systems; defining accountability measures and resolving the privacy issues so that client information can be shared among the three departments; clarifying the roles and responsibilities of all staff, and improving communications on these matters both within and between levels of government; and completing co-location, which is discussed in the next section.

Some additional issues noted by respondents were as follows:

• the pan-Canadian programs and whether or not the federal government is going to devolve any further responsibilities to the Province;
• the income tax issue with Skills Loans and Grants (i.e., the fact that unemployed clients have to pay tax to Revenue Canada on their training assistance);
• completing the negotiations on the agreement for transferred staff's pension (i.e., negotiations between the federal Treasury Board and the provincial Department of Finance);
• issues related to meeting the needs of persons with disabilities and rural clients;
• improving the usefulness of Labour Market Information;
• how HRD-NB and DOL are going to case manage clients (i.e., clarifying which department is responsible for the client at various stages of case management and service delivery); and
• the need for the federal government to gradually let go of its previous role.

5.3 Co-location

Despite the fact that a number of barriers (e.g., disagreements over resources, inadequate and incompatible information technology and resistance to organizational change) have delayed the co-location process, the majority of respondents in Phase I of the formative evaluation had not lost sight of the ultimate goal of one-stop shopping and seamless delivery of services and believed that co-location will provide many benefits for clients. In Phase II of the evaluation, many of the same barriers were still being encountered, and while a few sites are partially co-located, only one site (in Fredericton) is fully co-located. While clients are not dissatisfied with sites that have undergone partial and full co-location, the HRSC personnel have not seen the kind of improvements in their business operations that would lead to improvements in service delivery and their confidence in the advantages of co-location is waning. Three sites with various configurations of co-location (from full to not at all) were evaluated in depth by case studies. Focus group discussions and interviews with key informants provided further insights into the issue of co-location at a number of different sites that were either partially co-located or not at all.

Obstacles to Co-location

One of the most frequently reported obstacles to successful co-location was the difficulty of resolving administrative and operational questions between the two levels of government (federal and provincial), and among the three different government departments. For example, differences exist between HRDC and HRD-NB regarding hours of operation, statutory holidays, norms for client confidentiality, client service methods (i.e., appointments versus walk-in service), and views on how primary reception should operate. Several respondents from both the case studies and key informant interviews thought that HRDC and HRD-NB should have harmonized their operations and client service methods before co-locating. Other respondents would also have preferred administrative and operational questions to be decided upon centrally before proceeding with co-location either partial or full co-location (questions like responsibility for administrative costs, space allocation, signage, how reception would function and responsibility for its associated costs).

A less frequently reported but important obstacle to the co-location process has been financial. Many of the buildings currently housing HRDC or HRD-NB are too small to accommodate all three departments. In addition, the departments often have long-term leases, moving and/or renovating costs have been deemed too high and, in some cases, HRDC and HRD-NB disagreed on the best location for a new co-located HRSC (e.g., a city centre location versus a lower rent location farther away). These financial obstacles are the principal reasons why two of the regional sites evaluated in the case studies during Phase II have not been co-located. One of those sites has been slated for co-location in the near future when funds become available, but at the other regional site, HRD-NB and HRDC have decided not to co-locate in part because of their long-term leasing arrangements.

Social and organizational obstacles to co-location were also found to exist. Lower levels of management and staff directly involved in program delivery identified cultural differences between departments which stem from the characteristics of the traditional clientele of each department and the types of services provided. Co-location has brought about a collision of these different organizational cultures and poses a few problems (e.g., different visions between departments on the type of service delivery model to be adopted, unwillingness of EI clients, social assistance clients and employer clients to mix; security measures of one department are not suitable for the image or type of service of another).

A lack of communications between departments was reported by both staff and clients as another barrier to co-location. Key informant interviews with staff revealed that staff found it difficult to obtain information on the other departments' activities and focus group respondents perceived that the departments did not communicate with one another on their programs and activities.

Technical obstacles such as incompatible information systems between the province and HRDC and a lack of adequate technical resources, such as fax machines and photocopiers for each department, were reported by staff members.

Resistance to co-location by staff and management has also been reported as an obstacle. There was reported low enthusiasm from federal employees to co-locate and DOL staff would have preferred to co-locate with the economic development groups rather than with HRD-NB and HRDC. Key informant interviews and one case study indicated that in several instances the province delayed making decisions on whether to co-locate or pulled out and then re-entered the co-location process illustrating their unwillingness to participate in the process.

Expected Benefits and Impacts of Co-location

Throughout the evaluation, the majority of respondents expected that co-location would produce a number of benefits and have an overall positive impact. The evaluation fieldwork indicated that where co-location had not yet been implemented, most respondents expected that service delivery would be improved by co-location. Clients liked the idea of one-stop shopping, and staff felt that client service would be improved by increased communications and co-operation between the government departments. Personnel in key informant interviews indicated that co-location had a great potential to improve client service if the sites became central information centres for all social programs and if new partners could be brought to the co-located sites (i.e., if more government services were offered under the same roof). They felt seniors and persons with disabilities would benefit from co-located sites which improved their access and that processes would be sped up by co-location.

Where full or partial co-location had taken place, the majority of respondents found that client service had either improved or had stayed the same. Clients reported that co-location provided one-stop service, easier access and greater convenience. Personnel noted that the pooling of resources for self-serve information services and for the resource centres has provided better services to the public, that some program applications (such as DOL's Partners program), are being processed much faster, and social assistance clients are receiving more information and better services. Many personnel thought however, that client service really had not been altered by co-location (either full or partial co-location).

A smaller proportion of respondents felt that service had declined (e.g., information on programs was difficult to obtain, employers did not know who to talk to, the number of visits clients had to make increased, and line-ups at reception were longer). More staff representatives than clients perceived a decline in service and those that did felt that this was mainly due to the transition period.

A second expected benefit of co-location was improved client flow, but this has apparently not been the case. Client flow has been disrupted for a number of reasons. Clients found the co-located layouts confusing, signage is lacking or confusing, and reception services lack adequate information and direction in order to efficiently assist clients and refer them to the appropriate department or person. A large proportion of respondents identified reception as a very important factor influencing client flows and that the infrastructure for reception should be revisited to make co-location work for all three government departments.

A third benefit that co-location was expected to produce was increased inter-departmental communication and increased access to information for staff on other departments' programs and activities. Many respondents who have experienced full or partial co-location found however, that there has been little inter-departmental communication or knowledge exchange and that the three departments are not able to share as much information on clients as they would like. Improving information exchange and co-operation between departments were identified by many staff representatives as ways of improving client flows and client service and of decreasing duplication and misuse of programs.

Respondents reported that co-location has had a negative impact on management and decision making processes because decisions must be made jointly by three departments at the co-located site. Administrative problems do not get resolved quickly and mutually satisfactory decisions are difficult to reach in a timely fashion.

Co-location has also created some problems in staff relations (for example, former federal staff regret losing their seniority in the devolvement to the province, and there may be some conflict among SAR Case Managers and EI Case Managers regarding job complexity and remuneration), but it has also led to sharing of resources for professional development.

In general, client representatives felt that co-location once fully implemented would benefit them while personnel representatives were more skeptical and felt that co-location had the potential to positively impact services only if co-operation between departments and direction on how to better implement co-location were improved.

5.4 Service Delivery

Effectiveness of Client Flows

In order to ensure convenient access to federal and provincial programs as specified in the LMDA, co-located employment centres were intended to be established in the province. At the time of this report, only one co-located site was in operation. In many of the other locations, DOL staff and the transferred staff (mostly former federal employees who are now HRD-NB case managers) worked from within an HRCC. Apparently, some of these changes have had an impact on client flow. One key factor is the reception area which in many cases is run by HRDC staff. Many respondents report that the referrals made by EI application staff to direct clients towards case management is not systematic, both in co-located and non co-located offices. It appears that this is primarily caused by organizational barriers and a lack of communication between HRDC staff (who process EI applications) and HRD-NB staff (who conduct case management). There is often a misunderstanding of the roles and responsibilities of the staff of the other departments, both in co-located (as mentioned earlier) and non co-located environments. These factors may explain why client flow is not optimized and partly explain why in many offices, there are fewer people participating in programs such as SLG (according to respondents). It is generally thought that co-location will improve client flow; the experience in the active co-located site tends to show, however, that organizational barriers can remain despite the fact that all three departments share a common building.

Client flow problems were reflected in some of the comments provided by focus group participants (program participants and employers). The majority found out about the program in which they participated through word of mouth or through another party. Many clients mentioned that the programs are not well known among the general population. Clients and employers generally agreed that the programs are not well advertised by the Departments. According to some staff members, many clients were confused about the roles and responsibilities of each Department, especially in the months that followed the transfer of the federal employees to HRD-NB.

Flexibility and Responsiveness to Client Needs

From many viewpoints, programs are believed to be more flexible than before the LMDA. EAS is considered to be very flexible because services are generally offered to all unemployed people not just EI claimants or reachback clients. The program is also flexible in allowing local HRCC/HRD-NB officials to choose what types of EAS are needed to best meet local needs.

The most frequently cited example of the flexibility of the new programs is SLG. The fact that the program is geared towards individual clients to meet their personal needs in the area of training is the main reason why SLG is considered to be more flexible than the previous block purchases. The program is also considered to be flexible because of the autonomy given to case managers, who can negotiate directly with clients and adapt funding to individual needs. On the other hand, employers and DOL staff felt that the program should be more flexible to allow for block purchases in some circumstances (e.g., a newly established employer looking for workers). Although the degree of consistency of the application of the program from one office to another has raised some concern, the interdepartmental consultation committee appears to have been effective in establishing a number of standards to guide local case management, according to interview respondents.

There is a general agreement that programs meet the needs of those who participate in them. Clients are generally satisfied with programs and services, although many find case management to be superficial. The delay of the implementation of the SLG loan component did cause some dissatisfaction. Access to services in both official languages is not a problem, according to clients (even though some offices have admitted experiencing difficulties maintaining bilingual staff in key front-line positions). Employers also report that programs meet their needs, and adjustment services participants are generally satisfied with the assistance provided by DOL staff.

Many respondents mentioned, however, that the current monitoring mechanisms are inadequate to assess the capacity of programs to adapt to local and provincial needs. The available information mainly comes from NESS, which provides mostly short term information on client participation and impacts. There is some concern about the reach of the programs in the province overall: many programs are limited to EI and reach back clients, which represent only a portion of people looking for work in the province.

Access to PBMs by Designated/Equity Groups

Even though there are no specific LMDA programs that target these groups, local HRD-NB staff mentioned that their offices target many groups such as youth, women, Aboriginals, persons with disabilities, people who have been out of the labour force for a long period of time, and seasonal workers. The focus on each of these groups varies from one region to another. Some local staff members said that many people from these groups participate in both generic programs (i.e., intended for all clients) or specific local initiatives such as specialized EAS for equity groups. A number of officials mentioned, however, that only adequate monitoring mechanisms could really allow them to assess the degree of participation of equity group members in programs at the local level.

Many people in these groups, namely youth, older women, and persons with disabilities, have limited access to programs because many of them are not EI claimants or reachback clients, according to staff. Findings also suggest that many do not realize that they are eligible to receive services such as EAS, even though they have never worked or have been out of the labour force for a long period of time. Again, it was mentioned that these services could be better advertised to reach target groups. Other respondents suggested that there are fewer programs and projects that target equity groups as a result of the LMDA.

A number of officials mentioned that reaching the Aboriginal population is a challenge for the province, especially on the reserves which were traditionally served by HRDC officers. It appears that the division of responsibilities between both orders of government has not been clearly sorted out both in the context of the implementation of the LMDA and in general.

Labour Market Information and Labour Exchange

Labour Market Information (LMI) and Labour Exchange are key information systems for clients, case managers and local officials. According to the LMDA, the national labour market information and labour exchange systems remain a federal responsibility. It was expected, however, that both levels of government would cooperate to provide and share information from these sources.

The full implementation of accessible LMI and Labour Exchange systems had not yet been completed at the time of this report. Technical problems have prohibited the access to these systems by some of the provincial staff. Some information, for example, is only available on the Internet, which many provincial employees in local offices did not have access to from their desktop. Implementation was also affected by the working relationship between the departments, which has not been optimal on this issue because of misunderstandings about the roles and responsibilities of each level of government.

Findings also suggest that the current Labour Exchange mechanism does not meet the needs of employers since most do not use it to find workers, and that people with disabilities (e.g., the visually impaired) can have difficulties accessing labour market information because of the computerized systems used to deliver it.

5.5 Use of Services

The results of the participant survey clearly demonstrate that participants used the resources from government offices (i.e., HRCC, Canada Employment Centre) to complement their participation in employment programs. When asked if they had used services from a government office when they became unemployed, 70 per cent of respondents indicated that they had. Those respondents who had accessed services were most likely to have done so at an HRCC (70 per cent). Respondents were much less likely to have accessed these services from an employment (16 per cent of respondents unable to distinguish the specific government office), HRD-NB (13 per cent) or DOL office (one per cent) (Exhibit 5.1). When asked what kinds of services they had used on their own at a HRCC, respondents were most likely to indicate having used a Job Bank kiosk (76 per cent). Participants also accessed reports on the local job market (40 per cent), information on training or education programs (36 per cent), a computer (33 per cent), and directories of local services or programs (32 per cent) (Exhibit 5.1).

Participants were also asked if they had received services from a case manager or employment counsellor from the government prior to their first employment intervention and 28 per cent reported that they had (Exhibit 5.2). EAS and Job Action participants were more likely than Rural Experience or Partners participants to have accessed these services prior to their first employment program (51 and 39 per cent versus 25 and 17 per cent respectively). Of those who had received services, respondents were most likely to have received employment counselling (40 per cent), and referrals to another organisation (36 per cent), and job search advice on an individual basis (35 per cent).

Fully 49 per cent of respondents who had received services from a counsellor prior to their first employment program indicated that they had developed an action plan with an employment counsellor and of these respondents, 83 per cent had completed the activities in their plan (Exhibit 5.3). Among the reasons given for non-completion of the action plan, respondents were most likely to indicate that their action plan was still in progress (50 per cent), they had found a job (11 per cent), there were no spaces or funding available for programs (nine per cent) or that they had changed direction and were no longer interested in pursuing the original plan (seven per cent).

5.6 Satisfaction with Services

Client Satisfaction

Participants were asked separately to rate various characteristics of the services they had received from a government office and from a community agency (Exhibit 5.4). Respondents who had received services from a government office were most likely to indicate that they were very satisfied (responded 6 or 7 on a 7-point scale) with the level of knowledge of staff (82 per cent), followed by the speed of service (74 per cent), the information on available programs and services (64 per cent) and the quality of service (55 per cent). Ratings of satisfaction tended to be higher for services received from a community agency, with the exception of information on available programs and services. These respondents were most likely to be very satisfied with the level of knowledge of staff (85 per cent), followed by the quality of service (78 per cent), the speed of service (77 per cent), and the information on available programs and services (61 per cent).

Questions more specifically related to program interventions generally received higher satisfaction ratings. Fully 89 per cent of clients indicated that they were very satisfied with the quality of education or training they received, while 85 per cent were very satisfied with the quality of the job with their employer.

When asked how well the assistance received met their needs, respondents were most likely to indicate that it had met their needs to a small extent (45 per cent versus 31 per cent to a moderate extent and 22 per cent to a large extent). Of all the services they had received, whether from an employment program, government office or elsewhere, respondents to the participant survey were most likely to report that a loan or grant for education or training (18 per cent)¹² and employment experience (16 per cent) was most important for them in getting the help they needed (Exhibit 5.5). It is interesting to note that the largest proportion of respondents indicated that they did not know which service was most important to them (28 per cent), suggesting that services may have had only limited importance to a minority of respondents or, conversely, that they were unable to choose among the multiple "important services" they received.

The majority of respondents to the participant survey (63 per cent) indicated that they had used HRCC or Canada Employment Centre services prior to April 1, 1997 (Exhibit 5.6). Respondents were most likely to indicate that the service before and after April 1, 1997 was the same (53 per cent), although a slightly higher proportion felt it was better after April 1, 1997 (26 per cent) than before this time (18 per cent). Similar results were obtained with respect to HRD-NB and DOL services, although only 30 per cent of respondents indicated that they had used these services before April 1, 1997 (Exhibit 5.6).

Employer Satisfaction

The results of the employer survey show that, in general, employers tend to be quite satisfied with the wage subsidy programs they have used. The majority of employer respondents were very satisfied with the program overall (72 per cent) and were somewhat less likely to report that they were very satisfied with the program participants overall (63 per cent) (Exhibit 5.7a and b). With respect to various aspects of the programs themselves, employers were most likely to be very satisfied with the quality, comprehensiveness and clarity of the information about the program (91 per cent), efficiency and knowledge of the program officer (87 per cent), and support and service provided by the office (87 per cent). The lowest satisfaction ratings were observed for the marketing and outreach of the program (68 per cent reported being very satisfied) and the non-reimbursement period (63 per cent reported being very satisfied).

Fully 45 per cent of the employers who responded to the survey indicated that they had been involved in a similar wage subsidy program prior to April 1, 1997. On average, these respondents had been involved in 6.2 programs before that time. When asked to rate the work skills, attitudes to work and learning, and overall job readiness of participants in the program they had used most recently, 40 per cent of employers who responded to the survey indicated that program participants from the most recent program were better (responded with a 5, 6 or 7 on a 7-point scale) than participants from similar programs they had used in the past. Forty-six per cent of these employers indicated that the participants were the same, and only nine per cent felt that they were worse (Exhibit 5.8).

Employers were also asked if they had used any government assistance office in New Brunswick prior to April 1, 1997 and 48 per cent of respondents indicated that they had. When rating the quality of service of their most recent visit to an employment office with their experiences prior to April 1, 1997, 52 per cent of respondents felt that the service was the same, while 26 per cent felt that it had improved and only 12 per cent felt that it had become worse (Exhibit 5.8).

5.7 Bilingual Service

A key evaluation issue involved whether LMDA programs and services are being delivered in both official languages and evidence from the formative evaluation suggests that this requirement has been met very well. All LMDA program participants surveyed for the evaluation were asked if they had been able to receive services, program information or actually participate in programs in the official language of their choice and fully 97 per cent were satisfied with this aspect of the service. Only one per cent of survey respondents indicated that this had not been the case (n=24).¹³ Of these respondents, 64 per cent (n=17) were unable to identify the program or service for which they were unable to receive services in the official language of their choice. Three respondents indicated that they had been unable to receive employment counselling and two had been unable to access self-service resources in their official language. Furthermore, two Entrepreneur respondents and one SLG respondent indicated they had been unable to receive either services, program information and materials, or actually participate in the program in French. One Job Action program participant and one SLG participant had been unable to receive these services in English.¹⁴

The language needs of employers were also met very well. Of those employers who had participated in employment programs prior to April 1, 1997, none indicated that they had been unable to receive services in the official language of their choice at that time. Of all respondents to the employer survey, only three indicated that they had not been served in their official language and two indicated that they had been unable to obtain materials in their official language after April 1, 1997.

The qualitative components of the evaluation support the conclusion that the language needs of program participants and employers are well met by programs and services currently offered under the LMDA. Of all the employers and participants who were interviewed for the case studies and focus groups, only one focus group participant indicated having had difficulty accessing services in the official language of their choice. Evidence from the case studies and key informant interviews suggests that only minor, temporary difficulties have occurred with respect to providing services in either official language (i.e., loss of bilingual staff who have not yet been replaced). It is likely that these temporary circumstances account for the few employers and participants who were unable to receive services in their official language.

5.8 Use of Other Services

When asked if they had used employment-related services from an agency other than HRD-NB or a Human Resource Centre (i.e., from a third-party service deliverer), only 16 per cent (n=250) of respondents to the participant survey indicated that they had (Exhibit 5.9a). These respondents were most likely to have received job search advice on an individual basis (27 per cent), employment counselling (21 per cent), job search advice in a group (20 per cent), or referral to a job (19 per cent). Forty-three per cent of those respondents who indicated having received services from another agency indicated that they had developed an action plan with a counsellor from the agency and of these respondents, 74 per cent had completed the activities in their plan (Exhibit 5.9b). The most common reasons for non-completion of action plan activities were that the action plan was in progress (28 per cent), they found a job (25 per cent), or that they had changed direction and were no longer interested in pursuing the original plan (22 per cent).

EAS participants were also asked if they had received services to help them find a job that were not delivered or sponsored through a provincial or federal government office, and only 15 per cent of these respondents indicated that they had. These respondents were most likely to indicate receiving listings of job postings or vacancies (35 per cent), information on the local job market (30 per cent), job search advice (29 per cent) or training and education programs (21 per cent) (Exhibit 5.10).

5.9 Suggestions for Improvement

When asked for suggestions of how to improve the wage subsidy programs with which they were involved, employers were most likely to suggest a better match of participants to businesses (21 per cent), a longer period of wage subsidy (20 per cent), more timely or faster reimbursement and approval (15 per cent) and more wage subsidy (13 per cent) (Exhibit 5.11). It is interesting to note that 27 per cent of employer respondents were unable to offer a suggestion of how the programs might be improved, thus suggesting that for a large minority of respondents the current programs suit their needs fairly well.

Information from the qualitative components of the study (i.e., key informant interviews and focus groups with clients and employers), provided a number of suggestions of how the PBMs might be improved. These suggestions included broader eligibility criteria, more freedom to choose job candidates, better promotion, increased relevance to client and employer needs, adding the loan component to the SLG program, providing clearer guidelines for the administration of the SLG program, more follow-up after programs start, more courses and more timely courses for the Entrepreneur program, and faster reimbursement and approval processes for employers.

The following chapter presents information that describes how each of the PBMs is being used, as well as information from the surveys profiling the characteristics of clients who make use of the programs and services offered under the LMDA as well as some information on employers. Additional profile information on employers is presented in Appendix C. Please note that all analyses by LMDA program type are weighted by age and sex. Overall analyses (i.e., collapsed across program type) are weighted by age, sex and program type.

6.1 Partners

Clients

In the week prior to starting their first employment program within the time period defined for the study¹⁵, Partners program respondents were most likely to indicate that they were interested (responded with a 6 or 7 on a 7-point scale) in entering the workforce on a full-time or part-time basis (95 per cent), followed by entering an education or training program (47 per cent) and starting their own business (25 per cent).

In their Partners program jobs, respondents were most likely to indicate they worked as trades, transport or equipment operators (29 per cent), in sales and service occupations (22 per cent) or in clerical jobs (13 per cent) (Exhibit 6.1a). The majority of respondents indicated working for private employers (86 per cent) and that they worked an average of 31 to 40 hours per week (66 per cent). The reported gross weekly pay for these respondents was typically between $251 and $500 (62 per cent) (Exhibit 6.1b).

Exhibit 6.2 shows that relatively few Partners program participants indicated leaving their wage subsidy jobs prior to the completion of the full period of funding (15 per cent). Respondents' primary reasons for leaving were to take another job or start a business (41 per cent), they were asked to leave (13 per cent), the business failed or closed (10 per cent), for health reasons (nine per cent) and because of financial difficulties or an insufficient wage (eight per cent).

Employers

Partners program employers generally were not repeat users of wage subsidy programs. Partners program businesses tend to be small, private firms that operate in an eclectic mix of industries, as demonstrated by the jobs into which program participants were most commonly hired (i.e., trades, transport and equipment operators, clerical, and sales and service). Partners program businesses have, for the most part, shown recent growth and are expected to continue this trend.

Partners program employers were most likely to have heard of the program through word of mouth (32 per cent), from having visited a government office (29 per cent), or through some other contact with a government department (10 per cent). These results are presented in Exhibit 6.3.

Exhibit 6.4 presents Partners employers' primary reasons for participation in the Partners program. Partners program employers were most likely to have participated in the Partners program because of job creation (44 per cent). Other commonly cited reasons for participation include the wage subsidy itself (38 per cent), their belief in giving opportunities to people who need them (13 per cent), an upturn in business (13 per cent), and a need for employees (10 per cent).

On average, Partners program employers reported that employees hired under the program worked an average of 39.4 hours a week, a figure which was slightly lower than the overall average of 42.7 hours, and were paid an average of $8.60 an hour including the wage subsidy (the overall average was $9.00). Sixty per cent of Partners employers indicated that they hired participants into permanent full-time year-round jobs and another 26 per cent indicated that they hired participants into short-term full-time positions (Exhibit 6.5). The mean number of participants hired by Partners employers into permanent full-time year-round jobs is above the overall average (mean=16.6 versus mean=14.3 overall), and the proportion of Partners employers who hire into these positions is much higher than the overall average (60 versus 30 per cent).

During the program, 30 per cent of Partners employers hired participants for jobs in trades, transport, equipment operation and related areas, followed by 27 per cent who hired participants for clerical jobs, and 25 per cent who hired into sales and service occupations (Exhibit 6.6). Twelve per cent of Partners employers indicated hiring participants into other jobs, and of these respondents, 36 per cent hired for unskilled labour jobs, 14 per cent hired employees into each of business, finance and administration, natural and applied sciences, and health related fields. The mean number of participants hired by Partners employers was considerably lower than the overall mean for trades, transport and equipment operators (2.9 versus 7.3) and occupations unique to primary industries (1.2 versus 6.0).

6.2 Entrepreneur Program

In the week prior to starting their first employment program, Entrepreneur program respondents were most likely to indicate that they were very interested (responded with a 6 or 7 on a 7-point scale) in starting their own business (92 per cent). Respondents were moderately less likely to indicate that they were very interested in entering the workforce on a full or part-time basis (72 per cent) or in entering an education or training program (49 per cent).

The mean initial capital investment in Entrepreneur businesses was $36,583 and these businesses were most likely to have started in "other" service industries (such as golf courses, decorating, salons, etc.) (22 per cent), retail trade (14 per cent), and manufacturing (12 per cent) (Exhibit 6.7a). The majority of Entrepreneur businesses operate year-round (85 per cent operate 11 to 12 months) and only 14 per cent of respondents indicated that their businesses were seasonal or casual (operating 10 or fewer months of the year). Entrepreneur participants were most likely to indicate that they worked more than 40 hours per week on their businesses (68 per cent) and that they earned $250 or less per week (35 per cent) (Exhibit 6.7b). While these earnings may seem low, it is important to note that on average, Entrepreneur participants earned $448 per week. Only 18 per cent of Entrepreneur program participants worked at other jobs in addition to their business and for these respondents, their Entrepreneur business accounted for an average of 24.6 per cent of their personal earnings (Exhibit 6.8).

More than half of all Entrepreneur program respondents reported that they took training to help them start their business (51 per cent). These respondents were most likely to indicate that they received training in marketing (68 per cent), management (67 per cent), bookkeeping, accounting and taxes (66 per cent), financing and financial planning (52 per cent), and advertising, promotion and display (47 per cent) (Exhibit 6.9), and were most likely to have received more than 40 hours of training overall (56 per cent).

When asked if their business was still in operation, only 13 per cent of Entrepreneur program respondents indicated that it was not (Exhibit 6.10). These respondents were most likely to indicate that the business was no longer operating because it was not profitable (32 per cent), they experienced financing problems (20 per cent), for personal reasons (15 per cent) and because of other problems, such as marketing or legal issues (11 per cent).

6.3 Job Action

Clients

Job Action program respondents were most likely to indicate that they were interested (responded with a 6 or 7 on a 7-point scale) in entering the workforce on a full-time or part-time basis (93 per cent) in the week prior to starting their first employment program. They were much less likely to be very interested in entering an education or training program (51 per cent) or starting their own business (15 per cent) at that time.

In their Job Action program jobs, respondents were most likely to indicate they worked in clerical positions (17 per cent), in occupations unique to primary industry (13 per cent), or as teacher's aides (12 per cent) (Exhibit 6.11a). The majority of respondents indicated working for public sector employers (55 per cent) and were most likely to report working an average of 31 to 40 hours per week (89 per cent). The reported gross weekly pay for the majority of Job Action respondents was between $100 and $250 (56 per cent) (Exhibit 6.11b).

Exhibit 6.12 shows that relatively few Job Action program participants indicated leaving their wage subsidy jobs prior to the completion of the full period of funding (11 per cent). Respondents' primary reasons for leaving were to take another job or start a business (53 per cent), they were asked to leave (12 per cent) and transportation problems (seven per cent).

Employers

The majority of Job Action program employers who responded to the survey were repeat users of provincial wage subsidy programs. Their businesses tended to be medium sized (40-50 staff), public sector firms that operated in a range of industries, as demonstrated by the jobs into which program participants were most commonly hired (i.e., sales and service, clerical, and trades, transport and equipment operators). Job Action businesses have, for the most part, shown recent growth and expect to continue this trend. These employers tend to use the Job Action program to staff short-term full-time positions.

As shown in Exhibit 6.13, Job Action program employers were most likely to have heard of the program by visiting a government employment or assistance office (27 per cent), through word of mouth (15 per cent), from a flyer, pamphlet or poster (14 per cent) or through some other contact with a government department (12 per cent).

Job Action employers' most common reason for participation in the program was job creation (39 per cent). Other commonly cited reasons for participation include the wage subsidy itself (32 per cent), their belief in giving opportunities to people who need them (23 per cent), and a need for employees (12 per cent) (Exhibit 6.14).

On average, Job Action program employers reported that employees hired under the Job Action program worked an average of 38.6 hours a week, a figure which was slightly lower than the overall average of 42.7 hours, and were paid an average of $6.60 an hour including the wage subsidy (the overall average was $9.00). These employers were most likely to report that they hired program participants into short-term full-time positions (71 per cent), but were considerably less likely to hire participants into other types of positions (Exhibit 6.15). Job Action employers were considerably less likely to hire participants into permanent full-time year-round positions (five versus 30 per cent overall) and hired a much smaller mean number of participants into these positions (mean=2.0 versus mean=14.3 overall). These results are not surprising, however, when we consider that the Job Action program is designed for short-term or project based jobs.

Job Action program employers were most likely to report that they hired program participants into sales and service occupations (35 per cent), clerical jobs (31 per cent), and trades, transport, equipment operators and related professions (18 per cent) (Exhibit 6.16). About one third (33 per cent) of Job Action employers indicated hiring participants into other jobs. Of these respondents, employees were reported to have been hired into occupations in general labour (31 per cent), social science, education and government service (16 per cent), arts, culture and recreation (13 per cent), health related fields (13 per cent), management (nine per cent), business, finance and administration (six per cent), and trades (three per cent). Job Action employers reported hiring a slightly higher average number of participants into clerical (mean=2.3 versus mean=1.7 overall) and sales and service occupations (mean=2.8 versus mean=2.2 overall).

6.4 Skills Loans and Grants

Skills Loans and Grants (SLG) program respondents were most likely to indicate that they were interested (responded with a 6 or 7 on a 7-point scale) in entering a training or education program (86 per cent) in the week prior to starting their first intervention. They were only slightly less likely to be very interested in entering the workforce on a full or part-time basis (82 per cent) and were much less likely to be very interested in starting their own business at that time (19 per cent).

SLG respondents were most likely to indicate that the training programs they took under the SLG intervention prepared them for jobs in trades, transport or equipment operation (24 per cent) or in IT and computers (17 per cent) (Exhibit 6.17a). Respondents were most likely to report that they received their training from a community college (58 per cent) or private training institute (30 per cent). A large proportion of respondents indicated that their training program was less than six months in duration (45 per cent), and sizeable proportions were in training programs that lasted six months to a year (32 per cent) or more than a year (23 per cent). The average length of training programs was 42.3 weeks and the average number of hours per week that respondents spent in their programs was 36 hours (Exhibit 6.17b).

Close to half of all respondents indicated that their training program involved on-the-job training or work placement (48 per cent). This component of participant training lasted an average of 7.8 weeks. On average, participants contributed $3,057 to the cost of their training and were most likely to indicate that they contributed $1,000 or less (28 per cent), although 20 per cent of SLG participants did not contribute to the cost of their training at all (Exhibit 6.18).

The vast majority of SLG program participants either completed their training program (62 per cent) or were still in the process of finishing their training at the time of the participant survey (31 per cent) (Exhibit 6.19). Of those who left their program before it was completed (six per cent), the primary reasons were to take a job (36 per cent), financial difficulties (21 per cent), and family or personal problems (10 per cent).

6.5 Employment Assistance Services

In the week prior to starting their first intervention (within the time period under investigation in this study), Employment Assistance Services (EAS) program respondents were most likely to indicate that they were very interested (responded with a 6 or 7 on a 7-point scale) in entering the workforce on a full or part-time basis (90 per cent). A small majority were very interested in entering an education or training program (63 per cent) and relatively few were very interested in starting their own business (25 per cent).

The most frequently mentioned types of assistance that EAS participants reported receiving from third-party service providers were employment counselling (29 per cent), job search advice in a group (29 per cent), job search advice on an individual basis (23 per cent) and a referral to a job (20 per cent) (Exhibit 6.20a). Interestingly, three per cent of respondents indicated that they did not receive any services and 20 per cent were unable to indicate what services they received.

Fewer than half (41 per cent) of the EAS participants who responded to the survey reported that they had developed an action plan with an employment counsellor or case manager and of these participants, 75 per cent indicated that they completed the activities in their plan (Exhibit 6.20b). The reasons given for non-completion of an activity plan were that the respondent found a job (60 per cent), the action plan was in progress at the time of the survey (24 per cent) and the respondent was no longer interested in pursuing their plan (eight per cent).

6.6 Rural Experience (funded under Research and Innovation)

Clients

When asked to rate their interest in undertaking each of several employment initiatives in the week prior to starting their first employment program, Rural Experience program respondents were most likely to indicate that they were very interested (responded with a 6 or 7 on a 7-point scale) in entering the workforce on a full-time or part-time basis (94 per cent). These respondents were much less likely to indicate that they were very interested in entering an education or training program (43 per cent) or in starting their own business (26 per cent) at that time.

Rural Experience program respondents were most likely to indicate that they worked in occupations unique to primary industry (31 per cent), in trades, transport and equipment operation (21 per cent), as general labour (14 per cent) and in sales and service occupations (11 per cent) (Exhibit 6.21a). The largest proportion of respondents indicated working for private employers (42 per cent) and were most likely to report working an average of 31 to 40 hours per week (87 per cent). The reported gross weekly pay for the majority of Rural Experience respondents was most likely to be between $100 and $250 (51 per cent), with a mean wage of $289 (Exhibit 6.21b).

Relatively few Rural Experience program participants indicated leaving their wage subsidy jobs prior to the completion of the full period of funding (15 per cent). As shown in Exhibit 6.22, respondents' primary reasons for leaving were to take another job or start a business (52 per cent), to return to school (eight per cent), transportation problems (six per cent), and disability or maternity leave (five per cent).

Employers

Roughly half of Rural Experience employers were repeat users of wage subsidy programs. These businesses tended to be medium sized (50-60 staff), private sector businesses that operated in primary industries or within trades, transportation and equipment operation. Overall, Rural Experience businesses have shown recent growth and are expected to continue this trend. These employers tend to use the Rural Experience program to staff short-term full-time positions, which is the purpose of the program.

Rural Experience employers were most likely to have heard of the program through contact with a government department other than HRD-NB or DOL (18 per cent), from a flyer, pamphlet or poster (16 per cent), through word of mouth (13 per cent), or from their MLA or minister's office (11 per cent) (Exhibit 6.23).

Rural Experience employers' most common reason for participation in the program were job creation (55 per cent). Other commonly cited reasons for participation include the wage subsidy itself (23 per cent), their belief in giving opportunities to people who need them (13 per cent), a need for employees (12 per cent), and an upturn in business (10 per cent) (Exhibit 6.24).

On average, Rural Experience program employers reported that employees hired under the program worked an average of 50.2 hours a week, a figure which was considerably higher than the overall average of 42.7 hours, and were paid an average of $11.20 an hour including the wage subsidy (the overall average was $9.00). These employers were most likely to report that they hired program participants into short-term full-time positions (62 per cent), and permanent full-time year-round positions (16 per cent) (Exhibit 6.25).

Employers who hired into short-term full-time positions hired an average of 9.4 participants into these positions, followed by an average of 7.1 participants hired into permanent full-time year-round jobs and 5.9 participants hired into permanent full-time seasonal positions.

Rural Experience program employers were most likely to report that they hired program participants into occupations unique to primary industry (29 per cent), jobs in trades, transport and equipment operation (20 per cent), clerical occupations (19 per cent), and sales and service jobs (15 per cent) (Exhibit 6.26). Twenty-six per cent of Rural Experience employers indicated hiring participants into "other" jobs. Of these respondents, employees were most likely to have been hired into general labour jobs (52 per cent), social science, education and government service (20 per cent), and arts, culture and recreation jobs (12 per cent). The mean number of participants hired by Rural Experience employers was considerably higher than the overall mean for all three employer groups (mean=9.2 versus mean=5.2 overall).

6.7 Adjustment Service Initiative

The following is a brief profile of the six Adjustment Service Initiative (ASI) mini case studies that were completed as part of the study and is not meant to be representative of all ASI activities. Organisations with which the ASI mini case studies were conducted consisted of two community agencies (one public and one non-profit), as well as four private sector businesses. Two of the private sector firms required the ASI to promote the development of their businesses, while the remaining two required the ASI for the purpose of downsizing. The communities that were the focus of two of the ASI mini case studies included a relatively small community of 15,000 and a larger community of 50,000. The two firms that were dealing with downsizing issues tended to be fairly large (200 to 500 employees), whereas the firms for which employment development was the focus of the ASI were both quite small (eight to 12 employees). The industries within which organisations involved in these case studies operated varied from social services and government (for community ASIs), to manufacturing and primary industries (for ASI firms).

In only one ASI mini case study did the respondents indicate that there was not representation from all stakeholders and only one committee chair did not have previous experience with similar initiatives. In general, the ASI committees lasted approximately one year and case study respondents were unanimous in reporting that the ASIs had achieved their objectives.

In this chapter, we present findings on the success of the PBMs to date. Two main kinds of outcomes are discussed. In the first part of this chapter, we observe the extent to which the Cda/NB LMDA results targets have been met. In the remaining parts of this Chapter, we present evidence on impacts of the PBMs on participants, employers and communities.

It should be noted that the measures presented in this chapter are preliminary only. Moreover, the short-term nature of the period of investigation in this formative evaluation may favour interventions such as Partners and Entrepreneurs in which employment outcomes are often immediate, and which may be biased against interventions such as job search training assistance which have a longer "gestation" period. More definitive measures will be presented in a summative evaluation, when there will be more time available to detect impacts.

7.1 Results Targets Attainment

The Cda/NB LMDA, like all provincial labour market agreements, specifies results targets for EI reforms in three areas: participants returning to work, active EI claimants served,¹⁶ and EI benefits savings. The first column of Exhibit 7.1 presents the exact targets for the 1997/98 fiscal year for all active benefits and measures under the Cda/NB LMDA. In the second column are presented the actual results as in the 1998 EI Monitoring and Assessment Report.¹⁷ In the third to fifth columns, we use survey data collected for this evaluation to "validate" and update the first of the results (returns to work)¹⁸ and administrative data to look at the other two result indicators. Note that column two figures cover a period up to March 31, 1998, which is why we show employment rates for two different periods in columns three and four: before April 1, 1998 and on or after April 1, 1998. Note as well that total column includes amounts for those whose intervention end date was unknown, which is why it does not represent a sum of the previous two columns.

Note further that the figures in the first two columns of Exhibit 7.1 apply to all EI active benefits and measures under the Cda/NB LMDA including services such as individual counselling interviews and group services¹⁹ which EI clients can use independently of the benefits and measures being evaluated. On the other hand, the figures in the third column apply only to the six benefits and measures under study in this evaluation. For this reason, these figures should not be compared.

View EXHIBIT 7.1

Return to Work Results

With respect to the employment results target (participants returning to work following intervention), column two of panel 1 of Exhibit 7.1 indicates that the 1998 EI Monitoring and Assessment Report found that about 70 per cent of this target was attained (5546/7947). We confirmed this figure using the HRDC Results File administrative data. In the third and fourth columns, we present the results of our validation and update exercise of this result, which consisted of applying, for the two different time periods, (i) three employment rates derived from the survey results, to (ii) the count of the total number of PBM participants from the administrative data (8,881 up to April 1, 1998 and 8,768 on and after April 1, 1998). The three employment measures (with the values for the two different time periods in brackets) are: (1) employed at all following the intervention (71.7 and 51.9 per cent); (2) employed for 12 consecutive weeks following the intervention (70.5 and 49.2 per cent); and (3) currently employed (44 and 37.3 per cent). Note that the second employment measure — employed for 12 consecutive weeks — comes closest to the Cda/NB LMDA measure. Comparing column 2 to column 3 figures, and noting that the latter applies to a set of measures that excludes additional employment services as noted above, indicates that the New Brunswick administrative data (which the former is based on) appear to under-estimate employment outcomes.²⁰ The EI Monitoring and Assessment Report suggested that under-estimation may be due to the fact that the data systems used to track clients and the exchanging of data were not fully operational during the 1997/98 fiscal year. In the fourth column we update the employment counts for the period subsequent to the March 31, 1998 fiscal year and in the fifth column we provide the totals for the two fiscal years.

Active EI Claimant Share Results

In calculating this result, the EI Monitoring and Assessment Report included Rural Experience and EAS participants. Consequently for this evaluation report, these participants were also included (see second row of panel 2 in Exhibit 7.1). The majority of participants in these two programs are reachback clients. The Canada/New Brunswick Labour Market Development Agreement actually states that this accountability result is to be applied to Provincial Benefits only, and not to Measures. Since Rural Experience is funded under Research and Innovation (a Measure) and EAS is a Measure, this accountability result should be calculated excluding these two programs, as presented in the first row of panel 2 in Exhibit 7.1.

With respect to the active claimant share target, the first row of panel 2 in Exhibit 7.1 indicates that this target, too, has not been met. (In column 2, there is no entry because the 1998 EI Monitoring and Assessment Report provided only a national proportion result of 82.5 per cent, the reason being that administrative data systems were said to have not been in place to adequately measure this proportion.) In the 1997/98 fiscal year, 56 per cent of the participants in the Provincial Benefits under study were active EI claimants, based on population administrative data.²¹ Though once again we cannot really compare this result to the target, the proportion who were active claimants in the counselling and information sessions not under study would have to have been exceptionally high for the total share to be on target. Column 4 shows that the share has fallen to 41 per cent in the 1998/99 fiscal year, and the overall percentage of active EI claimants was 47 per cent. Management decisions about the targeting of programs will necessarily impact on the degree to which this (and other) results targets are achieved.

Unpaid EI Benefits Results

To check the third target — unpaid EI benefits — we first show the results presented in the 1998 EI Monitoring and Assessment Report in the third panel, second column of Exhibit 7.1. This report found that for all Cda/NB LMDA benefits and measures, there were $13.1 million in unpaid EI benefits in the 1997/98 fiscal year, which was barely one-half of the target. Once again, we confirmed this figure on the basis of the Results File.

As part of this evaluation, we computed unpaid EI benefits for PBMs as well but excluded services that were used by respondents exclusive of the PBMs under study. The results of this analysis for the 1997/98 and 1998/99 fiscal years and the total for both fiscal years including those with an unknown EI end date, are presented in the third, fourth, and fifth columns, respectively, of the third panel of Exhibit 7.1. In each cell of those columns in that panel, we show two rows of figures, corresponding to two different approaches used to compute unpaid EI benefits. The first is the "standard" HRIB approach based simply on the difference between paid and eligible EI benefits of claimants returning to work before the end of their benefit period.²² The second is the alternative ARC method,²³ which attempts to link unpaid EI benefits explicitly to a return to work. The main difference between the two approaches, then, is that the standard method attributes all unpaid EI benefits to a return to work, regardless of the reason why EI benefits were not paid. In contrast, under the ARC approach, the computation is linked to a return to work (RTW). As it turns out, the ARC method yields a smaller amount of savings because not all unpaid EI benefits can be attributed to a return to work. The standard approach can incorrectly identify a RTW when benefits are reduced for non-work reasons such as severance pay received, vacation pay, disentitlement, overpayments, etc..

To elaborate, under the first approach, unpaid EI benefits were computed by summing up the amount of savings for each participant, as indicated by a variable on the HRIB dataset received from HRDC. This amount was computed (by HRDC) for each individual as the difference between (1) EI benefits eligible and (2) EI benefits paid, which in turn was computed as the benefit rate multiplied by the difference between (1) the entitlement weeks and (2) the difference between the EI weeks paid and weeks paid in the RTW period. For each individual on the file, we compared the dates of each EI claim period to their interventions' start and end dates. Only EI claims overlapping with an intervention under study were included in the calculation; those claims not associated with counselling and group information sessions which were used exclusively by clients did not figure in the calculations. The majority of LMDA participants were single-intervention users, so for these individuals the savings derived from the EI claim were easily linked to the "most recent" intervention. In the case of multiple-intervention users, unpaid benefits associated with EI claims related to the different interventions were added up.

In the total population of 17,548 participants on file, 4,626 clients had EI claims linked to an intervention (i.e., excluding counselling and information session interventions as discussed above) and had reported unpaid EI benefits greater than $0. The total savings for these participants amounted to $14,145,474.

However, this is a figure that covers the better part of two fiscal years, i.e., participants whose intervention ended in either the 1997/98 or the 1998/99 fiscal year. Thus, in row one of columns three and four of the third panel of Exhibit 7.1, we show the savings realized separately for clients from the two fiscal years with known intervention end dates. We show in column three that only $7,933,245 were unpaid EI benefits realized in 1997/98. Adding in the some $62,000 savings (not shown) from individuals with a missing intervention EI end date, still brings the total savings for the 1997/98 fiscal year to less than $8 million. Row one of the fourth column indicates that about $6.15 million were realized in 1998/99 fiscal year, which brings the cumulative total to the some $14.1 million (column 5) including those with unknown intervention end dates as indicated above.

The alternative method of calculating unpaid EI benefits, as suggested by ARC, was based on the HRDC Status Vector file, which contains information on EI eligibility and benefits paid, and the Results File, which contains return to work (RTW) flags. The results of this exercise are presented in the second row of columns three to five in the third panel of Exhibit 7.1. In general terms, savings were computed as the difference between (1) EI benefits paid after the RTW date, and (2) remaining eligible EI benefits. In some cases, wages paid are such that EI benefits could be paid but at a reduced rate.

In order to calculate unpaid EI benefits under the ARC approach, the following detailed steps were taken for each individual:

• the RTW date was subtracted from each of the individual's EI-claims' start date, with the condition that the former date not be later than the latter date;
• the difference between the RTW and EI-claim start dates in step 1 (call it the "pre-period") was compared to the length of entitlement period related to this claim and, where the length of this pre-period was greater than the entitlement period itself, a value 0 was ascribed to the pre-period;
• the pre-period (in weeks) was then multiplied by the respective EI benefit rate (expressed as dollars per week), thus arriving at the total amount of EI benefits paid at the start of the RTW;
• the amount of benefits paid at the time of the RTW was further subtracted from the total EI eligible benefits, thus arriving at the amount of remaining EI benefits in the entitlement period;
• benefits paid up until the RTW were then subtracted from the total amount of benefits paid during the claim (i.e., including benefits paid after the RTW), thus calculating amount of EI benefits paid after the RTW; and
• finally, EI benefits paid after the start of the RTW (from step 5) were subtracted from the remaining eligible EI benefits (from step 4) to arrive at EI benefits unpaid for each individual.

The calculation was initially performed on the 6,144 persons on the Results File who participated in the six PBMs under study (i.e., excluding services that were used exclusive of those PBMs) and matched with the participant population from the main file. Missing information reduced the number of cases to 5,972, however. Further, another 3,800 individuals were assigned zero savings because the date of the RTW was beyond the entitlement period. This left 2,172 individuals who had "valid" dates, i.e., a RTW flag that was dated before the end of their entitlement period, and therefore potentially who could generate savings. Summing the computed savings across these individuals produced a total of $11,117,584 in unpaid EI benefits. Assigning zero savings to clients with negative savings (i.e. clients whose EI benefits had been extended), increases the total amount of savings calculated using ARC method to $11,260,748.

Once again, we differentiated clients for the 1997/98 and 1998/99 fiscal years because the targets specified for this evaluation were for 1997/98. Row two of the fourth column of the third panel of Exhibit 7.1 indicates that just over $6.5 million in unpaid EI benefits were realized in 1997/98 for the six PBMs under study. Adding in the $318,000 for persons whose EI intervention end date is not known (not shown) brings the savings up to barely $7 million for this fiscal year, even more short of target than using the HRIB approach. The fourth and fifth columns of row two of the third panel of Exhibit 7.1 indicate that savings from the 1998/99 fiscal year brought the total saved over the two fiscal years to the $11.3 million²⁴ as mentioned above.

Additional Return to Work Results

To amplify the return to work results, we present Exhibit 7.2, which compares incidence of 12 consecutive weeks of post-intervention employment across program types and claimant status. Though the employment results target was not specified according to program type and claimant status and though it was specified in terms of an absolute number not a percentage incidence, it is instructive to observe how the rate varies across the different groups.²⁵ First, the results indicate that active EI claimants were more likely to occupy jobs lasting 12 consecutive weeks than reachbacks, particularly in SLG and EAS programs. Second, the employment rate was particularly high for Partners and Entrepreneurs.

7.2 Impacts on Participants

Impacts on clients were measured using two approaches: (1) clients' and employers' subjective ratings of the importance of the help they received in obtaining employment; and (2) clients' objective labour-market outcomes, as revealed by their actual status. Both the subjective ratings and objective outcomes of PBM clients are compared with the rated and actual outcomes for the comparison group.

In interpreting the results based on participant survey data as presented in this chapter, the following should be noted. First, the results for participants overall are weighted by sex, age and program type; the participant results presented by program type are weighted by age and sex only; and the comparison group results are weighted by age, sex and the theoretical time of intervention. Note also that the sample size presented in the exhibits is unweighted. If second, the analysis of participant survey response data consisted of cross-tabulating post-intervention labour-market outcomes by type of intervention (i.e., benefit or measure) and by the characteristics of the participants. Participant outcomes are also compared according to EI-claimant and reachback status and compared to comparison group outcomes. Note, however, that, because a comparison group was not selected for the reachback group, the comparison between participants and non-participants is restricted to PBM participants who were active EI claimants. Note also that, while we observe differences in outcomes by characteristics of participants and program type, in such bivariate comparisons we are unable to control for pre-existing differences among participants and between the participant and comparison groups. For this reason, multivariate analysis was conducted where there were controls for these variables; the results of this analysis are presented in Chapter Eight.

The client outcomes we examined are in the following areas: employment, joblessness, job-search behaviour, income-support utilization, and attitudes and skills. As in Chapter Six, participant survey results for all participants and cross-tabulated by claimant status (active versus reachback) are weighted by sex, age and program type; participant survey results cross-tabulated by program type and comparison group survey are weighted by age and sex only. Employer survey results as they pertain to participants are unweighted.

Employment Outcomes

Employment for clients is the ultimate goal of PBM assistance. Employment impacts were measured in this evaluation using evidence gathered in the focus groups and case studies, as well as the employer and client surveys. In this section, we first present qualitative evidence on employment outcomes and then discuss a large number of employment outcomes based on survey evidence as follows: perceived importance of intervention on employment, employment rates, retention rates, non-completion, employment stability, employment status, hours and wages. Included with this discussion are results on related measures of retention, non-completion, and incrementality.

Qualitative Evidence on Employment Impacts

In the focus groups with clients, the general conclusion drawn was that, without the PBMs the clients had participated in, they would not have been in the jobs they were in or have started the businesses they started under the programs. In the focus groups with employers, it was observed that a number of clients had been hired by participating employers following the wage subsidy. It was also thought that many would in the future benefit in this way from improved skills and confidence resulting from their participation in the PBMs. However, a drawback is a lack of jobs in the local economy.

Positive employment outcomes were also identified in at least four of the case studies conducted for this evaluation. In both of the case studies of Adjustment Services Initiative (ASI) "down-side" adjustment situations (plant closings), we found that the committee formed and its recommendations were of benefit to the workers concerned. In one, committee deliberations and recommendations resulted in the establishment of an on-site employment resource centre and in about one-half of affected workers finding alternative employment, some of which was in an area different from their previous career. In addition, several workers returned to get their high-school certificates, which should result in employment in the future. In the other "down-side" adjustment situation, the ASI committee's work led to the establishment of an on-site drop-in centre. About eight in 10 of affected workers were able to return to the workforce.

In two other case studies, employment outcomes for clients were also identified. First, in the SLG case study, benefits from the intervention came in the form of jobs for several clients, some with better jobs. Second, in a case study of an ASI community development adjustment committee, the fact that the participating organization had become more business oriented as a result of the committee's work enabled it to be more socially proactive and to place clients from the community in greater numbers and diversity.

Perceived Importance of Intervention on Employment

The first approach to measuring employment outcomes in the participant survey was to directly ask participants employed at the time of the survey to indicate the extent to which they thought the employment benefit or measure enabled them to obtain their job. Respondents were asked to rate the importance on a seven-point scale ranging from one representing "not at all important" up to seven representing "extremely important", with responses of 6 and 7 re-grouped and identified as "important' in the results presented below. This question was addressed to only those who had a job in the post-intervention period and to those who were not "continuers", i.e., wage-subsidy participants who were hired by their host employer at the end of the subsidy and Entrepreneur participants whose business continued beyond the end of the assistance period. It was assumed that these continuers would have responded "extremely important" to this question, had they been asked it. Including these responses yields the results shown in Exhibit 7.3.

The results indicate that just over one-half of participants indicated (or it was inferred) that their assistance was important (reporting six or seven on a seven-point scale) in getting their job. Entrepreneur and Partners played an important role in obtaining their current job. For the other PBMs, between 38 and 51 per cent of respondents said the program had been helpful in getting their job. Not shown is the fact that the proportion saying the intervention played an important role in obtaining the current or most recent job hardly varies at all by the sex, age and education level of the participant, apart from a somewhat smaller proportion of those with a university degree. Comparing EI claimant to reachbacks (Exhibit 7.4), little difference is observed, implying equal perceived program effectiveness for these two client groups. Comparisons were not made with the comparison group because too few of this group made use of employment assistance services.

Employment Rates

The second way employment impacts were measured using participant-survey results was to observe the actual employment outcomes for participants following their intervention. Exhibit 7.5 presents the employment results for PBM participants and for the comparison group. Overall, the results of this bivariate analysis signal at least an initial advantage for PBM participants on some indicators compared to non-participants (though this positive result is tested more rigorously in the multivariate analysis).

The first panel of Exhibit 7.5 indicates that a majority (61 per cent) of participants was employed at some time since completing the intervention or since the reference date in the case of the comparison group. Again, there was little variation across age, sex, and education sub-groups (not shown). However, there were differences by program type. Of all the program groups, Entrepreneur and Partner participants were, by far, the most likely (96 and 88 per cent) to have had a job in the post-program period. These large proportions are largely attributable to the fact that 87 per cent of Entrepreneur clients reported that their business continued beyond the assistance period and that 57 per cent of Partners clients were working in jobs offered by their host employer (not shown). The results across other benefits and measures are similar (in the 60-65 per cent range), with the exception of SLG participants who were the least likely (49 per cent) to have had a job in the post-intervention period. EI claimant participants were more likely (68 per cent) to have been employed than reachbacks (55 per cent), which is not surprising given the former's more recent experience in the labour market. They were also more likely to be employed than comparison group members were (52 per cent), implying some advantage for PBM participants over participants in other employment assistance interventions.

Other results (not shown) indicate that employment outcomes vary by claimant status within program types. These results indicate that EI claimants are much more likely to be employed following participation in Job Action than reachback participants in Job Action, and more likely to be in jobs lasting at least 12 consecutive weeks. On the other hand, participation in SLG and EAS appears to benefit reachbacks much more than EI claimants.

Employment Stability

There were two measures of employment stability in participant survey results. First, participants were also asked if they had occupied a job that lasted (at least) 12 weeks since leaving the program, as an indicator of the durability or stability of employment. Returning to Exhibit 7.5, panel 2 indicates that the majority (59 per cent) of those who had worked found a job that lasted at least three months. Once again, Entrepreneur and Partners participants are distinguished from other participants by the fact that they were most likely (88 and 83 per cent) to have had fairly stable jobs and SLG participants by the fact that they were least likely (52 per cent) to occupy such jobs. The other interventions had similar levels of stability (54-63 per cent). EI claimants were more likely to have had stable jobs compared to reachbacks (71 versus 49 per cent), and there was some advantage for EI claimants in this respect over the comparison group (71 versus 60 per cent). Finally, older workers were more likely to have experienced 12 consecutive weeks of employment than workers in other age groups were (not shown).

View EXHIBIT 7.5

The second indicator of stability is the number of employers that participants had (panel 3 of Exhibit 7.5). The results indicate that a majority of participants across all programs who had at least one job since the intervention had only one employer during the post-program period. Once again, it was participants in Entrepreneur and Partners were most likely to have had just one employer (91 and 80 per cent). EAS participants were the least likely to have had just one employer (46 per cent). Other survey results (not shown) indicate little variation in these figures across age, sex and education participant groups. Finally, Exhibit 7.5 indicates that there was little difference between EI claimants on the one hand and reachback and comparison group participants on the other.

Employment Status Outcomes

In the participant survey, participants were asked about their employment status at two points in time: (1) in the first week following the intervention, and (2) currently, i.e., at the time of the survey. The results in panel one of Exhibit 7.6 indicate that, overall, about one-third (35 per cent) of PBM participants were employed or self-employed in the immediate term following the end of the program, but the proportion varied considerably across benefits and measures. The highest proportions were recorded for Partners (with 60 per cent in full-time jobs) and Entrepreneur (with 71 per cent in self-employment). These participants were most likely to be employed at this time as a result of their continuous employment with their employer (Partners) or sustained operation of their business (Entrepreneur). Employment shares were similar across the rest of the other PBMs, except for the somewhat higher incidence of part-time employment for EAS participants (13 per cent). As for unemployment share, the results indicate that almost half of participants (44 per cent) were unemployed and looking for work in the week after their program ended. This share varied considerably across PBMs, ranging from just 13 per cent for Entrepreneur to as high as 69 per cent for Rural Experience. There were few differences of note between EI claimants and reachbacks.

There were some noticeable differences in the first week, post-intervention employment status by age, and education (not shown). For example, younger participants (under 30) were the least likely to be out of the labour force (unemployed and not looking for work). Also, men were more likely to be employed full-time and women part-time, reflecting overall labour force patterns. And by education, the results indicate that those with no more than a high-school certificate were the most likely to be unemployed and looking for work, while those with a college education were most likely to be employed full-time. Once again, this reflects overall patterns by education level.

For current employment status (panel 2 of Exhibit 7.6), patterns by PBM program types are similar to the first-week status following intervention. Once again, the most positive results are for Entrepreneur and Partners participants. Only nine per cent of Entrepreneur participants were unemployed and looking for work at the time of the survey compared to rates as high as 46 per cent for Job Action and 55 per cent for Rural Experience. Among the other program groups, the percentage unemployed is similar (in the 24-29 per cent range). Partners participants have the highest incidence of full-time employment.

View EXHIBIT 7.6

Current employment status results are positive for both PBM participants and the comparison group. At the time of the interview, 44 per cent of EI claimant participants were employed (self-employed, full-time employed or part-time employed) compared to 50 per cent for the comparison group, but the incidence of full-time employment was higher among EI-claimant participants (33 versus 25 per cent). While the percentage who were unemployed and looking for work was similar between the two groups (about one-third), the proportion going to school was somewhat greater for EI claimants than for the comparison group (10 versus four per cent).

Comparing the post-intervention employment status results in the first week (Panel 1 of Exhibit 7.6) to that at the time of the survey (Panel 2) indicates some improvement over the post-intervention period. All employment shares (full-time, part-time and self-) rose somewhat between the two points in time. For example, the proportion in a full-time job increased from 25 to 29 per cent. Similarly, the proportion that is unemployed and looking for a job fell between the first two points in time (from 44 to 32 per cent). These results are similar for EI claimants and reachbacks. By program type, however, some full-time employment declines were recorded, e.g., the decline in the incidence of full-time employment for Partners participants (60 to 54 per cent).

Pre-/Post-Intervention Comparisons of Employment Status

Another way employment outcomes were measured using the participant survey results was to observe changes in employment status from before the intervention to after. In Exhibit 7.7 the selected results of cross-tabulating pre-intervention employment status with post-intervention first-week and current employment status are presented. The results show that, although the largest post-intervention proportions for each pre-intervention employment status group are also the largest proportions for the pre-intervention employment status group, some interesting changes did occur as follows:

• Among the unemployed in the week before entering the program, one-quarter (23 per cent in the first week and 26 per cent currently) ended up working full-time and six and nine per cent working part-time in the first week and currently, respectively. Over one-half (54 per cent) remained unemployed, and this proportion fell to 41 per cent over the post-intervention period.
• Among those who had been working part-time before entering the program, about one-fifth had changed their status to full-time after the intervention (20 per cent in the first job and 19 per cent currently). Note, as well, the increase in school attendance for this group over the post-intervention period (two to 21 per cent), an indication that these participants had considered education as a means of improving their employment status.
• A large share of the pre-intervention not-in-the labour force group ended up in full-time jobs. Note again that this proportion increased over the period following intervention (19 to 25 per cent), while the proportion unemployed fell (27 to 13 per cent). Interestingly, the decline in the percentage unemployed is commensurate with the increase in the combined percentages of participants in full-time employment and in school.
• Among those who were students before intervention, a large proportion ended up employed full-time (19 per cent) in the first week following the intervention. This proportion rose considerably to 45 per cent currently indicating a delayed reaction to intervention. Similarly, the proportion unemployed fell from 36 to 13 per cent between the first week and currently.

One other finding (not shown) is that about 70 per cent of those who were not or only somewhat interested in entering the labour before the intervention reported that they were interested after the intervention.

Characteristics of Current or Most Recent Job

In this section, the characteristics of participants' current job (at the time of the survey) or, if without a job at the time of the survey, the latest job (following the intervention) are briefly examined. Excluded were those who never had a job since the intervention or reference date. Tabular results are presented in Appendix D.

First, just under one-fifth (19 per cent) of PBM participants had returned after the intervention to the employer they had prior to it (among those who had a job before), with SLG participants being the most likely to do so (31 per cent). EI claimants were considerably more likely (27 per cent) to do so than reachback participants (11 per cent). Interestingly, however, EI claimants were less likely to return to their previous employer than members of the comparison group (32 per cent) were.

Second, of respondents in their current or most recent job, the majority was working full-time hours. Almost nine in 10 participants worked more than 30 hours per week.

Third, overall, a little over one-half (54 per cent) of PBM participants were working in year-round jobs at the time of the survey. Entrepreneur and Partners participants were most likely to be in year-round jobs (85 and 75 per cent). Conversely, Job Action and particularly Rural Experience participants were more likely to be employed on a casual/contract basis (32 and 50 per cent) or in seasonal jobs (24 and 18 per cent) than those in other PBMs, except for EAS participants who also had a high incidence of seasonal employment (20 per cent).

Fourth, there was considerable variation in weekly earnings across program groups. Mean gross weekly earnings for PBM participants in participants' current/most recent job ranged from as low as $342 for Job Action participants, $366 for Rural Experience participants and $372 for EAS participants to as high as $514 for SLG participants, almost one-half (47 per cent) of whom earned $500 or more a week (Partners participants earned an average of $415 per week and Entrepreneur participants had average weekly earnings of $472). Median wages indicate a similar ranking of programs though with smaller differences among programs. This can largely be explained by the educational profile of clients in the respective program, as discussed in the previous chapter. SLG clients tend to be more highly educated while Job Action clients tend to have no more than a high-school education. EI claimants ($479) earned somewhat more than the reachback group ($401) and the comparison group ($458); however, comparing the median wages across those three groups indicates no difference. Finally, reflecting patterns observed in the labour force overall, female clients earned less than males did, older clients earned more than those in other age groups did, and weekly pay rose with education level (not shown).

View EXHIBIT 7.7

Retention

A further measure of employment success is the extent to which wage-subsidy participants were retained by their host employers at the end of the subsidy.²⁶ In the participant survey, participants who remained with their host employers until the completion of the wage subsidy were asked this question.²⁷ The responses to this question, presented in Exhibit 7.8, indicate that just under one-half of participants (46 per cent) were retained by their host employers. Given the nature of the respective interventions, it is not unexpected that the retention rate was highest for Partners participants (66 per cent).²⁸ In contrast, Job Action and Rural Experience are, by definition, project-based interventions, with little expectation of participants continuing beyond the wage-subsidy period. Indeed, as the exhibit indicates, the proportions retained are only about one-quarter (28 and 23 per cent) for these two interventions. In addition, it is noted that the retention rate is somewhat higher for EI claimants compared to reachback clients (52 versus 45 per cent). Finally, retention declines with age and is similar for males and females (not shown).

Similarly, as the second row of Exhibit 7.8 indicates, Partners participants were most likely to be hired into the same job as they had during the intervention. Of those who were hired by their host employers, over 90 per cent of Partner and Job Action participants were hired into the same job compared to 86 per cent overall. The rate for Rural Experience participants was in the three-quarters range. EI claimants were somewhat more likely to be hired back into the same jobs than reachbacks (92 versus 83 per cent).

Finally, in the third panel of Exhibit 7.8 we observe the types of jobs participants in the different wage-subsidy interventions were hired into. The results indicate that Partners participants were much more likely (81 per cent) to be hired into year-round full-time jobs than participants in the other wage subsidies (42 and 28 per cent). There was little difference between EI claimants and reachbacks in this respect.

What are the reasons participants are not being hired? For this, we turned to the employer survey results. Among the reasons offered by employers for having not hired all workers (or intending not to hire all workers), Exhibit 7.9 indicates that the dominant reason is a lack of resources on the part of employers participating in Job Action and Rural Experience (37 and 30 per cent, respectively). Results not shown indicate that lack of resources was cited most frequently by larger, non-profit, municipal, older, year-round, and low-growth organizations. By industry, this reason was most frequently mentioned by businesses in arts, entertainment and recreation; education, health and social services; and transportation and distribution services. For Partners employers, the main reason is the fact that the jobs were only temporary or seasonal (16 per cent), which also was an important reason for Rural Experience employers (27 per cent). The latter also frequently mentioned lack of work (19 per cent) as a reason for not hiring participants after the subsidy. Lack of work and temporary work reasons were most likely to be mentioned by seasonal businesses, with no clear link with other organizational characteristics (not shown). Other reasons were mentioned typically by only a fraction of employers, with "workers not amenable to training and their attitude" being the most frequently mentioned among these reasons, particularly for Partners employers.

Non-completion

Large numbers of participants not completing their term under the wage subsidy would be an indication of lack of success. As Exhibit 7.10 indicates, about one-third of employers participating in wage-subsidy programs who responded to the employer survey lost some participants before the full duration of the subsidy. Among employers who experienced losses, the most frequent reason for non-completion cited by employers was that the participant voluntarily quit (36 per cent), which was the case for all three programs, but particularly for Rural Experience (42 per cent). Other prominent reasons cited were as follows: found another job and personal problems/illness, particularly for Job Action (20 per cent); not amenable to training/poor attitude, particularly for Partners (21 per cent); incompetence, for Partners and Rural Experience employers (17 and 16 per cent); and wrong skills for the job, particularly for Partners (17 per cent). Among these reasons, it should be said that finding another job would be considered a "good'' reason and not attributable necessarily to the intervention or the experience itself.

Incrementality

It was clearly stated in program documentation that employers are to hire workers under wage subsidy only into incremental jobs, i.e., into jobs that they would not have filled otherwise. In the employer survey, employers were asked to indicate what they would have done if the wage subsidy had not been available. Summing up the responses indicating incrementality, presented in Exhibit 7.11, indicates that about three-quarters (74 per cent) of wage-subsidy transactions were incremental. Incrementality was greatest for Job Action (85 per cent) and lowest for Partners (66 per cent). At the other extreme, about one-fifth (19 per cent) reported that they would have hired someone into the positions regardless of the wage subsidy, implying complete non-incrementality. This was the case particularly for Partners employers.

There were some interesting patterns in incrementality across organizational characteristics (not shown). The percentage of employers who said they would not have hired anybody without the wage subsidy rose gradually by firm size, was particularly high for municipal organizations, rose with the age of the business, and was higher for seasonal and low-growth businesses. The fact that incrementality appeared to be greater for larger firms may be explained by the fact that such firms tend to hire more workers under the programs thereby increasing the chances that at least some of the jobs were incremental. By industry, incrementality was particularly high for public administration and education, health, and social services and particularly low for traditional services, business services and manufacturing.

Another incrementality measure from the employer survey was that provided by a question on whether or not the wage subsidy enabled employers to hire faster than they would have without the subsidy. The results presented in the last row of Exhibit 7.11, indicate that about two-thirds of participating employers reported that the wage subsidy speeded up their hiring. This proportion did not vary greatly across wage-subsidy types, but, as results not shown indicate, tended to be higher in manufacturing (81 per cent) and businesses less than two years old (81 per cent).

Yet another measure of incrementality was that provided by an employer survey question on hiring behaviour after the end of the wage subsidy. Employers were asked to indicate whether or not, after the wage subsidy, they had hired, or intended to hire, other (non-program) workers into the jobs that were occupied by the participant during the wage-subsidy period. This could be viewed as a measure of incrementality (as well as satisfaction with wage-subsidy experience). However, as mentioned above with regard to retention rates, this would be the case only for Partners because Rural Experience and Job Action are project-based benefits and not intended to lead to lasting jobs. The responses to this question (not shown) indicate that just over one-fifth (20 per cent) of Partners employers have done so, or intend to do so, with little variation across wage-subsidy types. Participating government and municipal organizations, older businesses (over 30 years), and health, education and social service organizations were the least likely to have responded in the affirmative to this question.

Joblessness and Job Search Outcomes

In this section, we review the evidence on four post-intervention outcomes: number of weeks jobless, number of weeks searching while jobless, job-search activity, and interest in entering the labour force. "Jobless" individuals are here defined as those officially unemployed (i.e., out of work and looking for work) plus those not in the labour force.

Duration of Jobless Spells

Panel 1 of Exhibit 7.12 presents survey results with respect to the duration of jobless spells following intervention, scaled by the time since the intervention, as reported by respondents. The mean percentage of time not working during the post-program period for PBM participants was 29.1 per cent and the median was almost one-fifth (19.4 per cent). There was much variation across the program groups. Job Action and Rural

Experience participants experienced the most joblessness at over 35 per cent of the time since the intervention (mean and median), with less than 15 per cent of each group experiencing no joblessness. It is also noted that almost three in ten Rural Experience participants (29 per cent) were jobless in one-half of the weeks since their intervention. At the other extreme were Entrepreneur participants who on average were jobless for only eight per cent of the time (median = zero) and who were the most likely to experience no joblessness (78 per cent) and the least likely (six per cent) to have experienced joblessness for over 50 per cent of the post-funding period. This is largely attributable to the fact that most of these persons continued on in their self-employment business after funding had ended. As for differences by sociodemographic characteristics, results (not shown) indicate that the mean percentage of time not working rises with age, declines with education level, and is higher for female clients than for male clients.

Members of the reachback (mean=31.3 per cent) and comparison (mean=45.1 per cent) groups experienced considerably more joblessness than EI claimants did (mean=26.7 per cent). This is also apparent from the differences in the proportions experiencing joblessness for over 50 per cent of the time (27 and 40 per cent versus 18 per cent, respectively).

Duration of Job Search

In this section, we present results for job search while jobless in the post-program period, which is considered to be a positive outcome of the intervention. It should be noted that responses to the respective question were re-coded to include as zero those who were not jobless in the post-program period. Also, as above, responses were scaled by weeks since the intervention.

The percentage of time looking for work during the post-program period (panel 2 of Exhibit 7.12) parallels the above results. Entrepreneur participants were the least likely to look for work while unemployed in terms of mean and median percentage of the time (6.4 and zero per cent) and also had the highest proportion (91 per cent) who were searching for work while unemployed for one-quarter or less of the time since the intervention. Partners and EAS participants also demonstrated good results in this respect. Job Action and Rural Experience participants looked for work for the longest periods of time (means of over 27 per cent of the time). Differences by age, sex and education are in the same direction but muted compared to those observed above for weeks of joblessness (not shown). Finally, reachbacks tended to look somewhat longer for work while jobless than EI claimants. Comparison group members tended to look even longer (16.1 versus 9.3 per cent, medians).

View EXHIBIT 7.12

Job-Search Activity

Those who reported actively looking for work during the post-program period (both those who obtained a job and those who did not) were asked to specify the types of job search methods they used. Exhibit 7.13 presents the responses to this question. The top four methods of job search were the same for all respondent groups, including all program types as well as among EI claimants, reachbacks and the comparison group. Sending résumés or applications, making personal visits to the employer, using a resource centre or job bank, and word of mouth (asking friends and family) were the most important methods of jobs search for all groups. Less than five per cent of respondents used the Internet or participated in a job finding club for their job search.

Although there was little variation in job-search activity across the benefits and measures, some differences did arise, including the following:

• EAS participants were more likely to send resumes and applications compared to other participants and were also more likely than others to rely on newspapers.
• Rural Experience participants were by far the most likely to use word of mouth, family and friends to search for a job.
• Job Action participants, too, were likely to rely on word of mouth.

Some interesting differences in job-search activity also arose according to the sex, age and education level of participants (not shown). The proportions going to a resource centre, making telephone inquiries with employers and using word of mouth all rose with age, but the proportion sending out resumes declined with age. Men and women tended to pursue the same activities, apart from greater relative numbers of women sending out resumes than men did. Finally, those with no more than a high school certificate appear to be much more reluctant to send out resumes than those with a college certificate or a university degree (40 per cent versus 61 and 60 per cent).

View EXHIBIT 7.13

Interest in Entering Labour Force

Finally, those who were jobless (i.e., unemployed or out of the labour force) at the time of the interview were asked to rate their level of interest in entering the labour force in the next 12 months. Interest was gauged on a seven-point scale, ranging from one equal to "not at all interested" up to seven equal to "extremely interested". The re-grouped responses to this question, presented in Exhibit 7.14, indicate that a vast majority (over 90 per cent) of PBM participants were reported to be quite interested (6 and 7 on the scale) in entering the labour force. Other results (not shown) indicate that interest level did not vary much by age, sex and education (not shown); across program types, there were only minor differences as well. As for claim status, differences again were not great, with reachbacks indicating somewhat lower levels of interest in entering the labour force. Parallel ratings by the comparison group show more modest interest in entering the labour force, with only three in four indicating being interested in entering the labour force.

Utilization of Income Support

Another major indicator of program effectiveness is the extent to which interventions have reduced participants' reliance on income support such as employment insurance (EI) and social assistance (SA). Note that the results on EI use are based on the administrative data, which permit the capture of a return to EI after the period of EI associated with the intervention is completed. This is a true measure of EI use that is not possible using the survey results.²⁹

Exhibit 7.15 indicates that about one-quarter (24 per cent) of participants started a new spell of EI following program intervention. Reflecting the pattern of employment experiences observed above, there was wide range in EI use, with as little as three per cent among Entrepreneur participants having received EI to about one-third (32 per cent) of Partners clients and over one-half of Job Action and Rural Experience clients. Reflecting labour market experience patterns, women and younger persons were less likely to have collected EI than those in other sex and age groups (not shown). On average, as panel 2 indicates, participants received EI in eight per cent of the weeks following the intervention, translating to about one month on an annualized basis. Rural Experience and Job Action participants had the highest mean percentage of weeks receiving EI (22.9 and 17.1 per cent).

EI claimants were less likely than reachbacks to have received EI in the post-intervention period (21 versus 32 per cent) and for less of the time following the intervention (5.7 versus 11.1 per cent, means). However, non-participants were somewhat more likely (23 per cent) to have been on EI in the post-intervention period but for a smaller percentage of weeks (3.9 per cent mean).

As for reliance on SA, only a small minority (nine per cent overall) of participants reported receiving social assistance during the post-program period (panel 3 of Exhibit 7.15). Post-program SA-use was highest among Job Action (31 per cent) and EAS (17 per cent) participants and lowest among Partners, Entrepreneur and SLG participants. The pattern was similar in terms of the percentage of weeks receiving SA, with Job Action participants standing out in terms of mean percentage of weeks received SA (12.4 per cent) and the percentage (25 per cent, not shown) receiving SA for more than five per cent of the weeks. As for differences between reachbacks and EI claimants, it is observed that the former, as expected, were much more likely to have used it (15 versus five per cent) and during a greater percentage of the weeks since the intervention (six versus 1.6 per cent, means), though there were no differences in medians between the two groups. Comparison group members were less likely than EI clients to rely on SA (one per cent), and for a smaller percentage of the weeks since the intervention (0.3 versus 1.6 per cent). Not shown is the fact that female clients were twice as likely as male clients to have received SA after their intervention and in a greater percentage of weeks.

View EXHIBIT 7.15

Attitudes, Training and Skills

In this section, the results from the focus groups, case studies, union survey, and employer survey conducted for this evaluation are presented. First, in the focus groups with clients, it was observed that their participation had increased their skills and experience, which, down the road, would increase their chances for employment. In the focus groups with employers, it was generally believed, too, that the program had enhanced the clients' skills and experience. In addition, clients had benefited psychologically from their participation in terms of feeling that "they had accomplished something" and in terms of acquiring "a sense of dignity", some for the first time. Some clients returned to school following their participation, which would in the long run benefit them.

In the case studies, many of the outcomes identified involved improved attitudes and skills for clients. In one "down-side" adjustment situation, committee deliberations and recommendations resulted in not only direct employment for affected workers but in several workers' returning to get their high-school certificates. Increased labour-market intelligence and job-search and resume-writing skills were other positive effects for the workers. In the other "down-side" adjustment situation, the ASI committee's work led to restored optimism and positively altered attitudes to work and learning among affected workers. In the SLG case study conducted for this evaluation benefits from the intervention came in the form of not only jobs for clients but also improved attitudes to work and learning, increased labour-market knowledge, changed attitudes regarding the benefits of contributing to one's own human-capital development, and reduced drop-out rates in schools. A factor contributing to these positive outcomes was the flexibility of the program. In one of the ASI community development case studies, we found that the participating organizations' working closely with the schools enabled the development of more labour-market skills for clients.

In the three EAS case studies, the benefits identified appeared to be mainly psychological. Common outcomes identified were improved attitudes toward work and learning, a strengthened ability to set career goals, and increased confidence to apply for jobs. Impacts of more practical nature included increased job-search skills and job-specific skills. Identified success factors in these case studies were once again local flexibility but also the emphasis on placement.

In the union survey, the two union representatives who responded were asked to assess the impacts of the intervention on participants. Unfortunately, the one respondent who felt qualified to answer these questions was the one representative who did not know which PBM intervention the employer had participated in. Thus, the client outcomes we discuss here cannot be associated with any particular intervention. At any rate, the one representative who responded said that the intervention had positively impacted on clients' attitudes to work, attitudes to learning and training, confidence, attitudes to authority, and job-specific skills. Little or no impact was detected on basic reading and writing skills, computer skills, personal management skills, teamwork skills, job-search skills, and overall level of preparedness and job readiness. The respondent was adamant in his or her belief that the participant was job ready at the end of the intervention. The union representatives identified no negative impacts of the intervention on participants.

In the employer survey, there were two sets of questions on human capital development: training and skills. With respect to training, it would be one of the goals of the wage subsidy benefits to increase the skills of participants so that they may enter employment. In the survey, employers were asked to indicate what sorts of training and related assistance they provided their wage-subsidy participants. The results presented in Exhibit 7.16, which shows the proportions of employers providing the different types of training assistance, indicate that job-specific (83 per cent) and orientation training (76 per cent) were the most common sorts of training provided, which was true for all wage subsidy benefits.³⁰ In another tier were task rotation, personal skills training, mentoring, and job shadowing, all of which were provided by about one-half of employers. Unique among the activities listed in the exhibit is fringe benefits (36 per cent), which is not really associated with human capital development. The least frequently provided assistance was computer training and career counselling (particularly in Rural Experience), and job-search advice (particularly in Partners).

Not shown is the fact that the provision of training is strongly associated with the size of the organization. The fact that the incidence of training rises steeply with organization size clearly reflects long-established patterns from the general labour-market literature. Other organizational characteristics associated with the provision of training are being a municipal organization, being two years or younger, being a year-round business, and already training large proportions of the organization's workforce.

To what extent were skills of participants increased as a result of the training provided and the experience itself? On the employer survey, employers were asked to judge the impact of the wage-subsidy experience on participants. Results from this question presented in Exhibit 7.17 indicate the proportion of employers saying participants' experience under the wage subsidy had a positive impact on them (5, 6 or 7 on a 7-point scale where 1=extremely negative impact and 7=extremely positive impact). The results indicate a majority of employers believed the wage subsidy had a beneficial impact on clients. There was little variation in reported impacts across skills and wage-subsidy benefits, with 75-89 per cent of employers believing that participants benefited from their experience. The only differences of any note are those between Job Action and Partners in attitudes to learning and training (89 versus 74 per cent). Not shown is the fact that few employers believed the experience had a negative impact on clients. The results from another measure, which is also not shown, indicate that 60 per cent of employers believed that participants were job ready following the wage subsidy, with the biggest difference occurring between Partners (67 per cent) and Job Action (54 per cent). Finally, other results (not shown) indicate that skill and attitude impacts did not vary greatly with organization size, sector, age, and industry.

7.3 Impacts on Employers

In this section, we present (unweighted) impact results mainly from the employer survey, as well as the focus groups, case studies, and union survey. In the focus groups with employers, the evidence gathered indicated that employers' participation in the wage-subsidy interventions had benefited their organizations in both quantitative terms (increased production) and qualitative terms (enhanced service delivery).

In contrast to the ASI "down-side" adjustment committees reviewed above which benefited mainly the affected workers, the two case studies of "up-side" employment-development adjustment situations resulted in positive impacts for the participating organization. One case study found that the committee's work contributed to a change in operation style, to increased management and marketing knowledge on the part of the owners, and to the introduction of the use of the Internet in the organization. Increased sales and employment resulted from these changes, but also increased costs in covering a portion of the professional fees of the committee chair. Similarly, in the other "up-side" ASI case study, it was found that the committee led to a large number of positive outcomes for the organization, including a business plan, marketing strategy, reorganized finances, increased confidence in the company's direction, and increased awareness of the existence of business support services. Importantly, the recommended course of action led to increased access to, and sales in, new markets. An identified success factor was "buy-in" from senior management.

In the union survey, the two union respondents were asked to evaluate the impact of the interventions on the organization and its existing workforce. For the most part, the responses of the two representatives were similar. Both thought the intervention had a positive impact on employee morale and workload. One of the two respondents (although not necessarily the same one) also thought the respective program positively affected productivity, employment levels, interest in providing job opportunities to those in need, and the ability of the organization to evaluate new potential employees. Little or no impact was detected on existing employees' wages, advancement opportunities, the organizations' output and revenues, product or service quality, openings for new workers, hiring practices, and relations between management and employees. The union representatives identified no negative impacts of the intervention on the organization and its staff.

In the employer survey, there were two sets of impact questions: one on perceived impacts and one on the costs of participating. First, employers were asked to evaluate the impact of the wage subsidy on their organization along a number of dimensions using a seven-point scale, ranging from 1 (extremely negative impact) to 7 (extremely positive impact). The results presented in Exhibit 7.18 show the percentage of employers reporting 5, 6, or 7, indicating a positive impact. (Few employers believed the wage subsidy had a negative impact (1, 2, or 3) on the organization and the percentages indicating a neutral impact (4) were in the mid-teens for all wage subsidy types). The results indicate that a majority of employers thought that the wage subsidy benefited their organization. The greatest support was reported for benefiting the organization overall (82-86 per cent) and, among specific aspects of the organization, it appears that the workload and productivity of existing employees benefited the most (71-76 per cent). There was little variation across different wage-subsidy measures, with the greatest gap being between Partners and Rural Experience in terms of enhancing the organization's ability to evaluate potential new employees (85 versus 62 per cent).

For this set of measures, the age and the sector of the organization appeared to matter (not shown). For all organizational impact measures, the percentage indicating a positive impact clearly fell with the age of business. And for all measures but workload/ productivity, government organizations were least likely to benefit from their participation in the wage subsidy program. A few differences by industry also emerged: no more than one-half of participating organizations in construction, business services, and public administration indicated that their ability to evaluate new potential employees benefited from their participation in the program.

The second organization-impact measure was provided by a survey question on employer costs of participation. In the employer survey, employers were asked to identify areas where they had incurred significant costs resulting from their participation in the program. The results from this question presented in Exhibit 7.19 indicate that significant costs were incurred most prevalently in two areas: supervision/training and wages (39 and 31 per cent). This was true of all wage-subsidy programs, but particularly with respect to Partners, where close to one half (49 and 45 per cent) of employers incurred significant costs in these two areas. The third most frequently cited significant cost overall (15 per cent) was with respect to benefits; again this was higher for employers participating in Partners (23 per cent). Note that 41 per cent of employers reported that they did not incur any significant costs as a result of participating in the wage subsidy.

As for differences by organizational characteristics, private sector businesses were, by far, the most likely to say that wages represented a significant cost to their participation in the wage subsidy. Businesses with five or more employees most frequently mentioned the time and money spent supervising participants as a significant cost of participation. Non-profit employers were most likely to say that they had experienced no significant costs while participating in the program.

7.4 Impacts on Communities

Qualitative evidence gathered in the key informant interviews, focus groups, and case studies was used as the basis for the discussion in this section. Generally speaking, key informants were not able to detect benefits at the community level. One key informant observed that the local unemployment rate had not fallen during the time the new PBMs have been in place, while another observed that the lack of consultation at the local community level may even have led to negative effects for the community (though what these were was not specified). Still another believed that there was a plethora of employment programs designed to move social assistance recipients (SARs) off social assistance and onto Employment Insurance and that LMDA had not changed this situation. On the other hand, another observer thought that the drop in the number of SARs was attributable partially to the LMDA, but also to the improvement in the economy.

In the focus groups with employers, it was thought that the programs have resulted in enhanced infrastructure for the community. This will serve to attract tourism to the community and thus benefit it in the long term, as well as to expand the activities that the participating community organization can carry out. It was also noted that, during projects where local infrastructure was being built, there had been spin-offs for the local community, e.g., local suppliers had benefited from the material purchases.

The two ASI community development case studies found that communities definitely benefited from the ASI committee's work. In one community, an action plan was recommended and adopted by the community. This led to the decision to open an RCMP station in the community, to hire an economic development manager, to purchase land for an industrial park, and to begin negotiations with major employers to set up shop in the community. In addition, morale in the community was increased and greater ties between the community organization(s) and the town council were established.

In the other community-development ASI case study, both the community organization and the community benefited from the adjustment experience. The committee's recommendations led in the immediate term to the creation of a strategic plan, to hiring business operations managers, and to hiring additional workers to relieve staff shortfalls within the organization. These actions enabled the organization to identify additional revenue sources, enabled it to become more financially self-sufficient, and in general encouraged it to adopt a more business-oriented philosophy. This in turn enabled the organization in turn to be more socially proactive and to place clients from the community in greater numbers and diversity. Working more closely with the schools enabled the development of more labour-market skills for clients. An important consequence of these activities was greater morale in the community.

Finally, in an ASI downside case study, a finding was that the committee helped the community to adjust to the loss of a major local employer.

An essential question in the analysis of the impacts of PBMs is the incremental impact of these services. Simple comparisons between program participants and non-participants on key outcome indicators (e.g., employment status) may yield a biased estimate of program impact because of pre-existing differences between the comparison group and the PBM participants. In order to ensure that differences in measured outcomes were not the effect of pre-existing differences between the PBM participant group and the comparison group, and also among participants in different PBMs in terms of their labour market experience or background characteristics, multivariate analyses were conducted. Note that these analyses were based on unweighted data because of the inclusion of control variables, which figured in the computation of the weights. Note as well (and again) that this formative evaluation may bias results in favour of interventions such as Entrepreneur and Partners where the outcomes are immediate and against interventions such as EAS, Job Action, SLG and Rural Experience where the assistance is such that there are no immediate employment outcomes or the employment effects would be expected to be of a more long term nature.

8.1 Description of Approach

Ten dependent variables representing key employment, earnings,³¹ and income support use outcomes were tested in the models, corresponding to the key objectives of the LMDA which are sustained employment and reduction in dependency on income support. These variables are as follows:

• employed/self-employed (or not) at time of survey;
• full-time employed at time of the survey;
• worked 12 consecutive weeks since end of intervention/reference date;
• weeks working as percentage of weeks since intervention/reference date;³²
• weeks looking for work as a percentage of weeks since end of intervention/ reference date;
• weekly earnings of current or most recent job (at the time of the survey);
• absolute change in weekly earnings (compared to one year prior to intervention/reference date);
• per cent change in weekly earnings (compared to one year prior to intervention/reference date);
• a new spell of weeks on EI as a percentage of weeks since end of intervention reference date; and
• ever received Social Assistance since intervention/reference date.

The means or frequencies of the dependent variables are presented in Appendix E-1a.

Along with the intervention "dummy" variables, a common set of explanatory (control) variables was introduced into the models for each dependent variable. The purpose was to assess (or control for) the influence of other factors on the intervention's impact on the outcomes. These other factors included the time since the intervention and antecedent sociodemographic and employment-history variables. The means or frequencies of these variables are contained in Appendix E-1b. Noting that "intervention" here refers to the end of the intervention for participants in LMDA PBMs and to the reference date for comparison group members, the variables entered into the models follow:

• intervention status: one variable to indicate individuals' participation in one or more of six PBMs, or non-participation in any of the interventions (comparison group);
• length of time since the intervention (since this varied considerably);
• sociodemographic variables:³³ age, sex, education, mother tongue, minority status, marital status, and existence of dependants;
• prior labour force experience: employment status (employed, unemployed) in month before intervention (versus not in the labour force), whether employed or not one year before intervention/reference date (entered in stepwise fashion because of concerns with collinearity with the previous variable), interest in entering training/self-employment/ labour force prior to intervention, number of separations 1992-1997, weeks of EI eligibility overlapping with intervention, weeks EI benefits received 1992— 1997, and total gross earnings in the year prior to intervention; and
• service-delivery variables: whether individuals had used self-serve products, received counselling, participated in job-search activities or developed an action plan, or services other than from HRD-NB or a HRCC/HRSC.

Logistic (logit) regression was used for categorical dependent variables and Ordinary Least Squares (OLS) regression was used for continuous dependent variables. The modelling was conducted in a series of four stages. First, a flag representing participation in a PBM (compared to non-participation in that PBM) was entered in the model alone, i.e., without the controls. The coefficient for this variable measures the effect on the outcomes of the intervention not controlling for the influence of other factors. Second, controls for time since intervention and sociodemographic and work-history background variables were introduced into the model. Results from the bivariate analysis presented in the previous chapter had indicated that the profile of participants in the different interventions differed appreciably. The coefficients on the intervention variables would now tell us whether or not the previously found impact of the intervention had more to do with the nature of the participants in the program than with the program itself. In the third stage, variables related to related service delivery (e.g., action plans) were introduced.

Finally, in the fourth stage, controls computed to reduce self-selection bias (the so-called Heckman correction or the Inverse Mill's Ratio) were introduced into the model.

It should be noted that, originally, the econometric analysis was to be conducted for just active EI claimants because of the inability to draw a comparison group for reachbacks, as discussed in the previous chapter. However, because of concerns with the relatively small number of active EI claimants due to the larger than expected number of reachback participants in the survey dataset (and in fact in the population of participants), a group of what we called "near" reachbacks was included as EI claimants. These individuals had received EI six months or less prior to the intervention. Grouping them with EI claimants was justified because of their recent experience with EI and the labour market and their similarity with regard to other characteristics. In other words, the comparison group could serve as a comparison group for this group of participants as well.

8.2 Results

In the results that follow, we present in the tables four columns of coefficients (and their significance level) corresponding to each variable-entry stage as described above. This is done to observe how coefficients on the program variables change as control variables are cumulatively entered into the model. We also identify what role, if any, was played by the controls. The complete set of results for the final stage of the modelling exercise, i.e., including all control variables and the Heckman Correction factor, for each dependent variable is presented in Appendix E. In observing whether a variable exerts a positive or negative impact on the dependent outcome variables, we mention only variables exerting a statistically significant impact at the five per cent or lower level.

As there appeared to be differences in the results by gender and sex, segmented analysis by these variables was conducted. Interesting differences in the results between segments are commented on in the text below, based on the full results which are presented in Appendix E. Segmented analysis was also conducted by claimant status (active EI claimant participant versus near-reachback participants). As mentioned previously, the latter were grouped with active EI claimants for purposes of this analysis. As well, the results of the latter analysis are commented on in the body of this chapter and presented in Appendix E.

The results are presented below for three sets of outcome measures: employment/job search, earnings, and income support use. Each is discussed in turn.

Employment and Job Search Outcomes

Logistic regressions were run for three different binary employment measures as the dependent variable: currently employed (at the time of the survey), currently employed full-time, and employed for at least 12 consecutive weeks since the intervention. Ordinary Least Squares regression was run with the continuous variable percentage of weeks working since the intervention. We also present, in this section, the results of modelling percentage of weeks looking for work since the intervention.

In Exhibits 8.1a, b, and c, we present the coefficients for just the program (benefit and measure) dummy variables, for the first three employment measures. As previously mentioned, for each employment measure as well as for all the other outcome measures, we present four columns of coefficients corresponding to four different regression runs:

• the run with only the program variables in the model,
• the run with program variables plus time since intervention and sociodemographic and prior history background variables in the model,
• the run with the full set of variables including the service-delivery variables in the model, and
• the full model including the Heckman Correction (Inverse Mill's Ratio).

The results for currently employed (Exhibit 8.1a) indicate that, with only program variables in the model (column 1), four interventions make a significant impact on the likelihood of being currently employed. Compared to non-participants in the respective program, Entrepreneur, EAS and Partners significantly increase the chances of being employed (after the intervention at the time of the survey), while Rural Experience reduces the chances. After inclusion of the variables capturing weeks since the intervention and the sociodemographic and work history traits of the participant (but not the service delivery variables), column two of Exhibit 8.1a indicates that, the three interventions exerting a positive impact on the outcome variable still exert a positive impact, while Rural Experience no longer has a negative impact, implying that the participants appear to compensate somewhat for the negative effect of the intervention. With the addition of the service-delivery variables, we observe in the third column that EAS no longer has a positive impact, implying that these variables nullify EAS's positive influence. Finally, in the fourth column, we observe that correcting for self-selection bias has returned the EAS to its original positive impact on the chances of being employed, along with Partners and Entrepreneur and EAS.

Full-model results presented in column one of Appendix E-2a reveal that a number of the control variables have an influence on the likelihood of employment. Exerting a positive impact are: being unemployed in the month prior to the intervention (compared to not in the labour force), having a high-school certificate and post-secondary education (compared to less than a high-school certificate), an interest in entering the labour force, being eligible for EI for 37 weeks or more (compared to not being eligible) prior to or overlapping with the intervention. The variables exerting a negative impact are: being male, being in a minority group, having no dependants, and using self-serve employment services.

Because gender appears to affect the chances of employment, we ran segmented models for males and females separately. The results presented in Appendix E-2a indicate that Entrepreneur had a positive effect for both men and women, but Partners had a positive impact for only males. EAS, which was found to have a positive effect for the full population, had no effect for either the male and female segments. As for the controls, it appears that women's employment chances were affected by outside influences more than men's were. Women but not men were positively affected by having an interest in entering the labour and negatively affected by being employed one year before the intervention, being in a minority group, and having no dependants. Still, men but not women were positively affected by being unemployed one month prior to program entry and having some post-secondary education, and negatively affected by use of self-serve services.

By age, the results in Appendix E-2a indicate a few differences between younger (45 years and under) and older (over 45 years of age) participants. There was no difference in terms of the impact of the programs, but the younger group was more affected by outside influences. Only the younger group was negatively affected by being in a minority group and using self-services and positively affected by having at least a post-secondary education and eligibility for 37 or more weeks of EI.

For claimant status, Appendix E-2a indicates that Entrepreneur positively affects both active and reachback EI claimants, while Partners has positive employment impacts for near-reachbacks only. As for differences in the impacts of the controls, it is EI claimants who appear to be more affected. Positively affecting the employment of claimants only are: being unemployed in the month before the intervention (compared to not in the labour force), have some post-secondary education (compared to having less than a high school certificate), and having an interest in entering the labour force. Being male and no dependants negatively affects this group only.

The second set of results in Exhibit 8.1b are for being full-time employed at the time of the survey (post intervention). Full-time employment would be considered by many to be a better outcome than being employed part-time or self-employed, which would be included as part of the previous dependent variable. The results in the first column of the exhibit indicate that, with no controls entered in the model, three programs — Partners, Entrepreneur, and EAS — significantly increase the chances of being full-time employed. The inclusion of the background controls (column two) and service-delivery variables (column three) results in no change in the interventions' impact on the dependent variable. Even the addition of the Heckman Correction to the model (fourth column) makes little difference.

As for the impacts of the controls, the full-model results in column one of Appendix E 2b indicate that the chances of obtaining full-time employment increase with the length of time since the intervention. The chances are also increased by the amount of education the participant obtained prior to program entry, as expected. Similarly, earning $30,000 or more in the year prior to receiving assistance raised the chances of full-time employment (compared to earning less than $5,000). On the other hand, being 55 years and older reduce the chances of full-time employment (compared to being less than 35 years old). These are the only control variables found to have a significant impact on this dependent variable.

The results from the segmented analysis presented in Appendix E-2b indicate that participating in Partners and Entrepreneur both have positive effects on the likelihood of full-time employment for both males and females, with EAS having no impact on either group. However, being employed or unemployed one month before the intervention (compared to not in the labour force) reduces the likelihood for women only, as does being 55 years and older and being in a minority group. As well, the amount of education increases the likelihood for both men and women, while earning $30,000 or more increases it for men only. Having received EI for two or more years in the five years leading up to the intervention reduces the likelihood of full-time employment for men only.

By age, Appendix E-2b indicates that Partners and Entrepreneur continued to positively influence the chances of entering full-time employment for both age groups, as does having at least some post-secondary education. Also, earning more than $30,000 in employment income in the year before the intervention had positive effects on the younger age group only. As well, EI eligibility increases the chances of full-time employment for men only.

Finally, the results of the segmented analysis by claimant status presented in Appendix E.2b indicate that Entrepreneur had a positive impact on the likelihood of full-time employment for both EI claimants and reachbacks, while Partners had a positive effect for reachbacks only. Education had a positive impact on reachbacks, as does earnings level.

The third set of employment outcome modelling results presented in Exhibit 8.1c are for being employed for (at least) 12 consecutive weeks in the post-intervention period. These results correspond fairly closely to HRDC's definition of the employment result for the LMDA benefits and measures and is indicative of more stable employment. The program-only modelling results (column one) indicate that just Entrepreneur and Partners have a statistically significant positive impact on this employment outcome. However, the inclusion of background controls in the model (column two) and then service delivery variables (column three) adds SLG to the interventions exerting a positive impact on the outcome variable. However, once the Heckman Correction is introduced, SLG again does not exert a positive impact, though Entrepreneur and Partners continue to play a positive role.

Appendix E-2c indicates that Partners and Entrepreneur both exert positive impacts on the dependent variable. A number of controls also had an influence on the outcome variable. The results indicate that the chances of being employed for 12 consecutive weeks increase (not expectedly) with the weeks since the intervention and education level; are (surprisingly) higher for those unemployed one month before and those employed one year before the intervention; are higher for those with a prior interest in entering the labour force; and are (surprisingly) higher for those with three or more separations prior to the intervention (compared to those with 2 or less). Conversely, the probability of 12 consecutive weeks of employment declines with age and is lower for Francophones compared to Anglophones,³⁴ and for those who used self-serve services.

Appendix E-2c also presents the segmented results. By sex, both Partners and Entrepreneur exert positive impacts on the dependent variable for both males and females, as does the length of time since the intervention, interest in entering the labour force and having more than two job separations prior to program entry. Being employed one year before the intervention exerts a positive impact on the likelihood of 12 consecutive weeks of employment for females only, as does having at least a high-school certificate. Conversely, age, being a Francophone, and being in minority group negatively affects the dependent variable for males only. Being married has positive impact on the dependent variable only for males.

By age, the segmented results presented in the appendix indicate that, for both younger and older age groups, Entrepreneur exerted a significantly positive impact on participants' likelihood of entering 12 consecutive weeks of employment after the intervention and Partners for just the younger group. Being unemployed one month before the intervention and being employed one year before have positive influence on the dependent variable for the older age group only. Having separations before the intervention and an interest in entering the labour force both have a positive effect on the chances of 12 consecutive weeks of employment for the older group only, while being a Francophone has a negative effect for this group only.

By claimant status, Partners and Entrepreneur were found to positively affect the likelihood of three consecutive months of employment for near reachbacks only (Appendix E-2c). Weeks since the intervention positively affected both groups. Being unemployed one month before the intervention and being employed one year before have positive influence on the dependent variable for claimants only, as does interest in entering the labour force and pre-program separations. Being in a minority group negatively affects the dependent variable for active claimants only, while being 45-54 years of age (compared to less than 35 and being male negatively affect it for near-reachbacks only.

The OLS regression results for the fourth employment measure, weeks working since the intervention as a percentage of the weeks since the intervention, are presented in Exhibit 8.2. The results indicate that, with just the program variables in the model (column one), all programs were found to exert a positive influence on the percentage of weeks employed post-intervention, except Job Action, which was found to have a negative effect (column one). Adding the background variables to the model (column two) results in little change in the coefficients except that Rural Experience no longer has a significant impact on the outcome variable, as does the addition of the service delivery variables (column three). Finally, the addition of the Heckman Correction (fourth column) results in only Entrepreneur and Partners having a positive influence on the dependent variable, with EAS no longer playing a significant role, along with the other interventions.

As for the role played by controls, Appendix E-3 indicates that the time since the intervention increases the percentage of time working in the post-intervention period, as did prior interest in entering the labour force and having some post-secondary education (compared to having no more than some high school). Interestingly, being unemployed in the month prior to the intervention and having more than two separations in the five-years before the intervention also increased the time working. Also, being 55 years and older (compared to less than 35 years of age), being a Francophone, claiming over 25 weeks of EI benefits in the prior five-year period (compared to 24 or less), and using self-serve and non-HRDC employment assistance services reduced the percentage of time working since the intervention.

Turning to differences by gender, the results in Appendix E-3 indicate, first, that only females are positively impacted by SLG with respect to the percentage of time working since the intervention, with Entrepreneur exerting significant impacts on both males and females. Male participants were negatively affected by EAS. Also, women not men are positively affected by having some post-secondary education and prior interest in entering the labour force, and negatively affected by having weeks of EI claims prior to the intervention and using other employment assistance services. On the other hand, the negative age impact is only for men, as is having French as a mother tongue,³⁵ having up to 36 weeks of EI eligibility overlapping with the intervention, having received 25 or more weeks of EI prior to the intervention, and using self-serve services. Meeting a counsellor had a positive impact for men only.

The differences by age group include the fact that only younger participants are negatively affected by Job Action. Only older participants were positively affected by being unemployed in the month prior to intervention, being employed one year before, having a high-school certificate, and having more than two separations in the week prior to the intervention. Only they were negatively affected by being Francophone and having received 25-104 weeks of EI benefits prior to the intervention. As well, only younger participants were positively affected by having post-secondary education, speaking a language other than English and French, having a prior interest in entering the labour force, having three to five separations, and having met a counsellor. And they alone were negatively affected by having 1-36 weeks of EI eligibility, having received over two years of EI benefits before the intervention, having received SA benefits prior to intervention, and the use of self-serve as well as other employment services. As for differences by claimant status, Job Action negatively affected claimants only, and Rural Experience positively affected this group only; it was mainly EI claimants not near-reachbacks who were affected by the many of the factors identified above including participating in EAS (negative). One new factor was earnings of $10,000 or more which negatively affected EI claimants only. Being married had a positive impact on near reachbacks only.

The results for the per cent of weeks looking for work while jobless since the intervention are presented in Exhibit 8.3. With just the program variables in the model (column one), the results indicate that Job Action has a significant positive impact on the percentage of time job searching, whereas Partners and Entrepreneur have a significant negative impact.

Introducing the control variable weeks since the intervention and background controls (column two) does not change the role on job search played by Partners and Entrepreneur, but Job Action no longer has an impact. Moreover, SLG and EAS are now found to have a negative effect on job search, implying that the control variables — more than these programs themselves — are affecting job search. Adding the service-delivery variables (column three) does not change the impacts of any program variable except EAS, which was found to no longer have a negative effect on job search. Finally, the introduction of the correction factor (column four) results in only Entrepreneur having a negative effect on percentage of weeks looking for work, with Partners and SLG no longer having a significant effect.

As for the controls, Appendix E-4 indicates that the longer the time since the intervention, the lower is the percentage of that time spent looking for work, thereby implying discouragement on the part of the participant. Being employed one month before the intervention increases the percentage, as does interest in entering the labour force, being on social assistance one year before the intervention, and having used self-service and other employment assistance services. Negative factors include being employed one year before the intervention, speaking a tongue other than English or French, and having employment separations prior to program entry.

As for differences across segments, Appendix E-4 further indicates that, first, there are no differences between men and women in terms of program impacts but there are differences in the impacts of controls. For women but not for men, the percentage of weeks looking for work is positively affected by being employed one month before the intervention, having received over 24 weeks of EI benefits before the intervention, and using self-serve and other employment assistance services. Negative impacts for only women were found for being employed one year before the intervention and having a prior record of separations. On the other hand, men's and not women's job search is positively affected by being 55 years and over, having a prior interest in entering the labour force, and having received SA prior to program entry, but negatively affected by having a mother tongue other than French or English and being married. Having used self-serve services positively affects job search for both men and women, but meeting counsellor negatively affects men only.

By age, Appendix E-4 indicates that, in addition to Entrepreneur having a negative impact on job search for the both younger and older age groups, Job Action positively affects only the former. Other differences of note are that only the older group's job search is negatively affected by being employed one year prior to employment and positively by a pre-intervention interest in entering the labour. On other hand, only the younger group is positively affected by having 1-36 weeks of EI eligibility, 105 weeks of more of EI benefits, prior receipt of SA.

Finally, the percentage of weeks looking for work is negatively influenced by Entrepreneur only for active EI claimants and positively affected by Job Action. Also, only claimants' job search is affected by the time since the intervention (negative), being employed one month before (positive), being employed one year prior (negative), speaking a mother tongue other than French or English (negative), having a prior interest in being trained (positive), having a prior record of separations (negative), and having received SA before (positive). Only near-reachbacks' job search is positively affected by being 55 years and older and having used other employment assistance.

Earnings Outcomes

In Exhibits 8.4a, b and c, we present the results for three earnings outcome measures. These are weekly income from employment or self-employment in the current or most recent job at the time of the survey; absolute change in weekly earnings from employment or self-employment between the most recent job and the last job prior to the intervention; and percentage change in weekly earnings. Modelling absolute earnings change captures overall changes in the wage bill, while modelling percentage change captures relative changes. To illustrate the difference between the two different earnings-change measures, a $100 increase for someone earning $100 before the intervention represents a significant percentage change from the perspective of the individual (100 per cent), but not necessarily to the overall wage bill. It should be noted that segmented analysis was not conducted for earnings outcomes due to small sample sizes.

The first set of results for weekly earnings level (for the current or most recent job) indicates that, when only the program variables are entered into the model, Partners, Entrepreneur and SLG were found to exert a significantly positive impact (column one of Exhibit 8.4). Note that no controls such as prior earnings are entered in this first model; prior earnings would undoubtedly have an impact here. When we introduce the background controls (column two) and then service-delivery variables (column three), no change is observed in the impact of the intervention variables. Even the introduction of the Heckman factor (column four) fails to alter the impact of these interventions, with Partners, Entrepreneur and SLG still having a positive impact on current weekly earnings.

As for the impact of the controls, Appendix E-5a indicates that a large number of them affected post-intervention current weekly earnings. Exerting a positive impact were: the length of time since the intervention, being employed one year before the intervention, having a post-secondary education (compared to less than a high-school certificate), being male, and having received EI for at least one year over the 1992-1997 period. Also, post-intervention earnings rise with pre-intervention earnings level (compared to earning less than $5,000 in that period). Exerting a negative influence were being employed one month before the intervention (compared to not in the labour force), having French as a mother tongue, and having a pre-intervention interest in being trained.

Turning to segmented results by sex, we observe in Exhibit E-5a that, as far as program impacts are concerned, Entrepreneur affected males only with Partners and SLG positively affecting both males and females. Once again, it is women who are affected by the controls, apart the positive influence of weeks since the intervention and pre-intervention earnings for both sexes. Positively affecting women's weekly earnings only are having a secondary school certificate, and negatively affecting only the latter are being employed one month before the intervention, being a Francophone, and having no dependants.

By age, we observe that, while SLG and Entrepreneur positively affects both age groups, Partners affects only the younger age group. Education and being male positively affect both age groups, while a pre-intervention interest in being trained negatively affects both groups. Affecting only the older group are being employed one month before the intervention (negative) and a pre-intervention interest in entering the labour force (positively).

Finally, claimant segmented results indicate that Partners, Entrepreneur and SLG all positively affect the weekly earnings of reachbacks only, as does the length of time since the intervention. And, apart from being male increasing the earnings of both claimant groups and having a post-secondary education affecting only active claimants, other controls affect the earnings of only reachbacks. Affecting only the latter's earnings are: having a high-school certificate, weeks receiving EI in the years prior to the intervention, and pre-intervention earnings level (all positive) and having no dependants (negative).

The results in Exhibit 8.4b indicate that, with just program variables in the model (column one), Entrepreneur had positive effect on change in earnings, while Job Action, EAS and Rural Experience each had a negative effect. Then, with the inclusion of the background variables (column two), we observe that Entrepreneur continues to have a positive effect and it is joined by SLG. Rural Experience continues to have a negative effect, but Job Action no longer does, being replaced by Rural Experience. The addition of the service-delivery variables (column three) does not affect these outcomes, but the inclusion of the Heckman Correction (column four) results in Rural Experience no longer exerting a negative effect on the change in weekly earnings, with Entrepreneur and SLG continuing to exert a significantly positive effect.

Among the control variables (Appendix E-5b), weeks since the intervention, education, being 45-54 years old (compared to less than 45 years old), being male, having earnings of between $10,000 and $20,000 all had a positive impact on earnings growth. Playing a negative role were being employed one year before the intervention, having a pre-intervention interest in being trained, and meeting a job counsellor.

As for the segmented results by sex, Appendix E-5b indicates that the programs affect males earnings growth only. While being employed one year before negatively affect both groups, other controls affect the sexes in different ways. Having at least a post-secondary education positively affects females only, age positively affects males and negatively affects females, being in minority group negatively affects females only, and pre-intervention weeks of EI receipt and earnings level both positively affect female earnings growth only.

By age, Entrepreneur affects positively the younger age group only, but SLG was found to positively affect both age groups. Once again there were mixed results by segment, with only being employed one year before negatively affecting both age groups and being male positively affecting both groups. Having a high-school certificate positively influenced earnings growth of the older age group only, while the time since the intervention and pre-intervention earnings level affected only youth. Having a pre-intervention interest in being trained negatively affected the earnings of youth, while an interest in entering the labour positively affected the older group only.

By claimant status, this time it is the reachbacks who are the only group to be affected (positively) by the interventions. Once again, males positively affected earnings growth of both claimant groups. Weeks since the intervention positively affect reachback earnings only, as do pre-intervention earnings. Education level positively affects claimants only, while being employed one year before the intervention negatively affects the earnings growth of reachbacks only.

The results in Exhibit 8.4c indicate that three interventions increased the percentage change in weekly earnings. These were Partners, Entrepreneur and SLG. Adding in the background controls (column two), the service delivery variables (column three) and the Heckman Correction (column four) did not alter the effect of these interventions.

As for the controls (Appendix E-5c), the weeks since the intervention, being employed one year before, education level, being male, and earning $10,000 or more in the year prior to the intervention all have a positive impact on percentage change in earnings. Two variables had a negative impact: being a Francophone (which in turn may be linked to the economic conditions in the regions Francophones tend to be located in) and a prior interest in being trained.

The segmented results by gender presented in Appendix E-5c indicate that Partners and SLG both exert a positive impact on percentage earnings growth, but Entrepreneur positively affects males only. The only other differences by sex are that being a Francophone negatively affects females only, while having a pre-intervention interest in being trained negatively affects males only. As for age, Entrepreneur affects percentage earnings growth of males only as does the weeks since the interventions. Partners and SLG have a positive impact on both males and females. The only other difference by age is that being employed one year before the intervention positively affects the percentage earnings growth of the older age group only. Finally, all three programs noted above as positively affecting percentage earnings growth have an impact on near-reachbacks only, as does having received over two years of EI benefits in the five years before the intervention. Having no dependants negatively affects this group only as does use of other employment services. On the other hand, being employed one year before has a positive impact on active EI claimant participants only, while having a pre-intervention interest in training negatively affects this group only.

Income Support Dependence

In addition to the goals of sustained employment and increased earnings, LMDA seeks to reduce dependence on employment insurance (EI) and social assistance (SA). In this section, we present results for outcomes in these areas. As noted above, the EI results are for new spells of EI in the post-intervention period, beyond the spell that corresponded to their intervention. Note that a negative result is really the sought-after outcome.

For percentage of weeks on EI, column one in Exhibit 8.5 indicates that, with just the program variables in the model (column one), all programs had a significant impact on the dependent variable. Noting that a negative effect does in fact represent a positive outcome, Rural Experience, Job Action and Partners had a positive impact whereas Entrepreneur, SLG and EAS had a negative impact. However, introducing control variables (column two) indicates that Partners no longer has a positive impact, which means that the prior impacts (without the controls) were a function more of the characteristics of the participants than the program itself. Then, with introduction of the service-delivery variables, we observe in column three that SLG no longer exerts a negative impact. However, with the addition of the Heckman Factor (column four), SLG once again joins Partners, Entrepreneur and EAS in reducing the number of weeks on EI in the post-intervention period. Job Action is found no longer to increase post-intervention weeks on EI but Rural Experience continues to do so.

Looking at the impact of the control variables for the full model (Appendix E-6a), it is observed that the variables having a lengthening impact on EI spells are the weeks since the intervention, being Francophone, having three or more separations before the intervention, and being on social assistance one year before the intervention. Some variables acted to reduce the weeks in receipt of EI including having a post-secondary education and having overlapping weeks of EI eligibility.

As for differences in EI use across segments, Appendix E-6a indicates that it is only the males whose post-intervention EI claimant period is lengthened by Rural Experience, while only females' claimant period is reduced by Partners and SLG. Entrepreneur and EAS act to reduce the spells for both men and women. The effects of the controls include only females' post-intervention receipt of EI being positively affected by pre-intervention job separations and negatively by pre-intervention receipt of EI, while only males are positively affected by being married and receiving SA in the year before the intervention. By age, it is observed in Appendix E-6a that it is only younger participants whose post-intervention weeks of EI receipt are negatively affected by Partners. On the other hand, it is only older participants who are positively affected by being unemployed in the month before the intervention, a pre-intervention interest in being trained, and having received SA in the year before the intervention. Only younger participants' post-intervention receipt of EI is positively affected by a pre-intervention interest in entering the labour force and being Francophone and negatively by having received 25-52 weeks of EI in the pre-intervention period (compared to less than 25 weeks).

Finally, the claimant-status segmented analysis reveals some major differences between EI claimants and reachbacks. Appendix E-6a indicates that it is mainly the near-reachbacks weeks of post-intervention EI benefits that are affected by the interventions, as four different interventions reduce their EI duration: Entrepreneur, Partners, SLG and EAS. Rural Experience positively affects only active EI claimants. Having a post-secondary education negatively affects only near-reachbacks while a pre-intervention interest in starting one's own business positively affects only them. Having three or more separations and having received SA in the year prior to program entry positively affect only active claimants' weeks of EI benefits and not near-reachbacks'.

In Exhibit 8.6, we present the results of our efforts to model receipt of SA in the post-intervention period, the last outcome variable we modelled. The first column indicates that, with no control variables entered, Job Action, EAS and Rural Experience appear to increase the chances of receiving SA following the intervention. Introducing the background controls (column two) has no impact on the measured effect of EAS and Job Action but renders the effect of Rural Experience not significant. It is the nature of the clientele, rather than the program itself, which appears to have increased the chances of SA receipt. In the third column, we show that, after controlling for service-delivery variables, only EAS led to SA use. However, with the introduction of the self-correction factor, we observe in column four that none of the interventions play a role in subsequent SA use.

The results for the control variables presented in Appendix E-6b indicate that a number of background variables play a role in the use of SA. The variables significantly increasing the chances of using SA in the post-intervention period are receiving SA benefits in the pre-intervention period, the weeks since the intervention, and being a member of a minority group. Variables significantly reducing the likelihood of SA use are being married and having no dependants.

The segmented analysis of post-intervention SA use by sex, age and claimant status revealed some interesting differences between segments (Appendix E-6b). For males only, EAS had a positive impact, as did being in a minority group and pre-intervention receipt of SA. Being married and having no dependants reduced the chances of SA use for women only, as did being employed or unemployed in the month before the intervention. By age, we observe that Partners reduced the chances of SA receipt and that being employed one year before the intervention reduced post-intervention receipt of SA only for younger participants, as did having no dependants. Being in minority group increased chances of SA receipt for younger participants only. On the other hand, it was only older participants whose post-intervention receipt of SA was positively affected by being married. Finally, Exhibit 8.6 indicates that being married and having more than two separations in the pre-intervention period reduced the chances of SA use for just active claimants, while having three to five separations increased it for this group only. Being in a minority group increased SA use after the intervention for near-reachbacks, while having six or more separations reduced SA use for this group only.

Summary by Intervention

Exhibit 8.7 presents a summary of the results of the modelling exercises. The exhibit provides in each cell of the first line the significant effect (if any) of the program on the respective outcome for participants, and the second line indicates the, significant effects (if any) of the program in different segments. If no segment is separately affected by any of the programs, no information was entered in the second line of the cell. Note, once again, that the short-term nature of this formative evaluation tends to naturally favour interventions such as Entrepreneur and Partners that have immediate employment outcomes. A summary of the findings, based on the exhibit, follows:

• Partners — Controlling for other factors, this program had a significant positive impact on all employment outcomes, overall; this was also true for all individual segments but active EI claimants and except for percentage of weeks working where no segment was affected. Further, it had no impact on the weeks looking for work as a percentage of the weeks since the intervention, overall and in any segment. Partners contributed positively to current weekly earnings levels and their percentage growth from before to after the intervention, overall and for all segments but older and EI claimants. As for absolute earnings growth, Partners had no effect overall on it, but did increase it for just males. Finally, it reduced weeks of post-intervention EI receipt, overall but only for females, younger participants and near-reachbacks, and reduced the likelihood of receiving SA in the post-intervention period only for younger participants.
• Entrepreneur — Controlling for other factors, this program had a significant positive impact on all employment outcomes, overall and for every segment except for claimants in the case of 12 consecutive of weeks of employment. It reduced the length of job search in all segments but near-reachbacks. As well, Entrepreneur increased all three earnings measures, overall but generally only for males, younger participants and near-reachbacks. Further, it had a significant negative impact on the relative duration of EI receipt, overall and in all segments but EI claimant participants, but had no impact on SA receipt.
• Job Action — Controlling for other factors, the only employment impact this program had was a significant positive impact on being full-time employed just for older participants. It had no impact on relative post-intervention job search except to increase it among younger and claimant participants. There were no earnings or income-support dependence impacts found.
• SLG — Controlling for other factors, among all participants and for every segment, this program had no significant impact on all employment and job search outcomes, except for a positive impact on weeks working as a percentage of weeks since the intervention for females only. SLG positively affected all earnings measures, overall and for all segments but active EI claimants. SLG reduced post-intervention EI receipt, overall and for all segments but males and claimants. It had no impact on SA use.
• EAS — Controlling for other factors, this program had a significant positive impact on just two employment outcomes: currently employed (overall but only for the younger and older segments) and currently full-time employed (overall and only for the older segment). No significant impact was detected on the other two employment outcomes (except for a negative impact on weeks working for males only), the job search outcome, and the earnings outcomes. EAS was found to reduce post-intervention weeks of EI receipt, overall and for all segments but active EI claimants. No SA impact was detected overall, but EAS did increase the chances of SA receipt for males.
• Rural Experience — Controlling for other factors, this program had no significant effect on any employment outcomes. No impact was detected for the job search and earnings outcomes. Finally, Rural Experience increased post-intervention use of EI overall, and in all segments except for females and near-reachbacks. No impact was detected for SA use. Further, it is worth noting that the program's negative effects on currently full-time employed and change in current weekly earnings and its positive effect on SA receipt (with just program variables in the model) were nullified by the introduction of background controls, implying that the program's negative impacts may be compensated for by the characteristics of participants.

Client Profile Summary

We complete the summary by providing profiles of successful participants based on the modelling results. Starting with employment outcomes, those who participate in Entrepreneur do best, with Partners a fairly close second. EAS is also beneficial with respect to currently employed and full-time employed. Having a post-secondary education and having a prior interest in entering the labour force increase the chances of positive employment outcomes.

The finding of positive employment outcomes is generally true of the total population and for individual segments. The exceptions to these patterns, as revealed in the segmented analysis, include the fact that EI claimants do not benefit from their participation in Partners, that females do not profit from their participation in Partners in terms of being currently employed, and that only older segments' chances of full-time employment benefit from participation in Job Action and EAS.

With respect to earnings outcomes, participation in Entrepreneur, SLG and Partners, in that order, did make a significant difference, controlling for other factors. Attributes associated with success in this respect are being male, being less than 45 years old, having a post-secondary education, and earnings level prior to the intervention. The segmented analysis indicated that EI claimants did not tend to do well from the standpoint of earnings.

As for income-support dependence, Entrepreneur, EAS, SLG and to a lesser extent Partners were found, overall, to reduce the length of a new EI spell in the post-intervention period, controlling for other factors. Near-reachbacks were found to benefit from the interventions more than those who were EI claimants at the time of the intervention. Additionally, for Partners successful participants tended to be female. As well, having a post-secondary education, being eligible for EI overlapping with the intervention, and not receiving SA before the intervention acted to reduce the length of the EI spell, it is also important to point out that the longer the period of time from the intervention the greater the relative length of the new EI spell. Conversely, Rural Experience acted to increase post-intervention EI spells, overall, and for all segments but near-reachbacks.

As for SA receipt, Partners participation appeared to reduce the chances of receiving SA in the post-intervention period for the younger segments while EAS increased it for males alone. The chances were also increased by being unmarried, being in a minority group, having dependants, and having received SA in the year prior to the intervention.

View EXHIBIT 8.7

9.1 Rationale

Given that the PBMs focus on moving clients toward self-sufficiency, these programs are generally compatible with intent of the EI Act and the LMDA. In order to better link workers with employers, however, there is a need to improve the coordination of programming delivered by HRD-NB which deals with the worker/social assistance client and that of the Department of Labour which delivers active benefits and measures with the employer as the client.

Although the PBMs are perceived to overlap to a degree with other existing provincial and federal programs (e.g., the federal pan-Canadian programs for youth, Aboriginals and persons with disabilities), there do not appear to be any major problems of duplication or programs working at cross-purposes with one another. In the view of many key informants, any minor problems in this respect may be viewed as "growing pains" which will probably be sorted out as the LMDA implementation proceeds.

The PBMs appear to be generally relevant to the needs of clients and communities. The programs have sufficient flexibility to be adapted to client needs, they are providing helpful assistance to clients, and they are focused on helping people get back to work. Respondents expressed some concerns, however, about clients in need who "fall through the cracks". These include youth, persons with disabilities, people who are under-employed or who have had very little steady employment, and small— and medium-sized businesses in the midst of cut-backs (who cannot provide incremental employment opportunities and hence do not quality for wage subsidies under Partners). In many cases, however, these problems are viewed as stemming from the conditions of the EI Act rather than the LMDA per se. That is, it is the EI Act (and the definition of "EI clients") which renders some of these people ineligible for assistance.

9.2 Design and Delivery

There is still some confusion and discomfort over the new roles and responsibilities of the various players. For example: former federal staff who were transferred to the Province are still adjusting to their new role and environment/culture; there is a lack of shared understanding around roles and responsibilities for the reception function and for the service delivery model at co-located sites; some federal officials are unclear about their new role in the LMDA; staff are unsure about which level of government serves clients first in program areas perceived to overlap (e.g., programs for youth, persons with disabilities and Aboriginals); and some staff with HRD-NB and the Department of Labour are unclear about their respective responsibilities for Skills Loans and Grants. Much of this confusion appears to be due to inadequate communications both within and between levels of government.

The consensus was that the current players/partners — HRDC, HRD-NB and the DOL — are probably sufficient at this stage, and that the LMDA is complicated enough at present. Later, after the LMDA is fully implemented, there may be room for additional players to play a bigger role in LMDA consultation, planning and design. These partners might include private sector employers, the not-for-profit sector, local community organizations, and economic development commissions.

Several aspects of the LMDA implementation were regarded as successful. For example, the transfer of funds through the federal authorities and the process of transferring HRDC staff to the Province — including staff communications, sound change management and careful attention to staff's package in terms of money and protection — went smoothly. In addition, good cooperation among the three players and two levels of government as well as a willingness to work together in the field, through local implementation teams and regional workshops, greatly facilitated implementation. For the most part, program delivery and good client service were considered to have been maintained throughout the implementation process.

On the other hand, some aspects of the implementation process were not perceived as successful. For example, resources for implementing the LMDA have been a major issue and source of disagreement between the federal and provincial sides. In particular, provincial officials perceive that they are entitled to more resources to assist with the considerable costs associated with implementing the LMDA and the co-located sites. In their view, these costs were not sufficiently thought through when the Agreement was negotiated.

Related to this, the initial negotiation process was a source of dissatisfaction for both federal and provincial partners. Due to pressures on both sides, the Agreement was signed hurriedly with the understanding that many of the details — presented as general management concepts rather than in practical terms — would be worked out as implementation proceeded. Many of these details remain to be worked out, causing confusion and frustration for all parties.

In addition, some key informants identified the lack of a dedicated project manager as a significant barrier to proper LMDA planning, co-ordination and implementation. A project manager could have overseen and co-ordinated the overall LMDA implementation, facilitated communications between the three players, and dealt more effectively with the various problems as they arose. Inadequate information and monitoring systems, and poor integration of the systems operated by the three players, have also impeded proper planning and management of the LMDA.

Joint implementation planning has been attempted at both central and local offices, and this has been helpful in terms of sorting out roles and responsibilities. There have, however, been many delays in implementing plans (e.g., delays related to the transfer of federal employees, problems related to resources and incompatible information systems, delays in co-location). Also, as noted above, the lack of a dedicated project manager was viewed as a significant barrier to proper LMDA planning, co-ordination and implementation.

Given the complexity of the task of implementing the PBMs and the LMDA, there has been remarkably little disruption to client service in the view of many respondents. One exception has been the disruption to client service due to the delay in implementing the loan component of Skills Loans and Grants. Clients have called staff to ask when the full SLG program would be available, but have not been given a definite answer. Rather, clients have been asked to check in later to find out whether or not they will be receiving a loan. This uncertainty has interfered with clients' planning of their training/education. Also, the delays in co-location have created some confusion for clients, who may be sent to different places for assistance (e.g., the HRCC, HRD-NB and job clubs).

The LMDA implementation process has been somewhat disruptive for staff. Although the initial transfer of federal employees apparently went quite well (e.g., in terms of communications, change management and the financial package), some staff now working for the Province feel isolated and confused, and are having difficulty adapting to their new environment. Also, many staff continue to feel the need for clear communications regarding the PBMs and their new roles and responsibilities. For instance, a single, automated, user-friendly information system on the new programs would be very helpful for front-line staff.

There has been a history of good cooperation between the federal and provincial governments in New Brunswick, and the LMDA has provided an opportunity for the partnerships among HRDC, HRD-NB and DOL to become stronger. The New Brunswick Labour Force Development Board (with labour, management, equity groups and educational institution representatives) was also consulted regarding the LMDA implementation. In addition, consultations have been done with community organizations, though some respondents in the evaluation felt that these could be improved.

As noted above, much work remains to be done in adapting information systems for the monitoring of program delivery and results. There appears to be a need for clarification on accountability requirements and who is responsible for what results, for a clear definition of valid results measures, and for better integration of the (currently incompatible) information systems operated by the three players. There are also related client privacy issues because client information will need to be shared among the players. Systems officials feel the need for direction on these points from program management.

Turning to co-location, the impacts to date can be summarized as follows. (1) Service delivery has either improved or stayed the same. The improvements include: greater convenience of one-stop service; better resource centres for the provision of Labour Market Information; faster processes in approving some DOL programs; and better service for social assistance clients. (2) Clients are generally satisfied with the fully co-located site and like the convenience, but they do not perceive a significant difference in the service they receive. (3) At the partially and fully co-located sites, most staff did not feel that client service has been altered, but some felt that it had declined due to poorly conceived reception services, impeded client flows, and an increase in the number of required client contacts as well as conflicting departmental operations. Staff were more likely than clients to perceive that co-location has had a negative impact because they see the "behind the scenes" difficulties imposed by the coming together of the three partner organizations.

Co-location has not led to poorer service compared to when these types of services were delivered by the federal government, but neither has it created integrated information services and seamless service delivery. Staff had high expectations that co-location would increase interdepartmental communication and increase access to information on the other departments' programs and clients, and consequently better equip them to provide good service. Because this has not materialized to the extent they envisioned, staff members have been frustrated and disappointed.

In order to improve the delivery of PBM services through a co-located approach, the three partner departments will need to address the following issues identified in the evaluation: the harmonization of departmental operations and service methods; the provision of reception services that serve all three departments as well as the different target client groups, such that reception services can become a central information source for all clients' inquiries; shared responsibility for administrative costs; re-examination of the information needs of each department and improved information exchange; improved interdepartmental communications; and the establishment of a streamlined management structure that can resolve problems quickly and that represents the interests of an integrated HRSC. Co-location should be more than having all employment-related services under a single roof; it will require integration of the three departments' services and meaningful cooperation.

Considering service delivery, the qualitative evidence suggests that, for the most part, the programs are believed to be meeting clients' needs and to have sufficient flexibility and capability to adapt to local needs. The high degree of local flexibility in the delivery of SLG has however raised some concerns about consistency between case managers and regions. In addition, the promotion of all PBMs could be improved. Most worker clients and employers seem to learn about the programs by "word of mouth" and feel that programs are not well known.

Many target groups such as youth, older women and persons with disabilities are perceived to have less access to programs than others because they are not eligible for EI. Both levels of government are still sorting out their respective responsibilities with regard to Aboriginals.

Shared access to LMI and Electronic Labour Exchange systems among the three partner departments is still being sorted out. Technical problems have been a major obstacle.

Employer survey findings reveal that, overall, employers who have participated in wage subsidy programs tend to be fairly satisfied with the programs, although there is room for improvement in several program areas, notably marketing and outreach, and in the timeliness of certain program components (i.e., non-reimbursement period). Employers also found that the quality of program participants, the programs themselves, and service received from the government have improved slightly since the LMDA was implemented.

The results of the participant survey also suggest that, in general, participants are quite satisfied with the services they have received. The highest satisfaction ratings were obtained for services received from a community agency, followed by government services and their most recent employment program. The lowest satisfaction ratings were generally obtained with respect to the information on available programs and services, consistent with the qualitative findings indicating low client awareness of PBMs.

Based on employer and participant suggestions, the largest improvements to the programs could be made through better promotion and advertising, increased relevance to the needs of employers and clients, broader eligibility for programs, and more flexible regulations associated with their use.

9.3 Success to Date

Accountability is an important component of the LMDA. An indicator of this is the extent to which three results targets set for the LMDA are met. First, for the 1997/98 fiscal year the 1998 EI Monitoring and Assessment Report found that 5,546 (roughly 70 per cent) of the targeted 7,947 for all NB LMDA benefits and measures was attained. In this evaluation, we estimated that 6,261 of the some 8,000 PBM participants in the PBMs under study returned to work for 12 consecutive weeks. This indicates that the administrative data outcome indicator which applied to all benefit and measures likely underestimate returns to work. By February 1999, over 10,000 PBM participants had returned to work. Second, the 1998 EI Monitoring and Assessment Report found that roughly only 50 per cent of targeted savings from unpaid EI claims, due to early returns to work, were achieved. Our estimates indicated that for the benefits and measures under study there was $6.6 to $7.9 million in unpaid EI benefits in the 1997/98 fiscal year and a total of roughly $11.3 to $14.1 million over the 1997/98 and 1998/99 fiscal years. Third, only 56 per cent of participants in the PBMs under study were active EI claimants, somewhat below the target of 65 per cent, which applies only to Provincial Benefits.

Evidence gathered in this evaluation enabled us to assess the success to date from the perspective of clients, employers and communities.

Note that the measures of impacts discussed here should be considered preliminary only and that more definitive measures will be presented in a summative evaluation when there will be more time available to detect impacts. Interventions such as Entrepreneur and Partners which have immediate employment outcomes would be expected to have more favourable outcomes in a formative evaluation than the other interventions where employment effects are not immediate.

Client Outcomes

Employment

Over one-half of PBM participants with jobs at the time of the survey stated that their participation was important to attaining those jobs.

• This percentage was particularly high for Partners and Entrepreneur and was somewhat higher for reachbacks compared to EI claimants.

Direct observation of post-intervention employment status indicates that, on most counts, participants exhibited better employment outcomes than non-participants.

• Over 60 per cent of participants were employed at some time following their intervention.
• Particularly high percentages (88-96 per cent) were reported for Entrepreneur and Partners participants.
• Less than one-half of SLG participants were employed following completion of their participation, however.
• EI claimants were somewhat more likely to be employed than reachbacks and members of the comparison group.

The employment of Entrepreneur and Partners participants in the post-intervention period tended to be more stable.

• SLG graduates were the least likely to occupy jobs lasting at least 12 weeks while EAS participants were the least likely to have had only one employer. Older workers were the most likely to occupy longer lasting jobs.
• EI claimants tended to occupy longer lasting jobs than reachbacks, implying an advantage for the former.

Entrepreneur and Partners participants were most likely to be employed (the former self-employed and the latter in full-time jobs) following the intervention.

• Job Action and Rural Experience participants experienced the highest levels of unemployment in the post-program period, at about 50 per cent.
• Over the post-program period (i.e., between the first week and currently), participants in most programs experienced increases in employment incidence and school attendance and declines in unemployment incidence.
• While comparison group members were more likely to be employed than EI claimant participants, the incidence of full-time employment and the proportion going to school was higher in the latter.
• One-quarter of those unemployed and those not in the labour force before the intervention found full-time jobs and close to 10 per cent found part-time jobs after the intervention. And one-fifth of pre-intervention part-timers changed their status to full-time.

Job Quality

The vast majority (almost 90 per cent) of employed participants were working more than 30 hours a week in their current job.

• There was little variation across interventions, apart from the relatively high incidence of part-time hours among participants in Job Action and EAS and the high incidence of long hours among Entrepreneur participants.
• Male participants tended to work more hours than female participants did.
• About half of participants occupied year-round jobs with the highest incidence reported by Partners and Entrepreneur clients.
• Comparison group members were more likely to be in year-round jobs and less likely to be in casual/contract employment than EI claimants.
• Weekly pay was highest for SLG participants and lowest for Job Action participants.
• EI claimants earned somewhat more than reachbacks, reflecting the relative work experience of the two groups.
• Participants earned more than the comparison group, implying some advantage for those making use of PBMs.

Retention and Non-Completion

Just under one-half of wage subsidy participants were retained by their host employer.

• This rate was particularly high for Partners, relative to Rural Experience and Job Action, because of the project based nature of the latter two programs.
• EI claimants were somewhat more likely to be retained than reachbacks.
• Among the reasons for not hiring workers after the wage subsidy, the dominant reason suggested was a lack of resources, particularly by Job Action and Rural Experience employers.
• Another important reason suggested was that the work was only temporary or seasonal, particularly among Partners and Rural Experience employers; the latter also frequently mentioned lack of work.
• One-third of employers "lost" hires before the end of the wage subsidy.
• This proportion did not vary appreciably across the different wage subsidy programs.
• Rural Experience employers were the most likely to lose workers.
• The reason for losing hires most frequently mentioned by employers was that participants quit before the end of the subsidy. Lack of amenity to training, poor attitude, incompetence, personal problems and finding another job were also frequently mentioned.

Incrementality

Almost three-quarters of wage subsidy employers said the positions they hired workers into under wage subsidy were incremental.

• About one-fifth said they would have hired somebody without applying for assistance (complete non-incrementality).
• Job Action employers were the most likely to say the jobs they hired workers into under wage subsidy were incremental.
• Incrementality rose with the age and size of business and was high for municipal organizations and seasonal and low-growth businesses.

Joblessness and Job Search Outcomes

Participants on average experienced joblessness for about 30 per cent of the time following program participation and as of the survey date.

• The mean was greatest for Rural Experience and Job Action participants (36 and 43 per cent) and lowest for Entrepreneur participants (about eight per cent).
• EI claimants experienced less joblessness than reachbacks, reflecting work history, and than the comparison group, implying some advantage for PBMs.
• Joblessness rose with age, is higher for females, and declines with education level.
• The job search results indicated that Entrepreneur participants were the least likely to look for work while jobless. This may be because Entrepreneur clients can draw EI benefits while setting up their business and draw for the entire period without having to report earnings.
• Job Action and Rural Experience participants looked for work for the greatest percentage of time since the intervention.
• Comparison group members tended to be jobless longer and to look longer for work than participants, implying some advantage in favour of the provincial benefits and measures.
• EAS participants were most likely to send out resumes and rely on newspapers than participants in other PBMs.
• Rural Experience and Job Action participants tended to rely on word of mouth to find jobs.
• Sending out resumes was associated with being younger, female and better educated.

Utilization of Income Support

About one-quarter of participants received EI (new spell) following their intervention, ranging from three per cent of Entrepreneur participants to over 50 per cent of Job Action and Rural Experience clients.

• Non-participants were somewhat more likely to be on EI than participants, implying perhaps some advantage in participating in PBMs compared to other types of employment assistance.

Only about one-tenth of participants received SA in the post-intervention period.

• The highest incidence was reported among Job Action and EAS participants, who also were on it for the greatest percentage of time. Reachbacks were more likely to have been on SA than EI claimants, but comparison group members were less likely, though when they did collect it, it was for a greater percentage of the time.

Attitudes and Skills

Qualitative and quantitative evidence gathered in this evaluation indicates that participants' experience in the PBMs increased their skills and enhanced their attitudes to work and learning.

• Among the specific psychological outcomes identified were increased confidence, a feeling of accomplishment and dignity, and an understanding of the benefits of increased contributing to one's own human capital development.
• From a labour market perspective, participants benefited from increased labour-market intelligence, a strengthened ability to set career goals, and increased job-specific and job-search skills.
• Participants reported benefits were corroborated by the fact that in the survey between 75 and 89 per cent of wage-subsidy employers believed that participation in the programs benefited clients with respect to their attitude to learning and training, job-specific skills and preparedness and job readiness.
• The greatest difference was between Job Action and Partners regarding impacts on attitude to work and learning (89 versus 74 per cent). Six in 10 employers thought that participants were job ready following the end of the wage subsidy.

About eight in 10 employers provided job-specific and orientation training to PBM participants, which is true of participants in all wage-subsidy programs.

• About one-half provided task rotation, personal skills training, mentoring and job shadowing to their hires.
• Computer training and career counselling were the least frequently provided (particularly in Rural Experience) as was job-search advice (particularly in Partners).

Modelling Results

Econometric modelling was carried out for three sets of client outcome measures: employment, earnings and income support use. In this analysis, the outcome measures were explained in terms of the interventions while controlling for the time since the intervention, sociodemographic and employment history characteristics, and service-delivery use. Separate analyses were run for specific sex, age and claimant status segments. The latter segments include active EI claimants and "near-reachbacks" who are former EI claimants whose claim ended within six months of the intervention in question. Note again that more definitive results will be obtained in a summative evaluation when there will be more time to detect impacts of interventions such as Job Action, EAS and SLG which have a longer "gestation" period.

For the four employment measures — currently employed, full-time employed, employed at least 12 consecutive weeks in the post-intervention period, and weeks of post-intervention employment — participation in Partners and Entrepreneur consistently led to positive outcomes, compared to non-participants in these programs. Generally speaking, this result held after the control variables were introduced into the models, and across most sex, age and claimant status segments. The main exceptions were that participation in Partners did not benefit active EI claimants with respect to all employment outcomes, females with respect to being currently employed, older participants with respect to 12 consecutive weeks of employment, and no segments with respect to weeks working in the post-intervention period. Also Entrepreneur did not benefit claimants with respect to 12 consecutive weeks of employment. In general terms, having a post-secondary education and a prior interest in entering the labour force were significant predictors of employment success.

The results indicated that other interventions also had specific employment impacts in specific segments. With respect to the likelihood of being currently employed, besides Entrepreneur and Partners having a positive impact for most segments, EAS had a positive influence for older and younger participants. Once again, higher education and an interest in entering the labour force positively contributed to this outcome. On the other hand, being male, being in a minority group and having no dependants contributed negatively. Interestingly, being unemployed one year before the intervention positively affected the chances of post-intervention employment.

As for the likelihood of attaining full-time employment, besides Partners and Entrepreneur, EAS and Job Action benefited older participants. Once again, higher education was a good predictor of positive outcomes in most segments. Earning over $30,000 was also a good predictor of full-time employment for female, younger and near-reachback participants.

Turning to the chances of being employed for more than 12 consecutive weeks after the intervention, no other intervention had an impact on this outcome besides the across-the-board positive impact found for Partners and Entrepreneur. Not entirely unexpectedly, the chances of 12 consecutive weeks of employment increased with the time since the intervention in all segments. Higher education contributed to positive outcomes, particularly for female and younger participants, as did having just a high-school certificate for all segment but males and claimants. Age contributed negatively, particularly for females, males and EI claimants. Having French as a mother tongue also negatively affected this outcome in most segments, which likely reflects the fact that this group resides in regions of the province where economic conditions are poorer. Having a prior interest in entering the labour force and a prior history of separations were success factors for all segments but younger and near-reachback participants. Interestingly, use of self-serve services had a negative impact on the outcome variable, overall and for claimant participants.

For the fourth employment measure — weeks employed since the intervention — SLG was found to exert a positive influence but only for females, EAS negative for males, and Job Action negative for claimants, in addition to the positive impacts of Entrepreneur (overall and in all segments) and Partners (overall). Other interesting findings included the following:

• The time since the intervention increased the percentage of time employed since the intervention in all segments but near-reachbacks.
• Having post-secondary education increased the percentage of time employed for female, younger and claimant participants, while having a prior interest in entering the labour force increased it for all segments but older and near-reachbacks participants.
• Interestingly, both being unemployed in the month prior to the intervention and employed one year before were success factors in most segments, as was a prior history of separations.
• Use of self-serve employment assistance services reduced the percentage of time employed for male, younger and claimant participants. Meeting a counsellor was a positive influence, overall and for female and younger participants

Modelling results for the percentage of weeks looking for work indicate that only Entrepreneur reduced the percentage of time looking for work, overall and in all segments but near-reachbacks while Job Action was found to increase the percentage only for younger and claimant segments. In fact, near-reachbacks' job search does not appear to be affected by any intervention. Generally, this outcome variable is negatively affected by controls for weeks since the intervention and the number of separations prior to the intervention. Prior SA use and the use of self-serve employment assistance services positively affected job search in all segments.

Turning to earnings outcomes, Partners, Entrepreneur and SLG were found to have a positive impact on all three earnings outcomes. For SLG this was generally true for all segments, while for Entrepreneur and Partners it was generally true for male, younger and near-reachback participants. Other interesting findings include:

• Partner and Entrepreneur had no impact on the earnings of older and claimant participants. Among the controls exerting a positive influence on current weekly earnings were: having a post-secondary education, being male, receiving up to two years of EI benefits in the years prior to the intervention, and having higher earnings before the intervention.
• As for absolute earnings growth, Partners had an impact on only males, Entrepreneur on only male, younger and near-reachback participants, and SLG on all segments but female and claimant. Being employed in the year prior to the intervention and a higher education positively affected earnings growth. Age was a positive factor for males but a drawback for females.
• For percentage earnings growth, SLG increased it in all segments but claimants and Partners just for males, and Entrepreneur for male, younger and near-reachback participants. Also, the weeks since the intervention, being employed one year before the intervention, education, being male, and higher were found to be positive influences. A prior interest in being trained, use of other employment services and being Francophone contributed negatively to earnings growth, the latter likely because of concentration in areas with poor economic conditions.

For percentage of weeks receiving EI benefits, Entrepreneur, SLG and EAS acted to reduce EI use in all segments but claimants, while Partners reduced for only female, younger, and near-reachback segments. Rural Experience increased it in most segments, while Job Action played no role. As for the impact of the controls, weeks since the intervention (in most segments) and having a prior history of several separations (in all segments but male and near-reachback segments) increased the weeks of EI receipt, while having a higher education (just for near-reachbacks) and several weeks of overlapping EI eligibility (in all segments) reduced the weeks.

Finally, the modelling of post-intervention SA use indicated that Partners led to reduced use of SA among younger participants only, while EAS led to increased use of SA among males only. Prior use of SA increased the chances of post-intervention SA use in all segments but older participants, as did being in a minority group but only for male, younger and near-reachback participants. On the other hand being married and having no dependents reduced the chances of post-intervention SA use, the latter among all segments but male and older segments and the former among female, older and claimant participant groups.

Employer Outcomes

The focus groups and case studies indicated that employers benefited from their participation in the wage subsidy programs.

• Increased sales resulted from the two ASI up-side adjustment situations studied for this evaluation.
• Other benefits included increased management and marketing knowledge, increased confidence in the company's direction, a business plan and marketing strategy, and increased awareness of business-support services available.

Results from the employer survey indicated that their participation in the wage-subsidy programs benefited their organizations, with the results not varying much across the different wage subsidies.

• Over eight in 10 employers thought the wage subsidies had a positive impact on their organization overall.
• The widest gap in reported impacts occurred between Partners and Rural Experience employers in terms of the organization's ability to evaluate new employees (85 versus 62 per cent).
• The extent positive impacts were perceived declined with organization size.

The areas where participating employers reported they had incurred significant costs were in supervision/training and compensation (39 and 31 per cent of employers).

• This was true of all wage-subsidy programs but particularly Partners employers, who also frequently cited benefits paid as a significant cost of participating in the wage subsidies.
• Wages were most frequently mentioned as a significant cost by private sector employers.
• Over 40 per cent mentioned no significant costs, with non-profit organizations most likely not to do so.

Community Outcomes

There was varying opinion on the extent to which the benefits and measures had benefited the community.

• Key informants were not able to detect many impacts.
• Some employers participating in the focus groups, however, thought that the programs resulted in enhanced infrastructure for the community, which will benefit it in the long run.

In the two case studies of ASI community development adjustment experiences, it was found that communities did benefit from the ASI's committee's work.

• In both communities, a community action or strategic plan was implemented and morale was increased as a result of the projects.
• In one, the plan resulted in real benefits for the community, such as purchase of land for an industrial park and negotiations with prospective employers.
• In the other, the plan was developed for the sponsoring community organization, which strengthened its capacity to be more financially self-sufficient and thus to place greater numbers from the community into jobs.

9.4 Lessons Learned

The following is a preliminary overview of the major lessons that were learned over the course of the formative evaluation. The major lessons were:

• The evidence from most data sources in the study suggests that planning and coordination of activities related to implementation could have been better. To the extent that this did not occur in New Brunswick, it is likely that the speed with which the LMDA was negotiated and signed had a significant impact on the federal and provincial governments' ability to plan different aspects of the implementation of the agreement effectively. Specific areas in which planning and co-ordination of effort
  • Estimation of the schedule and costs of co-location in consultation with local authorities;
  • Plans for interim arrangements for service delivery prior to co-location;
  • The co-ordination of service delivery efforts within and among government departments;
  • Consistency in the application of programs, most notably with respect to the SLG program where minimum service delivery standards must be defined to ensure consistency in program delivery without compromising local flexibility (With respect to the SLG program, a consultation committee eventually filled that gap by establishing specific guidelines and standards to guide case management.);
  • Central project management to oversee and co-ordinate all aspects of the LMDA implementation; and
  • Proper attention to change management for government staff, as many felt isolated and confused with the change in roles and responsibilities resulting from the LMDA.
• A major issue for many key informants involved in the delivery of services under the LMDA concerned a lack of communication and team building within and among departments. One tangible result of this lack of communication was the low level of awareness among program staff concerning the appropriate target groups for PBMs.
• Overall, programs and services were working quite well to meet the needs of individuals, employers and the community and clients tended to be satisfied with the services received. The primary exceptions, however, included the promotion and marketing of programs and services, as well as various administrative aspects of programs (e.g., timeliness of approval process and non-reimbursement period for wage subsidy programs).
  • All programs and services are being delivered in both official languages.
• There was much evidence to show that many people involved in the design, delivery and receipt of programs and services believe the eligibility criteria should be broader to reach a wider population (although the criteria are set in the EI Act and not the LMDA per se).
• Existing data systems are not adequate to properly track and monitor the impacts of LMDA programs, and the attainment of targets. Difficulties here involve confusion over accountability requirements, the lack of clear definitions concerning impacts to be measured, and incompatible systems in place between provincial and federal governments.

9.5 Recommendations

A review of findings from the formative evaluation suggests that a number of concrete steps can be taken at this point in the implementation of the LMDA to improve the effectiveness of the overall LMDA infrastructure. These recommendations are as follow:

1. An overall internal communications strategy should be developed to address intra and inter-departmental concerns. Such a strategy would need to encompass issues related to clarifying the respective roles and responsibilities of both staff and LMDA partner departments, as well as changes in service delivery that have resulted from the implementation of the LMDA. One possibility may be to develop a single, automated, user-friendly information system on new programs to assist front-line staff.
2. Decisions around implementation issues of mutual concern to all partner departments will need to be made. These decisions would address issues related to the promotion of LMDA programs and services, as well as other service delivery issues, such as reception and appropriate signage in HRSCs.
3. The responsibility for serving the Aboriginal clientele, especially those living on reserves, needs to be clarified between federal and provincial partners.
4. With respect to promotion, an external communications strategy needs to be developed to raise awareness among active EI claimants, whose participation in the benefits and measures was well below target.
5. Further qualitative research needs to be conducted into the reasons for the high uptake among reachbacks, particularly in the Entrepreneur program. The research will identify whether the high uptake can be attributed to actions on the part of client service officers, to a larger than expected demand for assistance among reachbacks, or to some other factor.
6. Greater integration of information systems maintained by the three government departments would allow for proper monitoring and results tracking, thus greatly facilitating the management of program and service delivery, as well as the subsequent determination of impacts at the provincial and local levels. Related to this, there is a need for clarification of accountability requirements and who is responsible for what results, for a clear definition of valid results measures, as well as for a resolution of client privacy issues to improve information sharing among the LMDA partners.
7. Changes are required to ensure that programs are properly targeted to clients, as evidenced by the smaller than expected proportion of EI claimants who participated in LMDA programs. Related to this, some evidence suggests that EI clients were not being referred to PBMs. All this suggests that the lack of results target attainment may be related to issues of service delivery, inter-departmental communication and inappropriate targeting.
8. Greater co-ordination of program delivery among all three government departments would enhance service delivery and improve the reach of LMDA programs. These efforts would cover such areas as hours of operation and client referrals between departments.
9. The appointment of a dedicated project manager to oversee the LMDA implementation (including overseeing action on these recommendations) would minimize further difficulties that may arise throughout the remainder of the implementation process. Among the potential benefits of this appointment, such a manager could facilitate communications between the three departments, troubleshoot further difficulties that may arise and work to establish clear guidelines for many of the implementation issues that were not resolved prior to the signing of the LMDA (i.e., the establishment of respective responsibilities for administrative costs).
10. To address concerns over inconsistency across service delivery sites, the different sites should be encouraged to exchange views and experiences through e-mail, a special constructed Internet site or some other means.
11. Further research will be required, particularly in the summative evaluation, to make a more definitive assessment of the relative impacts of different programs on employment and other outcomes. It could be premature to make concrete recommendations about specific interventions based on the short-term outcome results generated by this formative evaluation. The main reason is that the PBM programs target different employability needs and would thus be expected to require different lengths of time before the full impacts are felt. For instance, the observed positive outcomes for the Partners and Entrepreneur programs in this formative evaluation are not surprising given that both these programs target career decision-making using a job placement strategy typically leading to immediate employment outcomes. Conversely, the limited impact of other interventions such as SLG may be due to the fact that they target an employability need (e.g., skill enhancement) necessarily requiring a longer time horizon to realize and detect labour-market impacts.
    • Based on the observed short-term impacts from this formative evaluation, it is possible to articulate certain hypotheses regarding possible modifications to certain interventions that may improve their effectiveness but which would require further research to substantiate. For instance:
      • the finding that self-serve products led to negative outcomes for men and younger participants may simply reflect the fact that the use of such products distracted these groups from seeking more intensive employment assistance. Further analyses may be able to shed light on the true impact of these products and suggest ways to improve the promotion of programs and/or referrals among LMDA partners; and
      • the poor showing for Rural Experience and Job Action relative to Entrepreneur and Partners in terms of employment outcomes is not surprising given that the former are designed to provide only short-term employment experience. Nonetheless, the short-term nature of these job placements may encourage recurring use of EI. Further research on longer-term impacts may be able to shed light on the extent to which this may be occurring.

Participant Survey

Data Acquisition

Participant data came from 10 different data source files, sent electronically at various times by Human Resources Development Canada (HRDC). With the exception of the EAS participants list, data lists were originally prepared by the Department of Labour. The EAS participants list was originally prepared by HRD-NB, while a file of matching administrative data for all EI clients over a three to five year period was prepared by HRDC. These data include:

• Participants in data file called CDF.RESU.XLS, including participants in the following intervention types: Partners/Wage Subsidy, Entrepreneur, Job Action/Creation, Skills Loans/Grants, and Employment Assistance. Received May 15, 1998 from HRDC (Strategic Services, Fredericton). No start or end dates for interventions were included; N=4,693.
• Participants not in the above file (i.e., not in CDF.RESU.XLS), including participants in the following intervention types: Job Action Top-up, Teachers Aid, SEB, Rural Experience, 1997-98 Partners, and 1996-97 Partners. Received from HRDC (Strategic Services, Fredericton), May 15, 1998; N=7,293.
• EAS participants. Received December 22, 1998 from HRDC (Fredericton); N=6,734.
• SLG participants. Received December 23, 1998 from HRDC (New Brunswick); N=250.
• Participants intervention start and end dates. Four files of start and end dates for interventions were received December 23, 1998 from HRDC (Strategic Services, Fredericton).
• Date of birth and language preference information from T1 data, received January 20, 1999 from HRDC (Hull).
• Additional EAS cases. Received February 11, 1999 from HRDC (Fredericton); N=50.

Data Issues

The first step in the sample development process was to merge the 10 participant data files and match them to the five administrative files to produce a single overall data file of 17,570 Cda/NB LMDA participants. When the 10 participant case files were merged together and matched to the five administrative information files, however, it was discovered that 250 participant cases did not appear in any of the administrative data files. As well, while addresses and phone numbers were found in all four participant files, as well as the "Name and Address" administrative file, there was a good deal of missing information in each file. Moreover, the data that were available were often inconsistent, which was most likely the product of inconsistent data collection protocols and changes of address owing to the fact that the data were provided by five different sources and were collected at different points in time. Where there was discrepant address and phone number information for a given case, the most recent complete entry was retained in the overall data file. Where data were missing for a given case, the most complete data from any source were retained.

Data Sampling

The data files were originally developed to include participants who participated in LMDA employment programs and services at any time between April 1, 1997 and October 31, 1998. As mentioned, these files were aggregated, yielding a single overall data file containing information for 17,570 participant cases, with the individual client as the unit of analysis. This was not equal to the sum of all the cases from the administrative data files because clients that had taken part in more than one intervention could appear in more than one file. Following the removal of all cases without valid phone numbers, start and end dates for EI benefits, and start and end dates for most recent interventions, the final data file consisted of 14,807 individuals.

The survey sample was randomly drawn from the final data file using a three to one "sample to survey completion" ratio for each different participant group (i.e., three times as many participants were sampled as were expected to complete the survey). For the EAS, Job Action and Entrepreneur participant groups, however, there were not enough cases available to obtain this three to one ratio. Thus, for an expected total of 1,665 survey completions, a total final sample of 4,564 cases was drawn from the data file of 14,807 program participants.

It is important to note that the small number of program participants in EAS proved insufficient to obtain the expected number of survey completions. To partially compensate for this, Ekos received an additional 50 unique names and phone numbers of EAS participants from HRDC in Fredericton. Although this provided a few more survey completions with EAS participants, there were still far fewer completions than originally expected (117 as opposed to 235). Thus, the final total number of completions for the participant survey was 1,600 rather than 1,665 as was originally anticipated.

Response Rate

Fieldwork for the survey began on January 15, 1999 and was completed on February 8, 1999. The response rates and refusal rates for participants in each type of program are presented in Exhibit 1. The response rate is the proportion of cases from the functional sample who responded to the survey, while the refusal rate represents the proportion of cases from the functional sample who declined to participate in the survey. The functional sample factors out the attrition in the survey, leaving only the sample which resulted in completions, refusals, and those numbers attempted but not reached by the completion of fieldwork (e.g., appointments for interviews that were not kept, retired phone numbers, respondents who were unavailable for the duration of the survey). Attrition includes numbers not in service, duplicate phone numbers, respondents in groups for which the quota had been reached and respondents who indicated no knowledge of the topic.

The overall margin of error for the survey is ±2.3 per cent. That is, the overall survey results are accurate within ±2.3 percentage points, 19 times out of 20. It should be noted that the response rate for the survey was very good, ranging from 45.5 per cent among SLG participants to 66 per cent for Job Action respondents, with an overall response rate of 53.5 per cent. The overall refusal rate was also highly satisfactory (7.1 per cent) and ranged from 4.8 per cent for Job Action participants to 10.9 per cent for EAS participants.

Comparison Group Survey

Data Acquisition

The comparison group sample was drawn from a file of EI claims that were active in 1998 (n=97,132) and a file of dormant EI claims (claims that were not being processed) from 1994 to 1998 (n=565,766), both of which were received from HRDC, Fredericton. These files were merged into a single file of EI claims (n=662,898), representing a total of 250,001 individuals. Entries for EI claimants who had participated in LMDA programs and services (n=17,570) or who had not had active EI claims since April 1, 1997 (n=130,204) were removed, leaving a comparison group population of 102,227 claimants from which to draw the comparison group sample.

Data Sampling

It was decided that the comparison group data file would be matched to the participant data file based on the time periods for which they were receiving EI. To accomplish this, three time periods were defined according to observed values for program end dates in the population of program participants. That is, within the population of program participants, the end dates of participants' most recent interventions ranged from April 1, 1997 to October 31, 1998 and this time period was divided into two six month and one seven month time period. For comparison group sampling purposes, the following time periods were derived: April 1, 1997 to September 31; October 1, 1997 to April 30, 1998; and May 1, 1998 to October 31, 1998. Reference data flags were then computed using the mid-point in each of these time periods (June 1 1997, December 1, 1997, and July 1, 1998) so that if an individual in the comparison group was EI eligible at the reference date (at the mid-point of the time period), that individual would fall into the time period cohort. This meant, however, that these were not necessarily mutually exclusive cohorts because an individual could have been EI eligible at more than one of the reference dates.

Based on the participant population characteristics, for each time period cohort, a listing was produced of the time in weeks between the end of the latest intervention and the start date of the most recent EI eligibility period. These time frames were further broken down into five categories based on the amount of time into the EI eligibility period that the participant's intervention came to an end. These categories were 13 weeks or less, 14 to 26 weeks, 27 to 39 weeks, 40 to 52 weeks, and 53 weeks or more. Each comparison group cohort was then similarly broken down into the same five categories based on the time in weeks between the reference date (the mid-point of one of the three time period cohorts) and the start date of the most recent EI eligibility period. (see Exhibit 2).

The comparison sample was drawn in the same proportions as were observed for each of the three time period cohorts in the participant population. The comparison group sample was further stratified by the number of weeks between the end date of the latest intervention and the start date of the most recent EI eligibility period.

To correct for the fact that the comparison time cohorts are not mutually exclusive, each time period cohort was sampled separately and a flag was computed to identify sampled cases. As such, it was possible to track these cases and not include them when sampling from subsequent time periods. Thus the final comparison group sample consisted of 2,400 cases in three mutually exclusive time period cohorts from a population 102,227.

Response Rate

Fieldwork for the survey began on February 9, 1999 and ended February 13, 1999. The response rate for the survey is presented in Exhibit 3 using the same response categories as described earlier. The overall margin of error is ±3.5 per cent. That is, the overall survey results are accurate within ±3.5 percentage points, 19 times out of 20. The response rate for the survey was 48.4 per cent and the refusal rate was 8.8 per cent.

Survey of Employers/Unions

Data Acquisition

A single data file was received from DOL which contained the contact information for 2,074 employers who had participated in the Partners, Job Action and/or Rural Experience programs. This data file was cleaned by removing all duplicate cases, that is, cases where the same combination of employer name and contract type (i.e., the program participated in) appeared more than once in the data file. This resulted in a file with 1,867 cases that were unique in so far as no two cases had the same employer name and contract type, however those employers who had participated in more than one program had more than one entry in the data file.

Data Sampling

The sample was drawn to ensure that three times as many employers were included in the sample as were needed to complete the survey. For the Job Action program, however, only 197 cases were available in the original data file, thus all Job Action cases were included in the sample. The sampling procedure also ensured that employers who participated in more than one program were only included in the sample for one type of program. In order to obtain 275 completions, a total of 772 cases were sampled.

Response Rate

The employer survey went into the field on February 12, 1999 and was completed on February 24, 1999. The response rate for the survey is presented in Exhibit 4 using the same response categories as used for the participant and comparison group surveys.

The overall margin of error is +/— 4.4 per cent. That is, the overall survey results are accurate within +/— 4.4 percentage points, 19 times out of 20. The refusal rate for all employer groups was very low and ranged from 3.9 per cent for Rural Experience employers to 6.8 per cent for Partners program employers, with an overall refusal rate of 5.3 per cent. The response rate for all employer groups was equally satisfactory, ranging from 41.6 per cent among Partners program employers to 54.7 per cent for Rural Experience employers. The overall response rate was 47.9 per cent.

The following sections describe usage patterns of individuals who participated in each intervention. Each section looks at participants in a particular intervention and presents the degree to which they participated in more than one intervention, with references to sociodemographic characteristics presented in Exhibit 3.4 (in the main body of the document). So, for example, in Exhibit 1, which shows participation levels for Partners, we see that the total number of Partners interventions (2,234) coincides with the total number of Partners interventions as presented in Exhibit 3.5 (in the main body of the document). Of the 2,234 interventions, Partners was the most recent intervention for 2,112 participants. Exhibit 1, however, also shows that those people who availed themselves of Partners benefits at some point in time also participated in other programs (e.g., 142 people who participated in Partners also participated in SLG, 39 people who participated in Partners also participated in Job Action), accounting for a further 234 interventions in non-Partners programs (totalling 2,468 interventions).

Analysis of the extent to which individuals participated in multiple interventions across all benefits and measures finds that multiple interventions tend to be within the same benefit or measure, indicating that individuals tend to engage in the same type of intervention multiple times. As a result, participation in a given program as a third most recent intervention, for example, does not necessarily mean that the first and second most recent interventions were in a different program.

Partners

As shown in Exhibit 1, although they tended not to participate in other benefits and measures, the other program of interest for Partners participants was Skills Loans and Grants.

Entrepreneur

Entrepreneur participants generally did not participate in other interventions, however, there was some use of Skills Loans and Grants (Exhibit 2).

Job Action

Some Job Action participants have also participated in Rural Experience and Skills Loans and Grants programs (Exhibit 3).

Skills Loans and Grants

Participation in SLG was generally homogeneous, as few of these numerous individuals participated in other interventions (Exhibit 4).

Employment Assistance Services

Participants in Employment Assistance Services tended not to use other LMDA benefits and measures (Exhibit 5).

Rural Experience

Rural Experience participants also participated somewhat in SLG, and to a lesser extent, Job Action (Exhibit 6).

Partners

Partners program employers were asked if they had participated in any other provincial wage subsidy programs since April 1, 1997, excluding their most recent experience with such a program. As shown in Exhibit 1, 67 per cent of Partners program employers had not participated in any other programs in this time period. Twenty-nine per cent of respondents had participated in the Partners program on a previous occasion, four per cent had used the Rural Experience program and two per cent had used Job Action previously.

On average, Partners program businesses employed 15.2 non-program staff (Exhibit 2). These businesses were most likely to hire non-program staff on a full-time permanent basis (85 per cent), followed by part-time permanent staff (39 per cent), and non-permanent casual or seasonal (12 per cent). The mean number of staff that businesses employed in each of these categories was also calculated. The mean number of employees on staff is highest for non-permanent casual or seasonal positions (mean=15.6) and permanent full-time positions (mean=12).

Over the last year, Partners employers were more likely to have hired employees than to have laid them off (69 versus 40 per cent), and the mean number of employees hired was higher than the mean number laid-off (mean=9.6 versus mean=5, respectively) (Exhibit 3). These employers tended to feel that this positive growth trend is likely to continue. Partners employers were also more likely to predict that they would hire employees over the next year than to predict they would lay employees off (60 versus 15 per cent). It is interesting to note, however, that employers who predicted they would hire estimated that they would hire a smaller mean number of employees than would be laid- off by those employers who predicted lay-offs (mean=4.7 versus mean=6.8 respectively).

Job Action

Other than their most recent experience with the program, 34 per cent of Job Action program employers had not participated in other provincial wage subsidy programs since April 1, 1997 (Exhibit 4). Forty-seven per cent of respondents had participated in the Job Action program, 35 per cent had used the Rural Experience program and 18 per cent had used the Partners program since April 1997.

On average, Job Action program businesses employed 48.9 non-program staff compared to 37.7 overall. Employers were most likely to report hiring non-program staff on a full-time permanent basis (70 per cent). They were also moderately likely to report hiring part-time permanent staff (45 per cent), and somewhat less likely to hire non-permanent casual or seasonal staff (23 per cent) and staff under contract (six per cent) (Exhibit 5). The average number of employees in each of these job categories was also calculated. The mean number of employees on staff is highest for full-time permanent positions (mean=50.3) and non-permanent casual or seasonal positions (mean=24.5).

Over the last year, 65 per cent of Job Action employers indicated they had hired employees, compared to only 29 per cent who indicated they had laid-off employees. Furthermore, those who hired over the last year hired, on average, more employees than were laid-off by firms who downsized (7.1 versus 3.1 respectively) (Exhibit 6). As was observed in relation to Partners program employers, Job Action employers feel this growth will continue. Job Action employers were more likely to report they would hire employees over the next year than to report they would be laying off employees (58 versus 14 per cent respectively). The small proportion of Job Action employers who reported they will lay-off employees, however, estimate they will lay-off a higher average number of employees than will be hired by firms over the next year (6.7 versus 5.8 employees).

Rural Experience

Not including their most recent wage subsidy experience, roughly half (48 per cent) of the Rural Experience employers who responded to the employer survey indicated that they had participated in a provincial wage subsidy program since April 1, 1997. These respondents were most likely to have participated in the Rural Experience program in this time (29 per cent), followed by the Partners program (22 per cent) and the Job Action program (18 per cent) (Exhibit 7).

On average, Rural Experience program businesses employed 53.2 non-program staff compared to 37.7 employees overall (Exhibit 8). Employers were most likely to report hiring non-program staff on a full-time permanent basis (68 per cent) and were moderately likely to report hiring part-time permanent staff (34 per cent). They were least likely to report that they employed non-permanent casual or seasonal staff (12 per cent), and contract staff (one per cent). Although the proportion of Rural Experience employers indicating that they employ full-time permanent staff is lower than the proportion overall who staff these positions (68 versus 75 per cent respectively), the mean number of employees that Rural Experience employers have in full-time permanent positions is much higher than the overall mean (mean=67.1 versus mean=38.3 overall).

Sixty-two per cent of Rural Experience employers reported hiring over the last year, compared to only 36 per cent who indicated laying employees off. Further, more employees were hired than were laid off by Rural Experience employers over the last year (mean=38.1 versus mean=20 respectively) (Exhibit 9). The finding that a higher proportion of Rural Experience employers predicted they would hire employees than predicted they would lay-off employees over the next year (55 versus 30 per cent respectively), suggests that the positive growth of Rural Experience businesses will continue. The mean number of employees that employers estimate they will hire, however, is only slightly higher than the mean number they predict they will lay-off (mean=26.9 versus mean=23.5 respectively).

In the appendix, we present additional results on the characteristics of the post-intervention job not discussed in the text. Exhibit D-1 displays the complete results which this discussion and the discussion in the body of the text refers to.

With respect to returning to a former employer; some interesting differences occurred across age, sex and education groups (not shown). Men were more likely to return to their former employers than women were and older clients (45 years and older) were more likely than those in other age groups. Finally, the tendency to return to a former employer declines by education level.

Little variation in weekly hours worked is observed across the different interventions, apart from the high proportions working part-time hours (less than 30) among Job Action and EAS participants (23 and 22 per cent), and the high proportion (64 per cent) of Entrepreneur participants working "long" hours (more than 40 hours per week) (panel 2 of Exhibit D-1). The latter is reflected in the high mean and median hours for the Entrepreneurs participants. EI claimants tended to work longer hours than reachbacks and the comparison group, though there is no difference in median hours between those two groups. Not shown is the fact that male clients worked more hours than female clients did which reflects mainly behaviour observed in the overall workforce; no differences were apparent by age and education. Finally, comparison group members tended to work somewhat fewer hours than EI claimants.

The incidence of year-round employment was, interestingly, higher for reachbacks compared to EI claimants (58 versus 50 per cent), while comparison group members had an even lower share (46 per cent) than EI claimants did (panel 3 of Exhibit D-1). The comparison group's share of casual/contract employment was lower than it was for EI claimants (29 versus 37 per cent), but its seasonal employment share was considerably higher (25 versus 13 per cent). Finally, once again reflecting general labour market patterns, the incidence of year-round employment rises with age and education level and is higher for men than women (not shown).

Not surprisingly, Entrepreneur participants were more likely to indicate that that they were currently or most recently self-employed (92 per cent) compared to less than five per cent for the other benefits and measures (panel 4 of Exhibit D-1). The latter rises with education level but does not vary much by sex or age (not shown). EI claimants were less likely to be self-employed than reachbacks (11 per cent). Participation in self-employment is higher for the comparison group than EI claimants, despite the presence of Entrepreneur participants in the latter group.

View EXHIBIT D-1

EXHIBIT E-1a - Weighted Means and Frequencies for Dependent Variables Used in the Models*

	Neumonic	Per Cent in Category or Mean
Currently employed (proportion)	dumemp1	46.9%
Currently full-time employed (proportion)	dumemp3	31.6%
Employed 12 consecutive weeks following intervention (proportion)	dumcons	66.4%
Percentage of weeks employed following intervention (mean)	weekwork	61.9%
Percentage of weeks looking for work while jobless following intervention (mean)	ratcl4	23.8%
Current weekly earnings (mean)	ci14d	$469.57
Absolute change in weekly earnings from one year prior to intervention (mean)	diffearn	$-89.53
Percentage change in weekly earnings from one year prior to intervention (mean)	diffearn2	234.1%
Percentage of weeks receiving EI in a new spell following intervention (mean)**	eipostp	6.2%
Received SA benefits following intervention (proportion)	dumsar	4.7%
* Based on survey results (unless otherwise indicated) for survey respondents from among EI claimants, "near" reachbacks (EI claim ended up to six months prior to the intervention), and the comparison group. ** Base on administrative data.

EXHIBIT E-1b - Weighted Means and Frequencies for Explanatory Variables Entered into the Models*

	Neumonic	Per Cent in Category or Mean
Intervention type (vs. non-participant in intervention)
Partners	Dumprg32	12.8
Entrepreneur	Dumprg12	6.8
Job Action	Dumprg42	7.9
SLG	Dumprg52	15.3
EAS	Dumprg22	4.1
Rural Experience	Dumprg62	12.4
Weeks since intervention ended	Weekbetw	54.5 weeks
Employment status one month before intervention (vs. not in labour force)
Employed	Dumpreun	45.3
Unemployed	Dumpreem	39.0
Employed one year before intervention (vs. not)	Duma22	80.1
Education level (vs. less than high school)
High-school certificate	Dumeduc1	34.4
At least some post-secondary	Dumeduc2	36.4
Age group (vs. <35 years)
35-44 years	Dumage1	54.4
45-54 years	Dumage2	25.0
55 years and over	Dumage3	14.4
Male (vs. female)	Dumsex	54.7
Mother tongue (vs. English)
French	Dumfre	40.3
Other	Dumoth	0.7
Married (vs. non-married)	Dummart	66.2
Minority (vs. not)	Dumminor	3.6
No dependents (vs. dependents)	Dumdep	45.5
Pre-intervention interest in:
Being trained (mean 1-7)	E5a	4.8
Starting own business (mean 1-7)	E5b	3.3
Entering labour force (mean 1-7)	E5c	6.5
Number of separations, 1992-1997 (vs. 2 or less)
3 to 5 separations	Roe35	40.4
6 or more separations	Roe6pl	27.8
No. of weeks eligible for EI overlapping with intervention (vs. not eligible)
1 to 36 weeks	Deielg1	30.9
36 to 52 weeks	Deielg37	35.7
No. of weeks received EI since 1992 prior to intervention (vs. 0-24 weeks)
25-52 weeks	Dewpr25	26.3
53-104 weeks	Dewpr53	27.5
105 weeks and more	Dewpr105	28.1
Earnings in year prior to intervention (vs.< $5,000)
$5,000 — 9,999	Dearn5k	25.9
$10,000 — 19,999	Dearn10k	28.8
$20,000 — 29,999	Dearn20k	11.5
$30,000 and over	Dearn30k	8.3
Received SA in year prior to intervention (vs. not)	Rsear1y	8.1
Use of other services
Used self-serve services (vs. not)	Dumself	49.5
Met a counsellor (vs. not)	Dumcouns	19.0
Set up an action plan (vs. not)	Dumact	8.3
Used other services (vs. not)	Dumother	7.0
n		1789
* Based on survey results for survey respondents from among EI claimants, "near" reachbacks (EI claim ended up to six months prior to the intervention), and the comparison group.

View EXHIBIT E-2a

View EXHIBIT E-2b

View EXHIBIT E-2c

View EXHIBIT E-3

View EXHIBIT E-4

View EXHIBIT E-5a

View EXHIBIT E-5b

View EXHIBIT E-5c

View EXHIBIT E-6a

View EXHIBIT E-6b

Footnotes

1	The evaluation survey results indicated that 70.5 per cent of survey respondents whose intervention ended in the 1997/98 fiscal year returned to work for at least 12 consecutive weeks following their last intervention. Applying this percentage to the population of participants in the intervention under study for the 1997/98 fiscal year (8,801) yields 6,261 participants. This figure, which is based only on the six benefits and measures evaluated in this study and excludes services such as individual counselling interviews and group services, still exceeds the Monitoring Report figure which is based on all Cda/NB LMDA benefits and measures.
2	Section 1.2 of the Cda/NB LMDA.
3	The term "reachback" refers to EI clients who fall under the definition of clients provided in Section 1.2 of the Cda/NB LMDA, i.e., persons who falls within categories (b) and (c) of EI clients as described on the previous page.
4	For fiscal 1999-2000, the 60 per cent rate has been increased to 70 per cent for the hiring of social assistance recipients, Aboriginals and persons with disabilities in addition to post-secondary graduates.
5	The Joint Evaluation Committee is composed of representatives of Human Resources Development Canada (HRDC), the New Brunswick Department of Labour (formerly the New Brunswick Department of Advanced Education and Labour [AE&L]) and Human Resources Development New Brunswick (HRDNB).
6	The NBLFDB representatives were interviewed in Phase II.
7	The systems and infrastructure key informants were interviewed in Phase I.
8	Source: 1998 Employment Insurance Monitoring and Assessment Report.
9	Note that the means shown are for those who had the respective type of income in the year, not for all participants.
10	Note that no reachback clients are included in the comparison group survey.
11	Although the CEIU did not participate in the negotiation of the Agreement, the union was involved in the LMDA implementation, which is the focus of this evaluation.
12	Eighty per cent of those who felt a loan or grant was most important were SLG participants
13	Please note that the data is weighted, thus the raw numbers do not always match the reported percentages perfectly. The raw numbers are presented in lieu of reported percentages because of the low n.
14	Multiple responses were allowed for this question, thus the total number of responses reported here sums to more than 100 per cent of those who answered the question.
15	The participant survey sample included only those individuals who were involved in an LMDA employment program between April 1, 1997 and October 31, 1998. As such, the first employment program refers to the first employment program the participant was involved in during that time.
16	This results target applies only to Provincial Benefits and not to Provincial Measures (Rural Experience and EAS).
17	Submitted to Human Resources Development Canada by the Canada Employment Insurance Commission, December 18, 1998, covering the period up to March 31, 1998.
18	Information to update result target attainment is based on the survey which took place in early 1999.
19	The 1998 EI Monitoring and Assessment Report notes that in the 1997/98 fiscal year there were in excess of 3,300 interventions where individuals made use of these services only. This evaluation is focused on six benefits and measures excluding those interventions, i.e., Partners, Entrepreneur, Job Action, SLG, EAS and Rural Experience.
20	Among those who completed their intervention on or before March 31, 1998, only 43 per cent of this group who said in the survey they had been employed for 12 consecutive weeks in the post-intervention period had a comparative 1997/98 employment outcome flag on the administrative data file. On the other hand, 78 per cent of those who had an employment outcome flag said in the survey they had been employed for 12 consecutive weeks following their intervention
21	We use administrative here because the corresponding question was not asked in the evaluation survey.
22	A claimant is considered to have returned to work if, for 12 consecutive weeks, they receive 25 per cent or less of their maximum EI entitlement within their benefit period. There are of course no paid EI benefits for reachbacks who were not receiving EI benefits at the time of the intervention.
23	This is the approach used by Applied Research Consultants to compute unpaid EI benefits for the Canada/British Columbia LMDA.
24	Technically, we still have not attributed the return to work to a specific intervention. Though we are dealing with participants in interventions, we have not taken into consideration the date of the intervention. For RTWs occurring before the intervention, there are no savings to be derived. Indeed, there were a few such individuals on the file. Subtracting the savings of this group from the computed sum reduces the total saved for the 1997/98 and 1998/99 fiscal years by $543,879 and $58,361, respectively.
25	The 1998 Formative Evaluation of the Employment Benefits and Support Measures - Final Report, for the EBSMs for Canada as a whole, conducted on behalf of HRDC, compared this measure of employment by program type and claimant status in this way.
26	Employers in the employer survey were also asked to indicate the percentage retained of the participants they had in the wage subsidy programs. However, inconsistent reporting by employers led to the conclusion that the percentage retained measure based on the employer survey results would be unreliable. At any rate, it is safe to say that participants individually would be in a better position to recall being retained than employers who would have to recall several different wage-subsidy events.
27	An error in survey coding resulted in a number of Job Action and Rural Experience participants being initially skipped over this question. But there is no reason to believe that those who did not answer this question are any different from those who did, implying that the percentage re-hired for these programs would still be valid. Also, those currently in the program should have been eliminated from this calculation, but as they represent such a small proportion of participants, their inclusion has likely not greatly affected the results.
28	It is interesting to point out that percentage retained by Partners employers computed on the basis of the employer survey results was 59 per cent, which is similar to this rate.
29	The survey simply asked if the participant was on EI following the intervention. The responses to such a question would have been biased against short-term interventions, after which there is more remaining EI entitlement than longer-term interventions.
30	It should also be noted that the participant survey results indicated that 45 per cent of SLG participants received on-the-job training from the employer whom they were placed with while participating in training through the SLG benefit.
31	Note that personal income was also considered as a dependent variable to be modelled. However, though the survey question referred to a one-year period, the period since the intervention and therefore the portion considered to be the "outcome", would vary for participants. Moreover, total personal income includes types of income that would not necessarily be linked to participation in the intervention.
32	In the univariate results presented in the previous chapter, we showed weeks not working. To be consistent with the other employment measures, which were all positive outcomes, we "reversed" this variable, so that the signs on the explanatory variables may be expected to be in the same direction as for these other outcome measures. In addition, because weeks since the intervention is controlled for in the model, it was deemed unnecessary to model the absolute number of weeks working as well.
33	Omitted from the analysis because of missing data is a regional control variable. It is quite likely that employment rates would be affected by local labour market conditions. This question will be pursued in the summative evaluation.
34	It is quite likely that the negative result for Francophones may have something to do with the fact Francophones are concentrated in a region of the province where unemployment is high and the work is seasonal. Indeed, differences in employment rates themselves are likely to some extent explained by differences in local labour markets. However, as noted above, for this formative evaluation, there was lack of data available to gauge this influence.
35	This may be a function of the fact that Francophones are concentrated in a high unemployment region. See previous footnote.