Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

• Initial results indicate employment impact in both gross and incremental terms. The type of employment measure (full-time or three consecutive months) and client characteristics need to be taken into account to interpret either gross or incremental results. Incremental results pertain to Employment Insurance (EI) claimants only, not reachback clients.
• All Employment Benefits and Support Measures (EBSMs) but Employment Assistance Services (EAS) have a positive impact on incidence of incremental employment for the three types of employment (current, full-time and three consecutive months of employment); EAS has a positive effect on just the last two employment measures. Impact varies across client groups, specifically, EBSMs have less impact for older clients and those with less recent employment attachment. Evaluation's incrementality analysis and client feedback points to lower impact of short-term group sessions compared to other initiatives.
• Evaluation confirmed employment status reported by Human Resources Development Canada (HRDC's) accountability found work performance indicator. Absence of operational data on client and employment characteristics limits the interpretation and utility of these indicators and comparability across areas.
• The positive impact of EBSM on client employment did not translate into an incremental reduction in EI use of a similar magnitude. There was, however, a slight reduction in claim duration (3 percent for both Self-Employment Assistance (SEA) and Training). More time needs to lapse post-intervention before reduction in Employment Insurance (EI) and Social Assistance (SA) use can be measured.

This chapter presents initial evidence of the impact of EBSM on client-oriented outcomes. Outcomes are measured by changes in labour market activity (employment and unemployment), earnings, post use of EI/Social Assistance and client attitudes. Some of these outcomes are compared to the experiences of non-participants to gauge early indications of net change due to program participation and test methodologies for use in the summative evaluation stage. Finally, given the importance of HRDC's primary employment performance indicator, this indicator was compared to evaluation findings to assess comparability and similarity of results. It should be pointed out that these incremental outcomes require sufficient post-intervention experience to fully emerge. For this evaluation, this lapsed time ranged between 6 to 18 months. This time period is not sufficient to capture the full incremental impact of all outcome measures presented in this report. These will be more fully explored in the summative evaluation.

5.1 Labour Market and Related Outcomes

The intent of EBSM interventions is to improve client's employment. The evaluation's findings for this key outcome and other associated labour market and earnings impacts are presented below.

a) Labour Market

The post intervention labour market activity of EBSM clients points to positive impacts from participation. For the key employment outcome, the evaluation survey indicated that 85 percent of EBSM participants found a job sometime within the 18 months after completing their intervention, with about half finding a job within one week of completion. The duration of unemployment experienced by EBSM participants averaged 13 weeks during the year following completion. Both of these percentages did not vary between EI or Reachback clients.

In terms of job characteristics, the majority of clients (84 percent) were working full-time in the current job (at the time of the evaluation survey) and two-thirds were in year round jobs. The vast majority had worked with only one employer and one in ten returned to their former employer.

Across programs, SEA participants registered the highest proportion that obtained post-intervention employment results, while EAS participants had the lowest. Targeted Wage Subsidy (TWS) participants were most likely to be working full time in the post-program period.

Post intervention employment varied across client types. Older clients and those self-identifying as being in a visible minority had a relatively lower share in employment or obtaining full-time employment post intervention.

b) Earnings

Results indicated that EBSM participants earned about $530 weekly in their current job (held at the time of the evaluation survey). SEA participants reported the highest earnings ($645) and TWS the lowest ($466). On average, there was little change in weekly earnings compared to those earnings in the longest job held in the year prior to the intervention. The exception was clients on SEA who experienced an increase in average earnings compared to their previous job.

c) Use of EI and Social Assistance

One of the goals of labour market interventions is to reduce participants' dependence on public income supports such as employment insurance and social assistance (SA). Note that a negative result is really the sought-after outcome, though when assessing shorter-term impacts, reliance on income support, particularly EI, is often difficult to interpret, indicating on the one hand a return to dependence (negative result) and, on the other, sufficient employment to re-qualify for benefits (positive result).

Analysis of the early results indicates that less than 10 percent of EBSM participants had collected employment insurance or social assistance in the post-program period (post program period averaged 15 months.) This measure is presented for descriptive purposes only. A greater lapse in time is necessary post-intervention before clearer measures of post EI use are possible. (See page 41 for further discussion of EI use.)

d) Behavioural and Attitudinal Impacts

Participation in EBSM initiatives has other impacts in addition to those directly reflected in economic terms. Client feedback pointed to several measurable and anecdotal effects of participation.

In the focus groups, participants described a number of impacts of participating in EBSMs. While some noted that their program had led to a job, others indicated other less tangible benefits of participation. These included, for example, establishing personal networks, development of skills, achievement of a license/certification, and building confidence and self-esteem.

Client attitudes on willingness to borrow in order to take training and confidence in skills. These results indicate that EBSM participants, particularly the reachback group are more willing to borrow money to invest in upgrading their skills (53 percent of the reachback group and 46 percent of EI claimants agreed they would be willing). This is significantly higher than the comparison group — only one in three indicated they would be willing to borrow in order to take training.

In terms of confidence in skills, EBSM participants — both EI claimants and reachback — demonstrate greater confidence in their skills than the comparison group. Whereas 78 percent of claimants and 80 percent of reachbacks agree that they "have the skills to move easily in today's labour market," 67 percent of the comparison group agreed with this statement.

5.2 Incremental Gains - Did EBSMs Make a Difference?

A key interest is whether EBSM (and other employment related) initiatives make a difference toward client's obtaining employment. Results presented above are "gross results"; that is, their occurrence or magnitude reflect in part, what would have taken place in any case without participating in an EBSM. Abilities of clients changes in the economy and labour markets are some of the factors that can contribute to achieving results independent of program participation.

Questions of service or program incrementality are usually, although not necessarily, examined in summative evaluations. Incrementality was examined in this formative evaluation to discover if there were early indications of incremental effects while at the same time exploring the methodology and adequacy of available data to undertake this measurement. The results presented therefore should be considered as preliminary, specifically for employment attained, earnings differences and subsequent use of income assistance. The summative evaluation will permit a more definitive statement on these measures.

In this evaluation the question of incrementality is considered through two means: client opinion and measured differences between the experience of participants and non-participants. It is important to note that results from the comparison (non-participant) group analysis pertain only to EI claimants, not reachback clients. Resource and information limitations did not permit the construction of a comparison group that included reachback. While it may be tempting to infer results based on similar client characteristics shared by EI and reachback clients, measurement of incremental impacts for reachback clients should wait for appropriate reachback comparison group analysis, either as supplementary analysis from this evaluation or a summative evaluation phase.

a) Client's opinion

EBSM participants who had found employment following the end of their program were asked to rate the extent to which the help they received from HRDC was important in obtaining employment.¹² Overall, 37 percent of EBSM clients rated HRDC's assistance as being very important.

Ratings varied (not surprisingly) by the factors that relate to programs with clear links between the initiative and employment outcome and also those with higher service intensity experienced by clients. The rating was highest for TWS participants who were hired by their employer after their subsidy ended (55 percent). Those rating between 37 percent and 55 percent were training participants, EAS clients rated the importance below 37 percent. Those participants with post-secondary education and those who found a job within 12 weeks of the end of their intervention provided higher ratings. While these ratings may be perceived as modest, they are higher than the parallel rating of 11 percent provided by clients who only participated in short-term employment group information sessions.

The responses by clients in EAS and clients solely attending an employment group service suggest clients perceive a less obvious link with employment and greater contribution played by other factors such as help from others, etc. This perception suggests that employment may be too far removed to be linked to some interventions and therefore is not the most appropriated indicator for all EAS interventions. A more intermediate result measure, focussing on pre-employment achievements (for example increased understanding of programs or the labour market) would seem more appropriate. The alternative measures could apply to some EAS activities, in particular to group sessions.

b) Incremental Measurement Using the Comparison Group

As noted in the report's methodology discussion, a comparison group was used to parallel the experiences of EBSM participants (EI clients only) with those of non-participants. Analysis of differences between these two groups included adjustments to control for pre-existing differences between these groups (referred to as multivariate analysis) as well as a control for local job availability the local unemployment rate, a measure of local labour market conditions traditionally used in analyses of this kind. The following presents the findings from this multivariate analysis on the effects for the key labour market activity outcome variables as well as earnings and use of EI/Social Assistance.

As noted earlier, the comparison group reflects only EI claimants. Therefore, incremental results shown below only pertain to EBSM participants who were on EI at the start of their intervention. Further, the evaluation does not address the question of whether the degree of incremental result measured is adequate. No prior targets or expectations for incremental impacts were available for comparison at the time of the evaluation. It is understood that work is underway in HRDC to explore in more detail incremental results. This work and findings from a summative evaluation will help answer this question.

Incremental Change in Labour Market Activity

As previously noted, the evaluation analysis examined the question of labour market impact from four perspectives: incidence of employment, worked full-time, worked for three consecutive months and percent of weeks working post-intervention. Exhibit 5.1 summarizes the employment effect for three of the measured employment categories (incidence of employment, worked full time and worked for three consecutive months), in terms of increased probability of being employed (compared to individuals who did not participate in an EBSM). The results suggest that, taken together, participation on an EBSM intervention increases the probability of a client attaining a particular post-intervention employment outcome by between 15 to 20 percent.

The role played by EBSM in the participant's post-employment experience varied in degree between the four EBSM categories (see the technical report for full discussion of incremental impacts for each category). Across all client types, the strongest incremental effect was in SEA and the weakest in EAS. In terms of the probability of being employed at the time of the evaluation survey, SEA had a 20 percent impact, Job Creation Partnerships (JCP) 18 percent, TWS 14 percent and Training, 8 percent. EAS did not have a statistically significant effect. On average, EBSM had less of an employment impact for older participants (over 45), members of visible minority groups and participants with family dependents.

Analysis within sub-categories — that is, selected client groups and types of employment — showed considerable variation. TWS, Job Creation Partnership (JCP) and Training had a significant and positive impact on the incidence of employment for younger participants (under 45). Training also had a positive impact for younger and female participants in terms of probability of full time employment and three months consecutive employment. JCP, in addition, had a positive effect on the latter employment indicator for females only.

Having close attachment to the labour market prior to the intervention (not surprisingly) was a factor in post intervention employment - clients with a more recent job prior to the start of the intervention had a higher probability of being employed for 3 consecutive months. This would underline the importance of EI claimants taking up EBSMs early into their claim. It also points to the need for having sufficient information on client characteristics and details on employment outcomes in order to analyze results —aggregate, cursory analysis could very well mask underlying impacts.

In terms of HRDC service or delivery characteristics that may influence incremental employment, clients were more likely to be employed for three consecutive months (at the time of the evaluation survey) in Human Resource Centres of Canada (HRCCs) that had a positive assessment of business planning capacities.¹⁴ Use of group sessions versus one-on-one counselling prior to participation in the specific employment benefit did not appear to have an effect on incremental employment. There was insufficient information to gauge whether the move toward third party service delivery had an effect on incremental employment. Not surprisingly, employment results were lower in areas with higher unemployment rates. Although it was not possible to analyse this last relationship in detail, it points to the importance of considering labour market conditions when interpreting results, setting targets and comparing employment results among HRDC offices.

In addition to the three employment measures examined, the evaluation looked at the duration of employment in the post-intervention period. Exhibit 5.2 presents the comparison group analysis results on weeks working since intervention as a percentage of weeks since the intervention. For this outcome measure, JCP participants had statistically significant positive results: an increase of 13 percent. Training and EAS had a positive impact when considering younger participants only.

Factors improving the proportion of weeks worked for both participants and the comparison group were: having a job close to the start of the intervention, length of time since intervention, being male, having greater prior earnings, and having a strong interest in entering the labour market. Older participants, visible minorities, those engaged in extended job search and from areas of higher unemployment rates had a poorer result.

Incremental Change in Earnings

Three earnings outcome measures were examined: 1. weekly income from employment or self-employment (earnings) in the current or most recent job at the time of the survey; 2. absolute change in weekly earnings from employment or self-employment from the longest job in the year prior to the intervention; and 3. percentage change in weekly earnings.

In sum, EBSM appear to have had little incremental impact on earnings. SEA is the only program that had a statistically significant impact for the three variables tested, while EAS had a negative effect on percentage change in weekly earnings. Older participants experienced a lower percentage change in earnings, while those having a job within a month prior to the intervention experienced a more positive impact.

Incremental Reduction of EI Use

The above comparison group analysis indicates that there are early indications of incremental employment (and unemployment) effects from participation in EBSM. The question is whether this has translated into measurable change in use of EI and SA.

The analysis suggests participation in the EBSMs did reduce the percentage of weeks on EI. Both SEA and participation in Training were found to exert a statistically significant 3 percent reduction in the proportion of weeks on EI in the post-intervention period. TWS participants, on the other hand, were found to have had an increase in the percentage of weeks on a post-intervention EI claim. This finding for TWS participants is likely due to their ability to accumulate insurable weeks during their work placement which was more likely than other participants to be in full-time employment (not self-employment). As for SA use, SEA again appeared to increase the likelihood of using SA after the intervention, while other EBSMs had no statistically significant impact.

These results are intended to provide early indications of EBSMs impact on EI and SA. They do not measure the total effect of EBSMs on EI or SA. The results are influenced by the relatively short time (average of 15 months) that elapsed between program completion and the telephone follow-up survey with clients. The follow-up period was too brief to span an employment/unemployment cycle (that is, find employment, lose employment and file for EI). Subsequent analysis after two or more years, therefore, is required, either in a summative evaluation or supplementary analysis, to this effect. It is our understanding that at the time of this writing, an HRDC project is underway to determine the methodology for and produce estimates of the impacts of EBSM and LMDA intervention on EI and SA.

5.3 Assessment of Found-Work Accountability Measure

One of the accountability result measures for EBSM is obtaining employment after the intervention. The evaluation sought to assess the estimated employment as measured by HRDC's accountability system. The evaluation did this assessment by comparing the differences between the employment result reported by the accountability system and similar information collected during the evaluation process. The evaluation did not examine the detail process of deriving the accountability employment estimates nor did it look at the second accountability indicator — calculation of the unpaid benefit. A detailed review of the accountability process of both these indicators can be found in the evaluation of the British Columbia Labour Market Development Agreement.

HRDC compiles the employment statistic using a number of sources. EBSM participants are considered to be employed when:

• a claimant receives 25 percent or less of their maximum EI entitlement within their benefit period for 12 consecutive weeks; or
• follow-up with participants indicates that they are employed; or
• for TWS, the participant continues to work after the end of their intervention.

Exhibit 5.3 presents a comparison of HRDC administrative data and survey results for the 1,443 EBSM participants included in the formative evaluation. The indicator of three consecutive months employment is the most conservative employment measure available from the survey and the one that is closest to the definition of employment used for HRDC administrative data.

The second set of figures from Human Resources Investment Branch (HRIB) administrative data shows the proportion of survey respondents with an employment indicator on the 1997/8 or 1998/9 files (to December 1998).

The table indicates that, compared to the survey results, the HRIB file consistently underestimates the proportion of participants employed — even using the most conservative indicator of employment on the survey (employment for three consecutive months). The extent of the difference is about 20 percentage points for EBSM participants overall. Employment indicators are most consistent between the HRIB file and the survey data for TWS participants (a difference of only four percentage points) and least consistent for SEA, JCP and short-term group session participants (a difference of over 30 percentage points). Not surprisingly, the estimation of employment is closer to the survey results for EI claimants than for the reachback group, given that employment for EI claimants is tracked both through follow-up and through amount of claim.

The divergence in the employment results between the two sources is likely the combined result of differences in the definition of "employed" used in the administrative and survey data and in the follow-up of participants, as well as vagaries in the capture of the data at the HRCC office level.

Footnotes

12	Rating is based on a seven-point scale. The findings reported are for the top two ratings — "very" and "extremely important".
13	Probabilities for the "All EBSMs" category were calculated using a separate regression equation than the individual EBSM component interventions. Therefore, the probabilities for all EBSMs may fall outside the range of probabilities reported for the five component interventions.
14	Assessment through review of published business plans.