Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

An essential question in the analysis of the impacts of PBMs is the incremental impact of these services. Simple comparisons between program participants and non-participants on key outcome indicators (e.g., employment status) may yield a biased estimate of program impact because of pre-existing differences between the comparison group and the PBM participants. In order to ensure that differences in measured outcomes were not the effect of pre-existing differences between the PBM participant group and the comparison group, and also among participants in different PBMs in terms of their labour market experience or background characteristics, multivariate analyses were conducted. Note that these analyses were based on unweighted data because of the inclusion of control variables, which figured in the computation of the weights. Note as well (and again) that this formative evaluation may bias results in favour of interventions such as Entrepreneur and Partners where the outcomes are immediate and against interventions such as EAS, Job Action, SLG and Rural Experience where the assistance is such that there are no immediate employment outcomes or the employment effects would be expected to be of a more long term nature.

8.1 Description of Approach

Ten dependent variables representing key employment, earnings,³¹ and income support use outcomes were tested in the models, corresponding to the key objectives of the LMDA which are sustained employment and reduction in dependency on income support. These variables are as follows:

• employed/self-employed (or not) at time of survey;
• full-time employed at time of the survey;
• worked 12 consecutive weeks since end of intervention/reference date;
• weeks working as percentage of weeks since intervention/reference date;³²
• weeks looking for work as a percentage of weeks since end of intervention/ reference date;
• weekly earnings of current or most recent job (at the time of the survey);
• absolute change in weekly earnings (compared to one year prior to intervention/reference date);
• per cent change in weekly earnings (compared to one year prior to intervention/reference date);
• a new spell of weeks on EI as a percentage of weeks since end of intervention reference date; and
• ever received Social Assistance since intervention/reference date.

The means or frequencies of the dependent variables are presented in Appendix E-1a.

Along with the intervention "dummy" variables, a common set of explanatory (control) variables was introduced into the models for each dependent variable. The purpose was to assess (or control for) the influence of other factors on the intervention's impact on the outcomes. These other factors included the time since the intervention and antecedent sociodemographic and employment-history variables. The means or frequencies of these variables are contained in Appendix E-1b. Noting that "intervention" here refers to the end of the intervention for participants in LMDA PBMs and to the reference date for comparison group members, the variables entered into the models follow:

• intervention status: one variable to indicate individuals' participation in one or more of six PBMs, or non-participation in any of the interventions (comparison group);
• length of time since the intervention (since this varied considerably);
• sociodemographic variables:³³ age, sex, education, mother tongue, minority status, marital status, and existence of dependants;
• prior labour force experience: employment status (employed, unemployed) in month before intervention (versus not in the labour force), whether employed or not one year before intervention/reference date (entered in stepwise fashion because of concerns with collinearity with the previous variable), interest in entering training/self-employment/ labour force prior to intervention, number of separations 1992-1997, weeks of EI eligibility overlapping with intervention, weeks EI benefits received 1992— 1997, and total gross earnings in the year prior to intervention; and
• service-delivery variables: whether individuals had used self-serve products, received counselling, participated in job-search activities or developed an action plan, or services other than from HRD-NB or a HRCC/HRSC.

Logistic (logit) regression was used for categorical dependent variables and Ordinary Least Squares (OLS) regression was used for continuous dependent variables. The modelling was conducted in a series of four stages. First, a flag representing participation in a PBM (compared to non-participation in that PBM) was entered in the model alone, i.e., without the controls. The coefficient for this variable measures the effect on the outcomes of the intervention not controlling for the influence of other factors. Second, controls for time since intervention and sociodemographic and work-history background variables were introduced into the model. Results from the bivariate analysis presented in the previous chapter had indicated that the profile of participants in the different interventions differed appreciably. The coefficients on the intervention variables would now tell us whether or not the previously found impact of the intervention had more to do with the nature of the participants in the program than with the program itself. In the third stage, variables related to related service delivery (e.g., action plans) were introduced.

Finally, in the fourth stage, controls computed to reduce self-selection bias (the so-called Heckman correction or the Inverse Mill's Ratio) were introduced into the model.

It should be noted that, originally, the econometric analysis was to be conducted for just active EI claimants because of the inability to draw a comparison group for reachbacks, as discussed in the previous chapter. However, because of concerns with the relatively small number of active EI claimants due to the larger than expected number of reachback participants in the survey dataset (and in fact in the population of participants), a group of what we called "near" reachbacks was included as EI claimants. These individuals had received EI six months or less prior to the intervention. Grouping them with EI claimants was justified because of their recent experience with EI and the labour market and their similarity with regard to other characteristics. In other words, the comparison group could serve as a comparison group for this group of participants as well.

8.2 Results

In the results that follow, we present in the tables four columns of coefficients (and their significance level) corresponding to each variable-entry stage as described above. This is done to observe how coefficients on the program variables change as control variables are cumulatively entered into the model. We also identify what role, if any, was played by the controls. The complete set of results for the final stage of the modelling exercise, i.e., including all control variables and the Heckman Correction factor, for each dependent variable is presented in Appendix E. In observing whether a variable exerts a positive or negative impact on the dependent outcome variables, we mention only variables exerting a statistically significant impact at the five per cent or lower level.

As there appeared to be differences in the results by gender and sex, segmented analysis by these variables was conducted. Interesting differences in the results between segments are commented on in the text below, based on the full results which are presented in Appendix E. Segmented analysis was also conducted by claimant status (active EI claimant participant versus near-reachback participants). As mentioned previously, the latter were grouped with active EI claimants for purposes of this analysis. As well, the results of the latter analysis are commented on in the body of this chapter and presented in Appendix E.

The results are presented below for three sets of outcome measures: employment/job search, earnings, and income support use. Each is discussed in turn.

Employment and Job Search Outcomes

Logistic regressions were run for three different binary employment measures as the dependent variable: currently employed (at the time of the survey), currently employed full-time, and employed for at least 12 consecutive weeks since the intervention. Ordinary Least Squares regression was run with the continuous variable percentage of weeks working since the intervention. We also present, in this section, the results of modelling percentage of weeks looking for work since the intervention.

In Exhibits 8.1a, b, and c, we present the coefficients for just the program (benefit and measure) dummy variables, for the first three employment measures. As previously mentioned, for each employment measure as well as for all the other outcome measures, we present four columns of coefficients corresponding to four different regression runs:

• the run with only the program variables in the model,
• the run with program variables plus time since intervention and sociodemographic and prior history background variables in the model,
• the run with the full set of variables including the service-delivery variables in the model, and
• the full model including the Heckman Correction (Inverse Mill's Ratio).

The results for currently employed (Exhibit 8.1a) indicate that, with only program variables in the model (column 1), four interventions make a significant impact on the likelihood of being currently employed. Compared to non-participants in the respective program, Entrepreneur, EAS and Partners significantly increase the chances of being employed (after the intervention at the time of the survey), while Rural Experience reduces the chances. After inclusion of the variables capturing weeks since the intervention and the sociodemographic and work history traits of the participant (but not the service delivery variables), column two of Exhibit 8.1a indicates that, the three interventions exerting a positive impact on the outcome variable still exert a positive impact, while Rural Experience no longer has a negative impact, implying that the participants appear to compensate somewhat for the negative effect of the intervention. With the addition of the service-delivery variables, we observe in the third column that EAS no longer has a positive impact, implying that these variables nullify EAS's positive influence. Finally, in the fourth column, we observe that correcting for self-selection bias has returned the EAS to its original positive impact on the chances of being employed, along with Partners and Entrepreneur and EAS.

Full-model results presented in column one of Appendix E-2a reveal that a number of the control variables have an influence on the likelihood of employment. Exerting a positive impact are: being unemployed in the month prior to the intervention (compared to not in the labour force), having a high-school certificate and post-secondary education (compared to less than a high-school certificate), an interest in entering the labour force, being eligible for EI for 37 weeks or more (compared to not being eligible) prior to or overlapping with the intervention. The variables exerting a negative impact are: being male, being in a minority group, having no dependants, and using self-serve employment services.

Because gender appears to affect the chances of employment, we ran segmented models for males and females separately. The results presented in Appendix E-2a indicate that Entrepreneur had a positive effect for both men and women, but Partners had a positive impact for only males. EAS, which was found to have a positive effect for the full population, had no effect for either the male and female segments. As for the controls, it appears that women's employment chances were affected by outside influences more than men's were. Women but not men were positively affected by having an interest in entering the labour and negatively affected by being employed one year before the intervention, being in a minority group, and having no dependants. Still, men but not women were positively affected by being unemployed one month prior to program entry and having some post-secondary education, and negatively affected by use of self-serve services.

By age, the results in Appendix E-2a indicate a few differences between younger (45 years and under) and older (over 45 years of age) participants. There was no difference in terms of the impact of the programs, but the younger group was more affected by outside influences. Only the younger group was negatively affected by being in a minority group and using self-services and positively affected by having at least a post-secondary education and eligibility for 37 or more weeks of EI.

For claimant status, Appendix E-2a indicates that Entrepreneur positively affects both active and reachback EI claimants, while Partners has positive employment impacts for near-reachbacks only. As for differences in the impacts of the controls, it is EI claimants who appear to be more affected. Positively affecting the employment of claimants only are: being unemployed in the month before the intervention (compared to not in the labour force), have some post-secondary education (compared to having less than a high school certificate), and having an interest in entering the labour force. Being male and no dependants negatively affects this group only.

The second set of results in Exhibit 8.1b are for being full-time employed at the time of the survey (post intervention). Full-time employment would be considered by many to be a better outcome than being employed part-time or self-employed, which would be included as part of the previous dependent variable. The results in the first column of the exhibit indicate that, with no controls entered in the model, three programs — Partners, Entrepreneur, and EAS — significantly increase the chances of being full-time employed. The inclusion of the background controls (column two) and service-delivery variables (column three) results in no change in the interventions' impact on the dependent variable. Even the addition of the Heckman Correction to the model (fourth column) makes little difference.

As for the impacts of the controls, the full-model results in column one of Appendix E 2b indicate that the chances of obtaining full-time employment increase with the length of time since the intervention. The chances are also increased by the amount of education the participant obtained prior to program entry, as expected. Similarly, earning $30,000 or more in the year prior to receiving assistance raised the chances of full-time employment (compared to earning less than $5,000). On the other hand, being 55 years and older reduce the chances of full-time employment (compared to being less than 35 years old). These are the only control variables found to have a significant impact on this dependent variable.

The results from the segmented analysis presented in Appendix E-2b indicate that participating in Partners and Entrepreneur both have positive effects on the likelihood of full-time employment for both males and females, with EAS having no impact on either group. However, being employed or unemployed one month before the intervention (compared to not in the labour force) reduces the likelihood for women only, as does being 55 years and older and being in a minority group. As well, the amount of education increases the likelihood for both men and women, while earning $30,000 or more increases it for men only. Having received EI for two or more years in the five years leading up to the intervention reduces the likelihood of full-time employment for men only.

By age, Appendix E-2b indicates that Partners and Entrepreneur continued to positively influence the chances of entering full-time employment for both age groups, as does having at least some post-secondary education. Also, earning more than $30,000 in employment income in the year before the intervention had positive effects on the younger age group only. As well, EI eligibility increases the chances of full-time employment for men only.

Finally, the results of the segmented analysis by claimant status presented in Appendix E.2b indicate that Entrepreneur had a positive impact on the likelihood of full-time employment for both EI claimants and reachbacks, while Partners had a positive effect for reachbacks only. Education had a positive impact on reachbacks, as does earnings level.

The third set of employment outcome modelling results presented in Exhibit 8.1c are for being employed for (at least) 12 consecutive weeks in the post-intervention period. These results correspond fairly closely to HRDC's definition of the employment result for the LMDA benefits and measures and is indicative of more stable employment. The program-only modelling results (column one) indicate that just Entrepreneur and Partners have a statistically significant positive impact on this employment outcome. However, the inclusion of background controls in the model (column two) and then service delivery variables (column three) adds SLG to the interventions exerting a positive impact on the outcome variable. However, once the Heckman Correction is introduced, SLG again does not exert a positive impact, though Entrepreneur and Partners continue to play a positive role.

Appendix E-2c indicates that Partners and Entrepreneur both exert positive impacts on the dependent variable. A number of controls also had an influence on the outcome variable. The results indicate that the chances of being employed for 12 consecutive weeks increase (not expectedly) with the weeks since the intervention and education level; are (surprisingly) higher for those unemployed one month before and those employed one year before the intervention; are higher for those with a prior interest in entering the labour force; and are (surprisingly) higher for those with three or more separations prior to the intervention (compared to those with 2 or less). Conversely, the probability of 12 consecutive weeks of employment declines with age and is lower for Francophones compared to Anglophones,³⁴ and for those who used self-serve services.

Appendix E-2c also presents the segmented results. By sex, both Partners and Entrepreneur exert positive impacts on the dependent variable for both males and females, as does the length of time since the intervention, interest in entering the labour force and having more than two job separations prior to program entry. Being employed one year before the intervention exerts a positive impact on the likelihood of 12 consecutive weeks of employment for females only, as does having at least a high-school certificate. Conversely, age, being a Francophone, and being in minority group negatively affects the dependent variable for males only. Being married has positive impact on the dependent variable only for males.

By age, the segmented results presented in the appendix indicate that, for both younger and older age groups, Entrepreneur exerted a significantly positive impact on participants' likelihood of entering 12 consecutive weeks of employment after the intervention and Partners for just the younger group. Being unemployed one month before the intervention and being employed one year before have positive influence on the dependent variable for the older age group only. Having separations before the intervention and an interest in entering the labour force both have a positive effect on the chances of 12 consecutive weeks of employment for the older group only, while being a Francophone has a negative effect for this group only.

By claimant status, Partners and Entrepreneur were found to positively affect the likelihood of three consecutive months of employment for near reachbacks only (Appendix E-2c). Weeks since the intervention positively affected both groups. Being unemployed one month before the intervention and being employed one year before have positive influence on the dependent variable for claimants only, as does interest in entering the labour force and pre-program separations. Being in a minority group negatively affects the dependent variable for active claimants only, while being 45-54 years of age (compared to less than 35 and being male negatively affect it for near-reachbacks only.

The OLS regression results for the fourth employment measure, weeks working since the intervention as a percentage of the weeks since the intervention, are presented in Exhibit 8.2. The results indicate that, with just the program variables in the model (column one), all programs were found to exert a positive influence on the percentage of weeks employed post-intervention, except Job Action, which was found to have a negative effect (column one). Adding the background variables to the model (column two) results in little change in the coefficients except that Rural Experience no longer has a significant impact on the outcome variable, as does the addition of the service delivery variables (column three). Finally, the addition of the Heckman Correction (fourth column) results in only Entrepreneur and Partners having a positive influence on the dependent variable, with EAS no longer playing a significant role, along with the other interventions.

As for the role played by controls, Appendix E-3 indicates that the time since the intervention increases the percentage of time working in the post-intervention period, as did prior interest in entering the labour force and having some post-secondary education (compared to having no more than some high school). Interestingly, being unemployed in the month prior to the intervention and having more than two separations in the five-years before the intervention also increased the time working. Also, being 55 years and older (compared to less than 35 years of age), being a Francophone, claiming over 25 weeks of EI benefits in the prior five-year period (compared to 24 or less), and using self-serve and non-HRDC employment assistance services reduced the percentage of time working since the intervention.

Turning to differences by gender, the results in Appendix E-3 indicate, first, that only females are positively impacted by SLG with respect to the percentage of time working since the intervention, with Entrepreneur exerting significant impacts on both males and females. Male participants were negatively affected by EAS. Also, women not men are positively affected by having some post-secondary education and prior interest in entering the labour force, and negatively affected by having weeks of EI claims prior to the intervention and using other employment assistance services. On the other hand, the negative age impact is only for men, as is having French as a mother tongue,³⁵ having up to 36 weeks of EI eligibility overlapping with the intervention, having received 25 or more weeks of EI prior to the intervention, and using self-serve services. Meeting a counsellor had a positive impact for men only.

The differences by age group include the fact that only younger participants are negatively affected by Job Action. Only older participants were positively affected by being unemployed in the month prior to intervention, being employed one year before, having a high-school certificate, and having more than two separations in the week prior to the intervention. Only they were negatively affected by being Francophone and having received 25-104 weeks of EI benefits prior to the intervention. As well, only younger participants were positively affected by having post-secondary education, speaking a language other than English and French, having a prior interest in entering the labour force, having three to five separations, and having met a counsellor. And they alone were negatively affected by having 1-36 weeks of EI eligibility, having received over two years of EI benefits before the intervention, having received SA benefits prior to intervention, and the use of self-serve as well as other employment services. As for differences by claimant status, Job Action negatively affected claimants only, and Rural Experience positively affected this group only; it was mainly EI claimants not near-reachbacks who were affected by the many of the factors identified above including participating in EAS (negative). One new factor was earnings of $10,000 or more which negatively affected EI claimants only. Being married had a positive impact on near reachbacks only.

The results for the per cent of weeks looking for work while jobless since the intervention are presented in Exhibit 8.3. With just the program variables in the model (column one), the results indicate that Job Action has a significant positive impact on the percentage of time job searching, whereas Partners and Entrepreneur have a significant negative impact.

Introducing the control variable weeks since the intervention and background controls (column two) does not change the role on job search played by Partners and Entrepreneur, but Job Action no longer has an impact. Moreover, SLG and EAS are now found to have a negative effect on job search, implying that the control variables — more than these programs themselves — are affecting job search. Adding the service-delivery variables (column three) does not change the impacts of any program variable except EAS, which was found to no longer have a negative effect on job search. Finally, the introduction of the correction factor (column four) results in only Entrepreneur having a negative effect on percentage of weeks looking for work, with Partners and SLG no longer having a significant effect.

As for the controls, Appendix E-4 indicates that the longer the time since the intervention, the lower is the percentage of that time spent looking for work, thereby implying discouragement on the part of the participant. Being employed one month before the intervention increases the percentage, as does interest in entering the labour force, being on social assistance one year before the intervention, and having used self-service and other employment assistance services. Negative factors include being employed one year before the intervention, speaking a tongue other than English or French, and having employment separations prior to program entry.

As for differences across segments, Appendix E-4 further indicates that, first, there are no differences between men and women in terms of program impacts but there are differences in the impacts of controls. For women but not for men, the percentage of weeks looking for work is positively affected by being employed one month before the intervention, having received over 24 weeks of EI benefits before the intervention, and using self-serve and other employment assistance services. Negative impacts for only women were found for being employed one year before the intervention and having a prior record of separations. On the other hand, men's and not women's job search is positively affected by being 55 years and over, having a prior interest in entering the labour force, and having received SA prior to program entry, but negatively affected by having a mother tongue other than French or English and being married. Having used self-serve services positively affects job search for both men and women, but meeting counsellor negatively affects men only.

By age, Appendix E-4 indicates that, in addition to Entrepreneur having a negative impact on job search for the both younger and older age groups, Job Action positively affects only the former. Other differences of note are that only the older group's job search is negatively affected by being employed one year prior to employment and positively by a pre-intervention interest in entering the labour. On other hand, only the younger group is positively affected by having 1-36 weeks of EI eligibility, 105 weeks of more of EI benefits, prior receipt of SA.

Finally, the percentage of weeks looking for work is negatively influenced by Entrepreneur only for active EI claimants and positively affected by Job Action. Also, only claimants' job search is affected by the time since the intervention (negative), being employed one month before (positive), being employed one year prior (negative), speaking a mother tongue other than French or English (negative), having a prior interest in being trained (positive), having a prior record of separations (negative), and having received SA before (positive). Only near-reachbacks' job search is positively affected by being 55 years and older and having used other employment assistance.

Earnings Outcomes

In Exhibits 8.4a, b and c, we present the results for three earnings outcome measures. These are weekly income from employment or self-employment in the current or most recent job at the time of the survey; absolute change in weekly earnings from employment or self-employment between the most recent job and the last job prior to the intervention; and percentage change in weekly earnings. Modelling absolute earnings change captures overall changes in the wage bill, while modelling percentage change captures relative changes. To illustrate the difference between the two different earnings-change measures, a $100 increase for someone earning $100 before the intervention represents a significant percentage change from the perspective of the individual (100 per cent), but not necessarily to the overall wage bill. It should be noted that segmented analysis was not conducted for earnings outcomes due to small sample sizes.

The first set of results for weekly earnings level (for the current or most recent job) indicates that, when only the program variables are entered into the model, Partners, Entrepreneur and SLG were found to exert a significantly positive impact (column one of Exhibit 8.4). Note that no controls such as prior earnings are entered in this first model; prior earnings would undoubtedly have an impact here. When we introduce the background controls (column two) and then service-delivery variables (column three), no change is observed in the impact of the intervention variables. Even the introduction of the Heckman factor (column four) fails to alter the impact of these interventions, with Partners, Entrepreneur and SLG still having a positive impact on current weekly earnings.

As for the impact of the controls, Appendix E-5a indicates that a large number of them affected post-intervention current weekly earnings. Exerting a positive impact were: the length of time since the intervention, being employed one year before the intervention, having a post-secondary education (compared to less than a high-school certificate), being male, and having received EI for at least one year over the 1992-1997 period. Also, post-intervention earnings rise with pre-intervention earnings level (compared to earning less than $5,000 in that period). Exerting a negative influence were being employed one month before the intervention (compared to not in the labour force), having French as a mother tongue, and having a pre-intervention interest in being trained.

Turning to segmented results by sex, we observe in Exhibit E-5a that, as far as program impacts are concerned, Entrepreneur affected males only with Partners and SLG positively affecting both males and females. Once again, it is women who are affected by the controls, apart the positive influence of weeks since the intervention and pre-intervention earnings for both sexes. Positively affecting women's weekly earnings only are having a secondary school certificate, and negatively affecting only the latter are being employed one month before the intervention, being a Francophone, and having no dependants.

By age, we observe that, while SLG and Entrepreneur positively affects both age groups, Partners affects only the younger age group. Education and being male positively affect both age groups, while a pre-intervention interest in being trained negatively affects both groups. Affecting only the older group are being employed one month before the intervention (negative) and a pre-intervention interest in entering the labour force (positively).

Finally, claimant segmented results indicate that Partners, Entrepreneur and SLG all positively affect the weekly earnings of reachbacks only, as does the length of time since the intervention. And, apart from being male increasing the earnings of both claimant groups and having a post-secondary education affecting only active claimants, other controls affect the earnings of only reachbacks. Affecting only the latter's earnings are: having a high-school certificate, weeks receiving EI in the years prior to the intervention, and pre-intervention earnings level (all positive) and having no dependants (negative).

The results in Exhibit 8.4b indicate that, with just program variables in the model (column one), Entrepreneur had positive effect on change in earnings, while Job Action, EAS and Rural Experience each had a negative effect. Then, with the inclusion of the background variables (column two), we observe that Entrepreneur continues to have a positive effect and it is joined by SLG. Rural Experience continues to have a negative effect, but Job Action no longer does, being replaced by Rural Experience. The addition of the service-delivery variables (column three) does not affect these outcomes, but the inclusion of the Heckman Correction (column four) results in Rural Experience no longer exerting a negative effect on the change in weekly earnings, with Entrepreneur and SLG continuing to exert a significantly positive effect.

Among the control variables (Appendix E-5b), weeks since the intervention, education, being 45-54 years old (compared to less than 45 years old), being male, having earnings of between $10,000 and $20,000 all had a positive impact on earnings growth. Playing a negative role were being employed one year before the intervention, having a pre-intervention interest in being trained, and meeting a job counsellor.

As for the segmented results by sex, Appendix E-5b indicates that the programs affect males earnings growth only. While being employed one year before negatively affect both groups, other controls affect the sexes in different ways. Having at least a post-secondary education positively affects females only, age positively affects males and negatively affects females, being in minority group negatively affects females only, and pre-intervention weeks of EI receipt and earnings level both positively affect female earnings growth only.

By age, Entrepreneur affects positively the younger age group only, but SLG was found to positively affect both age groups. Once again there were mixed results by segment, with only being employed one year before negatively affecting both age groups and being male positively affecting both groups. Having a high-school certificate positively influenced earnings growth of the older age group only, while the time since the intervention and pre-intervention earnings level affected only youth. Having a pre-intervention interest in being trained negatively affected the earnings of youth, while an interest in entering the labour positively affected the older group only.

By claimant status, this time it is the reachbacks who are the only group to be affected (positively) by the interventions. Once again, males positively affected earnings growth of both claimant groups. Weeks since the intervention positively affect reachback earnings only, as do pre-intervention earnings. Education level positively affects claimants only, while being employed one year before the intervention negatively affects the earnings growth of reachbacks only.

The results in Exhibit 8.4c indicate that three interventions increased the percentage change in weekly earnings. These were Partners, Entrepreneur and SLG. Adding in the background controls (column two), the service delivery variables (column three) and the Heckman Correction (column four) did not alter the effect of these interventions.

As for the controls (Appendix E-5c), the weeks since the intervention, being employed one year before, education level, being male, and earning $10,000 or more in the year prior to the intervention all have a positive impact on percentage change in earnings. Two variables had a negative impact: being a Francophone (which in turn may be linked to the economic conditions in the regions Francophones tend to be located in) and a prior interest in being trained.

The segmented results by gender presented in Appendix E-5c indicate that Partners and SLG both exert a positive impact on percentage earnings growth, but Entrepreneur positively affects males only. The only other differences by sex are that being a Francophone negatively affects females only, while having a pre-intervention interest in being trained negatively affects males only. As for age, Entrepreneur affects percentage earnings growth of males only as does the weeks since the interventions. Partners and SLG have a positive impact on both males and females. The only other difference by age is that being employed one year before the intervention positively affects the percentage earnings growth of the older age group only. Finally, all three programs noted above as positively affecting percentage earnings growth have an impact on near-reachbacks only, as does having received over two years of EI benefits in the five years before the intervention. Having no dependants negatively affects this group only as does use of other employment services. On the other hand, being employed one year before has a positive impact on active EI claimant participants only, while having a pre-intervention interest in training negatively affects this group only.

Income Support Dependence

In addition to the goals of sustained employment and increased earnings, LMDA seeks to reduce dependence on employment insurance (EI) and social assistance (SA). In this section, we present results for outcomes in these areas. As noted above, the EI results are for new spells of EI in the post-intervention period, beyond the spell that corresponded to their intervention. Note that a negative result is really the sought-after outcome.

For percentage of weeks on EI, column one in Exhibit 8.5 indicates that, with just the program variables in the model (column one), all programs had a significant impact on the dependent variable. Noting that a negative effect does in fact represent a positive outcome, Rural Experience, Job Action and Partners had a positive impact whereas Entrepreneur, SLG and EAS had a negative impact. However, introducing control variables (column two) indicates that Partners no longer has a positive impact, which means that the prior impacts (without the controls) were a function more of the characteristics of the participants than the program itself. Then, with introduction of the service-delivery variables, we observe in column three that SLG no longer exerts a negative impact. However, with the addition of the Heckman Factor (column four), SLG once again joins Partners, Entrepreneur and EAS in reducing the number of weeks on EI in the post-intervention period. Job Action is found no longer to increase post-intervention weeks on EI but Rural Experience continues to do so.

Looking at the impact of the control variables for the full model (Appendix E-6a), it is observed that the variables having a lengthening impact on EI spells are the weeks since the intervention, being Francophone, having three or more separations before the intervention, and being on social assistance one year before the intervention. Some variables acted to reduce the weeks in receipt of EI including having a post-secondary education and having overlapping weeks of EI eligibility.

As for differences in EI use across segments, Appendix E-6a indicates that it is only the males whose post-intervention EI claimant period is lengthened by Rural Experience, while only females' claimant period is reduced by Partners and SLG. Entrepreneur and EAS act to reduce the spells for both men and women. The effects of the controls include only females' post-intervention receipt of EI being positively affected by pre-intervention job separations and negatively by pre-intervention receipt of EI, while only males are positively affected by being married and receiving SA in the year before the intervention. By age, it is observed in Appendix E-6a that it is only younger participants whose post-intervention weeks of EI receipt are negatively affected by Partners. On the other hand, it is only older participants who are positively affected by being unemployed in the month before the intervention, a pre-intervention interest in being trained, and having received SA in the year before the intervention. Only younger participants' post-intervention receipt of EI is positively affected by a pre-intervention interest in entering the labour force and being Francophone and negatively by having received 25-52 weeks of EI in the pre-intervention period (compared to less than 25 weeks).

Finally, the claimant-status segmented analysis reveals some major differences between EI claimants and reachbacks. Appendix E-6a indicates that it is mainly the near-reachbacks weeks of post-intervention EI benefits that are affected by the interventions, as four different interventions reduce their EI duration: Entrepreneur, Partners, SLG and EAS. Rural Experience positively affects only active EI claimants. Having a post-secondary education negatively affects only near-reachbacks while a pre-intervention interest in starting one's own business positively affects only them. Having three or more separations and having received SA in the year prior to program entry positively affect only active claimants' weeks of EI benefits and not near-reachbacks'.

In Exhibit 8.6, we present the results of our efforts to model receipt of SA in the post-intervention period, the last outcome variable we modelled. The first column indicates that, with no control variables entered, Job Action, EAS and Rural Experience appear to increase the chances of receiving SA following the intervention. Introducing the background controls (column two) has no impact on the measured effect of EAS and Job Action but renders the effect of Rural Experience not significant. It is the nature of the clientele, rather than the program itself, which appears to have increased the chances of SA receipt. In the third column, we show that, after controlling for service-delivery variables, only EAS led to SA use. However, with the introduction of the self-correction factor, we observe in column four that none of the interventions play a role in subsequent SA use.

The results for the control variables presented in Appendix E-6b indicate that a number of background variables play a role in the use of SA. The variables significantly increasing the chances of using SA in the post-intervention period are receiving SA benefits in the pre-intervention period, the weeks since the intervention, and being a member of a minority group. Variables significantly reducing the likelihood of SA use are being married and having no dependants.

The segmented analysis of post-intervention SA use by sex, age and claimant status revealed some interesting differences between segments (Appendix E-6b). For males only, EAS had a positive impact, as did being in a minority group and pre-intervention receipt of SA. Being married and having no dependants reduced the chances of SA use for women only, as did being employed or unemployed in the month before the intervention. By age, we observe that Partners reduced the chances of SA receipt and that being employed one year before the intervention reduced post-intervention receipt of SA only for younger participants, as did having no dependants. Being in minority group increased chances of SA receipt for younger participants only. On the other hand, it was only older participants whose post-intervention receipt of SA was positively affected by being married. Finally, Exhibit 8.6 indicates that being married and having more than two separations in the pre-intervention period reduced the chances of SA use for just active claimants, while having three to five separations increased it for this group only. Being in a minority group increased SA use after the intervention for near-reachbacks, while having six or more separations reduced SA use for this group only.

Summary by Intervention

Exhibit 8.7 presents a summary of the results of the modelling exercises. The exhibit provides in each cell of the first line the significant effect (if any) of the program on the respective outcome for participants, and the second line indicates the, significant effects (if any) of the program in different segments. If no segment is separately affected by any of the programs, no information was entered in the second line of the cell. Note, once again, that the short-term nature of this formative evaluation tends to naturally favour interventions such as Entrepreneur and Partners that have immediate employment outcomes. A summary of the findings, based on the exhibit, follows:

• Partners — Controlling for other factors, this program had a significant positive impact on all employment outcomes, overall; this was also true for all individual segments but active EI claimants and except for percentage of weeks working where no segment was affected. Further, it had no impact on the weeks looking for work as a percentage of the weeks since the intervention, overall and in any segment. Partners contributed positively to current weekly earnings levels and their percentage growth from before to after the intervention, overall and for all segments but older and EI claimants. As for absolute earnings growth, Partners had no effect overall on it, but did increase it for just males. Finally, it reduced weeks of post-intervention EI receipt, overall but only for females, younger participants and near-reachbacks, and reduced the likelihood of receiving SA in the post-intervention period only for younger participants.
• Entrepreneur — Controlling for other factors, this program had a significant positive impact on all employment outcomes, overall and for every segment except for claimants in the case of 12 consecutive of weeks of employment. It reduced the length of job search in all segments but near-reachbacks. As well, Entrepreneur increased all three earnings measures, overall but generally only for males, younger participants and near-reachbacks. Further, it had a significant negative impact on the relative duration of EI receipt, overall and in all segments but EI claimant participants, but had no impact on SA receipt.
• Job Action — Controlling for other factors, the only employment impact this program had was a significant positive impact on being full-time employed just for older participants. It had no impact on relative post-intervention job search except to increase it among younger and claimant participants. There were no earnings or income-support dependence impacts found.
• SLG — Controlling for other factors, among all participants and for every segment, this program had no significant impact on all employment and job search outcomes, except for a positive impact on weeks working as a percentage of weeks since the intervention for females only. SLG positively affected all earnings measures, overall and for all segments but active EI claimants. SLG reduced post-intervention EI receipt, overall and for all segments but males and claimants. It had no impact on SA use.
• EAS — Controlling for other factors, this program had a significant positive impact on just two employment outcomes: currently employed (overall but only for the younger and older segments) and currently full-time employed (overall and only for the older segment). No significant impact was detected on the other two employment outcomes (except for a negative impact on weeks working for males only), the job search outcome, and the earnings outcomes. EAS was found to reduce post-intervention weeks of EI receipt, overall and for all segments but active EI claimants. No SA impact was detected overall, but EAS did increase the chances of SA receipt for males.
• Rural Experience — Controlling for other factors, this program had no significant effect on any employment outcomes. No impact was detected for the job search and earnings outcomes. Finally, Rural Experience increased post-intervention use of EI overall, and in all segments except for females and near-reachbacks. No impact was detected for SA use. Further, it is worth noting that the program's negative effects on currently full-time employed and change in current weekly earnings and its positive effect on SA receipt (with just program variables in the model) were nullified by the introduction of background controls, implying that the program's negative impacts may be compensated for by the characteristics of participants.

Client Profile Summary

We complete the summary by providing profiles of successful participants based on the modelling results. Starting with employment outcomes, those who participate in Entrepreneur do best, with Partners a fairly close second. EAS is also beneficial with respect to currently employed and full-time employed. Having a post-secondary education and having a prior interest in entering the labour force increase the chances of positive employment outcomes.

The finding of positive employment outcomes is generally true of the total population and for individual segments. The exceptions to these patterns, as revealed in the segmented analysis, include the fact that EI claimants do not benefit from their participation in Partners, that females do not profit from their participation in Partners in terms of being currently employed, and that only older segments' chances of full-time employment benefit from participation in Job Action and EAS.

With respect to earnings outcomes, participation in Entrepreneur, SLG and Partners, in that order, did make a significant difference, controlling for other factors. Attributes associated with success in this respect are being male, being less than 45 years old, having a post-secondary education, and earnings level prior to the intervention. The segmented analysis indicated that EI claimants did not tend to do well from the standpoint of earnings.

As for income-support dependence, Entrepreneur, EAS, SLG and to a lesser extent Partners were found, overall, to reduce the length of a new EI spell in the post-intervention period, controlling for other factors. Near-reachbacks were found to benefit from the interventions more than those who were EI claimants at the time of the intervention. Additionally, for Partners successful participants tended to be female. As well, having a post-secondary education, being eligible for EI overlapping with the intervention, and not receiving SA before the intervention acted to reduce the length of the EI spell, it is also important to point out that the longer the period of time from the intervention the greater the relative length of the new EI spell. Conversely, Rural Experience acted to increase post-intervention EI spells, overall, and for all segments but near-reachbacks.

As for SA receipt, Partners participation appeared to reduce the chances of receiving SA in the post-intervention period for the younger segments while EAS increased it for males alone. The chances were also increased by being unmarried, being in a minority group, having dependants, and having received SA in the year prior to the intervention.

View EXHIBIT 8.7

Footnotes

31	Note that personal income was also considered as a dependent variable to be modelled. However, though the survey question referred to a one-year period, the period since the intervention and therefore the portion considered to be the "outcome", would vary for participants. Moreover, total personal income includes types of income that would not necessarily be linked to participation in the intervention.
32	In the univariate results presented in the previous chapter, we showed weeks not working. To be consistent with the other employment measures, which were all positive outcomes, we "reversed" this variable, so that the signs on the explanatory variables may be expected to be in the same direction as for these other outcome measures. In addition, because weeks since the intervention is controlled for in the model, it was deemed unnecessary to model the absolute number of weeks working as well.
33	Omitted from the analysis because of missing data is a regional control variable. It is quite likely that employment rates would be affected by local labour market conditions. This question will be pursued in the summative evaluation.
34	It is quite likely that the negative result for Francophones may have something to do with the fact Francophones are concentrated in a region of the province where unemployment is high and the work is seasonal. Indeed, differences in employment rates themselves are likely to some extent explained by differences in local labour markets. However, as noted above, for this formative evaluation, there was lack of data available to gauge this influence.
35	This may be a function of the fact that Francophones are concentrated in a high unemployment region. See previous footnote.