Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

3.1 Impacts on labour market activity

Participation in an Employment Benefit affects short-term employment. The size and direction of this effect varies among the interventions and between genders.

Table 13 provides the results of the regression modelling performed in this evaluation. The dependent variable used was percentage of time employed²³ in the post-program period.

The independent variables that were used to explain post-program labour market success are defined below. They include participation variables to allow estimation of the apparent effect of each of the three interventions. To enhance the precision of those estimates, 17 other independent variables are included. One of these (Months since program) is included since past research indicates that the effect of employment programs (whether positive or negative) changes over time. The other 16 independent variables adjust for other factors that are known to affect the success of individuals seeking employment (e.g. age, education, location, previous employment history, past use of EI or social assistance, etcetera). Consistent with the approach used in past evaluations of Canadian and U.S. employment programs, separate regression models were employed for males and females.

The regression models used did not include adjustment for selection bias. Selection bias occurs because candidates are not selected randomly for participation. Instead selection decisions are made by individuals who decide to apply for the program; by HRDC or third party officials; and/or employers or sponsors who decide whether to accept them. All three groups will make their decisions based on factors which can only be imperfectly quantified and are thus not typically included in an Ordinary Least Squares (OLS) regression model. Since participants may differ from the comparison sample on these unmeasured characteristics, regression models that do not account for selection decisions — such as those used in this evaluation - produce potentially biased estimates of program impact.

1. JCP Participation.
2. Participation in Targeted Wage Subsidies.
3. Participation in the Self-Employment Benefits program.
4. Reachback status. Applies both to participants marked as reachback and the comparison group members matched to those participants.
5. Graduation from high school.
6. Graduation from a college or university.
7. Average total income from 1991 to 1996 inclusive.
8. Attachment to the fishery (1 if any fishing income in any of the three previous years, 0 otherwise).
9. Urban/Rural (1 if St. John's, Mount Pearl, Grand Falls Windsor or Corner Brook).
10. Age and Age squared.
11. Aboriginal status (self-declared).
12. Visible minority (self-declared).
13. Disabled (self-declared).
14. Dependence on EI 1991 — 1996²⁴.
15. Central district.
16. Western District.
17. Labrador District.
18. Social Assistance Recipient (SAR) in quarter enrolled or previous quarter.

Interpretation of these coefficients depends on the variable and its units. For example, the coefficient of 17.729 for post-secondary graduation in the female regressions means that, on average, the model estimates that females who have a post secondary degree or diploma will be employed 17.729% more of the time than females who do not have such a qualification.

For both males and females the models are significant and explain a large amount of the variation in post-program employment with R²=.196 for males and R²=.257. These are strong results in explaining variation in employment of individuals and consistent with what is generally seen in the literature and other evaluations. For example, in Goss Gilroy's 1989 Evaluation of the Job Development Program — a national program targeting the long-term unemployed, extensive econometric modelling was conducted for various sub-populations. In models with a similar set of dependent variable, R²s in the 18% - 25% range were typical. At the extremes, individual regressions had R²s as low as 14% and as high as 36%.

As noted in the Job Development Evaluation Report: "As a rule, regressions using disaggregated data (such as observations on individuals as opposed to, say, observations on aggregates such as the average employment rate for the labour force) will produce lower R-squared values, so the value of 0.20 is not surprising and compares favourably with other regressions using similar data²⁶".

While these regression results are strong and explain significant variability in post-program employment, their interpretation is complicated by the presence of selection bias.

Table 14 below summarises the estimated effects of participation for the three interventions for each gender. Results are provided in terms of percentage time employed and the estimated increase (or decrease) in number of weeks worked per year. Note that the estimated effects of participation for all three interventions are statistically significant for males. For females the small estimated benefit for TWS is not significant and the small estimated negative effect for JCP is significant at the 5% level but not at the 1% level.

Effect of Job Creation Partnerships

Males who participated in a JCP project were employed after the program an estimated 12.1% more of the time than their previous work history and other characteristics would predict according to our models. A likely interpretation is that participation does significantly increase the likelihood that males will work more after the program — at least in the short term. However, as noted earlier, it is possible that some of this effect is due to selection bias or otherwise attributable to unmeasured characteristics of participants rather than the intervention itself. Correspondingly, females who participated were employed an estimated 9.0% less of the time after their program than the model would predict. A possible explanation is that at the end of a JCP project, many individuals are unemployed²⁹, and that female participants surveyed had not yet recovered from this initial unemployment³⁰. Another possible explanation could be differences (on average) in the projects which males and females have participated in. An important question for the summative evaluation will be to determine if this apparent negative effect persists and whether the gap between males and females remains.

In the key informant interviews, it was noted that improving short-term post-program employment of participants has not been a major focus for JCP. While JCP projects are expected to provide participants with skills and experience that may help their future employment, two other important goals were also noted³¹:

• Projects which offer economic development advantages may be approved at least partially on that basis. For example, one of the case studies was the Stephenville Theatre Festival. Part of the rationale for funding this project is for skill development of individuals. However, the rationale also includes the benefits to other industries of tourists attending the festival as well as the longer term potential for growth in cultural industries; and,
• Providing short-term income to participants who otherwise have limited opportunities for income. Particularly in small communities with limited (or seasonal) employment opportunities, participants may be largely motivated by short-term income considerations.

Effect of Targeted Wage Subsidies

Participants in the Targeted Wage Subsidies employment benefit have been employed significantly more in the post-program period than the models suggest based on their past labour market success and their other characteristics. This is especially the case for males who were employed 26.1% more of the post-program period than would have been predicted if they had not participated. The apparent benefit for female participants is much smaller at 5.6% of the post-program period. This estimated benefit for females is not statistically significantly different from zero.

Wage subsidy programs typically provide, at least, short term employment benefits since it is not uncommon for employers to retain some participants after the subsidy ends. Also some participants move immediately to a new job with another organization which they dealt with in their project employment. As noted in Table 7, 59% of TWS participants surveyed had a job immediately upon the end of the project. The important evaluation question — which can be addressed via the summative evaluation - is whether these short-term benefits persist.

Effect of Self-Employment Benefit

The Self-Employment Benefit intervention has an apparent strong positive effect on subsequent employment for both men and women:

• Men participating in SEB were employed an estimated 35.5 percent of the time more than if they had not participated. This amounts to an estimated 18.5 weeks of additional employment on an annual basis; and,
• Women participating in SEB were employed an estimated 38.8 percent of the time more than if they had not participated — an additional 20.2 weeks of employment on an annual basis.

Past evaluations of similar programs in Canada and other countries, have demonstrated that the positive benefits of such programs come from the survival of some businesses after the program and above-average employment success of those individuals who elect to close their businesses.

While the above estimates of program impact are extremely positive they are largely due to short term survival of the established businesses (75% of participants continued to operate their businesses at the time of their survey interview — 86% of those operated their businesses on a year round basis). Important questions for the summative evaluation will be the longer-term survival of those businesses and the employment success of individuals who elect to close their business.

3.2 SAR and Other Reachback clients

Reachback clients have mostly participated in JCP. Their post-program employment has not differed substantially, on average, from other JCP participants.

Employment Benefits are available to active EI claimants. They are also available to "reachback clients". Reachback clients are defined as those who do not have an active EI claim but who:

• had an active claim within the past three years; or,
• had an active claim or established a benefit period for parental or maternity benefits or sick benefits within the past 5 years.

Within the models, an indicator variable for reachback status was included to test whether the results varied for clients in these groups. For both males and females, the coefficient of this variable was not statistically significant indicating that no such effect exists.

The raw data in Table 7 indicates a somewhat more complicated picture. In terms of percent employed at the time of the survey (40% for reachback versus 38% for all JCP) and the average percent time employed since the program (47% for reachback versus 46% for all JCP) there seems to be very little difference between the reachback clients and other clients. However, reachback clients are more likely to have not worked at all (34% versus 23% of all reachback clients) and quite unlikely to have had limited work (3% were employed 1 to 24% of the time versus 9% of all JCP clients).

Recent receipt of social assistance is a more useful predictor of post-program employment success than reachback status. As can be seen in Table 13, for males, recent receipt of Social Assistance has an estimated reduction of 12.5 percentage points in the percentage time employed after the program. The regression model also estimates a negative effect for females although this estimate is not statistically significantly different from 0.

3.3 Effects of Interventions Pre- and Post-LMDA

As noted in the description of the methodology, the survey of participants concentrated mostly on participants since the Labour Market Development Agreement became effective but did include some participants in the time period between the effective date of the new EI legislation and the LMDA. In particular, of 1,493 participants interviewed, 448 started their intervention between July 1, 1996 and March 31, 1997 while 1,045 started after the latter date. The reason for including the first group was to test the hypothesis that effect of the interventions varied in the time periods.

Footnotes

23	Percentage time employed was calculated as the ratio of # months employed since the program to # months since completing the program. Time attending a post-secondary institution was excluded from both the numerator and denominator. Also, individuals for whom the denominator was less than three months (after excluding time in a post-secondary institution) were dropped from the regression.
24	Defined as the % of total earnings during the 1991-1996 period which were EI benefits
25	A "?" indicates a dummy variable which takes the value 1 if the characteristic applies and 0 otherwise.
26	Such as, for example, econometric models relating to other elements of the Canadian Jobs Strategy and U.S. job training programs.
27	This estimate is significantly different from 0 at the 5% level of significance but not at the 1% level.
28	This estimate is not significantly different from 0.
29	Since many JCP projects are with non-profit organizations, it is unusual for individuals to be retained after the project. This contrasts significantly with TWS and SEB where continuation of the job or the business (at least for the short term) is the norm. See Table 7 where 59% of TWS participants and 82% of SEB participants had a job at the end of the program compared to only 34% of JCP participants.
30	This interpretation is supported by the fact that coefficient of Months since program is double the equivalent coefficient for males.
31	In fact, all key informants noted that improving the skills and experience of participants was typically a lower priority than these other objectives.