|
|
     |
||
 |
     |
 |
|
This chapter presents the econometric analysis of the impacts of Employment Benefits and Support Measures in Nova Scotia. Because there are several outcomes of interest, the chapter proceeds in stages, examining outcome measures in the same sequence as in the previous chapter. Note that the measures of impacts presented here should be considered preliminary only. More definitive measures will be presented in a summative evaluation when there will be more time following Employment Benefits and Support Measures (EBSM) participation to detect impacts. 9.1 Recent Labour Market HistoryThis section analyzes the relationship between participating in EBSM and the time spent in three main activities: work, in school, and unemployment. The survey of participants and non-participants asked about time (in months) devoted during each of the years 1995 to 1998 to the following labour market activities: In School and Not Working, In School and Working, Working and Not in School, and Not Working and Not in School. Because only a small amount of time is devoted to "In School and Working," these four activities were combined into three: Working, In School, and Unemployed (Not in School/Not Working). None of the findings is altered by treating the four activities separately. The focus of the analysis is the time devoted to these activities in 1998. Because some of the participants were still in EBSM during 1998, the analysis in this section is restricted to participants who completed or dropped out of the program during 1997. If this was not done, it would not be possible to separate the effects on time spent on various activities that are associated with participating in EBSM from the impacts of EBSM on these activities. The comparison group consists of regular Employment Insurance (EI) recipients who did not participate in EBSM during the sample period.38 The sample analyzed consists of 559 participants and 499 non-participants. For each of the three principal activities three types of estimates of program impacts are obtained. The first are simple linear regression models with and without controls for observable demographic and individual characteristics (referred to in the tables as "demographic controls"). Explanatory variables used in these regressions are as follows: age, educational attainment, gender, marital status, single parent status, number of children under 6 years of age, number of children 6 to 18 years of age, visible minority status, limitation/disabled status, aboriginal status, and language (English, French, Other). Age is allowed to have a non-linear effect on the outcome of interest, as most research finds that the effect of Age on various outcomes displays diminishing returns. These regressions compare participants and non-participants in the post-program year 1998. Although they control for the influence of observable factors that may differ between participants and non-participants, they do not account for possible non-observable factors that may influence selection into the program. The other estimates reported utilize the longitudinal nature of the data on time spent in labour force activities in order to account for certain types of non-random selection into the program. In the presence of these types of non-random selection the simple linear regression models discussed above would yield biased estimates of program impact. Two types of longitudinal estimators are employed; both take advantage of pre-program information obtained in the survey on time spent in each activity in 1995 and 1996 as well as 1998. "Difference-in-differences" estimators provide unbiased estimates of program impact if selection into the program is based on unobserved person-specific and time-invariant factors that also influence the time spent on these activities by individuals. Linear regression estimates that control for pre-program levels of the activities in question are also reported. These longitudinal estimators are sometimes referred to as "unrestricted difference-in-differences" estimators because they do not restrict the coefficient on the pre-program level to be unity, as is the case with difference-indifferences estimators. Table 9.1.1 reports the estimated program impacts on the three activities based on these alternative specifications. Estimates from a variety of specifications are reported in order to determine whether the estimated impacts are sensitive to alternative assumptions about the nature of selection into the program. All estimates use only the subset of the sample of participants who completed EBSM in 1997. In 1995 and 1996 (i.e., prior to EBSM), those who became EBSM participants spent less time working, more time in school and more time unemployed than did those who did not participate in the program. These differences between participants and non-participants were approximately -1.1 to -1.4 months working per year, +0.6 to +0.7 months in school each year, and +0.4 to +0.7 months unemployed each year. EBSM clearly attracted or selected individuals who were more likely to be in school or unemployed during the year and less likely to be working than those who did not apply for or who were not selected into the program. These systematic differences in pre-program activities suggest the presence of non-random selection into the participant and non-participant groups. The differences between participants and non-participants are much smaller after the completion of the program. In 1998, participants continued to spend less time working than non-participants, but the gap between the two groups had narrowed to -0.1 months, i.e., had almost disappeared. Similarly the gap in time spent in school narrowed to +0.2 months and the gap in time spent unemployed was actually reversed in sign to -0.2 months. In the post-program year 1998, only the difference in time spent in school is statistically significant (see equation 9.1.1). This narrowing of differences between participants and non-participants is principally due to participants increasing their time spent working relative to pre-program levels and decreasing time spent in school and unemployed relative to pre-program levels of these activities. These results suggest that the program may have had a positive impact on time spent working and negative impacts on time spent in school and unemployed.
The first two sets of estimates reported in Table 9.1.1 (equations 9.1.1 and 9.1.2) compare participants and non-participants in the post-program period, 1998. These indicate that participants spent less time working and more time in school than non-participants and, after controlling for demographic characteristics, more time unemployed than non-participants. None of the differences between the two groups is large, and only the difference in time spent working is statistically significant. However, these estimates do not take into account the clear evidence of non-random selection into the program. For this reason the longitudinal estimates are preferred (equations 9.1.3 to 9.1.10). For the activity "In School," all the longitudinal estimates yield the same conclusion —that EBSM did not have a significant impact on the time spent in this activity. The estimated impacts are all small in magnitude and none is significantly different from zero. The evidence for the activities "Working" and "Unemployed" is more mixed, with some estimates indicating a positive impact on time spent working and a negative impact on time unemployed and other estimates suggesting no impact. In turn the evidence relating to each of these activities is discussed. The difference-in-differences estimates (without demographic controls) suggest that the program may have had a positive impact on time spent working — estimates of +1.0 months using 1996 as the base year and +0.7 months using 1995 as the base year. However, these estimates become smaller once demographic controls are included and the estimate for the 1995 base year is no longer significantly different from zero. Because the specification with demographic controls is more general, these estimates are preferable to the simple difference-in-differences estimates without controls. These two estimates indicate that the program increased time spent working by 0.4 months using 1995 as the base year to 0.7 months using 1996 as the base year. This is a fairly narrow range, and the two estimates are not significantly different from each other in a statistical sense.40 This is an important finding because the assumptions under which the difference-in-differences methodology provides unbiased estimates of program impact imply that the estimated impacts should not be sensitive to the choice of base year. The estimates reported in equations 9.1.4 and 9.1.6 pass this specification test. Thus the difference-in-differences estimates suggest that the program may have had a small positive impact on time spent working (0.4 to 0.7 months per year); however, the 95 percent confidence interval on the lower estimate of 0.4 months includes zero, so this evidence is also consistent with the view that the program had no impact. Somewhat in contrast, the regressions which control for pre-program activity levels (equations 9.1.8 to 9.1.10) indicate that the program did not have a statistically significant impact on time spent working. Choosing between these longitudinal estimates and the differences-in-differences estimates is not straightforward as each is based on a different set of assumptions about the nature of selection into the program. Given the clear evidence from the pre-program data of non-random selection into EBSM, the difference-in-differences estimates are preferred when, as is the case here, these pass the specification test of being invariant with respect to the choice of base year. Turning to the final activity, there is weak evidence from the difference-in-differences estimates that the program may have had a small negative impact on time spent unemployed, with the estimates ranging from —0.5 months for 1996 as the base year to -0.1 months for 1995 as the base year. However, the former estimate is only marginally significant (10 percent level) and the latter is not significantly different from zero. Furthermore the estimates that control for pre-program activity levels as well as demographic factors are also not significantly different from zero. As a consequence, the evidence from the various longitudinal estimates suggests that EBSM did not significantly affect time spent unemployed. In summary, EBSM does not appear to have had any impact on the time spent in school. With respect to time spent working and unemployed the evidence is more mixed. The differences between participants and non-participants in the time devoted to these activities narrowed following the program, and in the desired directions: toward more time working and less time unemployed. However, these changes were not large enough for EBSM to have a statistically significant impact on time spent unemployed, at least in the short term (first year following the program). The evidence relating to work activity is somewhat more positive, with the preferred estimates being in the 0.4 to 0.7 months range; nonetheless it cannot be claimed with confidence that this impact is significantly different from zero. Tables 9.1.2 to 9.1.6 report the results of the analysis by program component. The sample of 499 non-participants is used throughout, and the sample of 559 participants is examined separately for each of the five program components: Self-Employment (Table 9.1.2), Job Creation Partnerships (Table 9.1.3), Targeted Wage Subsidies (Table 9.1.4), Employment Assistance Services (Table 9.1.5), and Purchase of training (Table 9.1.6). For Self-Employment, each of the estimated impacts on time spent working is positive but none is statistically significant. The longitudinal estimates range from +1.1 to +2.1 months. Similarly, the estimated impacts on time spent unemployed are consistently negative, but also generally not statistically significant. The longitudinal estimates range from -1.2 to -2.8 months. The longitudinal estimates of the impact on time spent in school are consistently positive, small in magnitude (0.1 to 0.5 months) and not statistically significant. Overall these results suggest that Self-Employment may have had a positive impact on time spent working and a similarly desirable negative impact on time spent unemployed, but larger sample sizes would be needed to obtain more precise estimates which are significantly different from zero. The results for Job Creation Partnerships are similar to those for EBSM as a whole. When participants and non-participants are compared in the post-program period alone (and after controlling for demographic factors), participants spent significantly less time working and significantly more time unemployed than non-participants. However, the longitudinal estimates indicate that these post-program differences should not be attributed to the program. Indeed, the simple difference-in-differences estimates suggest a positive impact of EBSM on time spent working. However, once demographic factors are taken into account the evidence is more mixed, with one estimate of +1.4 months (1996 base year) being significantly greater than zero and the other estimate of +1.1 months (1995 base year) not being significantly different from zero. Note that these estimates of +1.1 to +1.4 months do not differ significantly from each other, and thus pass the specification test discussed previously. This result increases confidence in the difference-in-differences estimates; nonetheless, because the 1995 base year estimate is not significantly different from zero it cannot be concluded that there is unambiguous evidence that the program increased time spent working. Turning to unemployment, although most of the longitudinal estimates are negative, none is statistically significant. Thus there is no evidence that the program reduced the duration of unemployment. Finally, the estimated impacts on time in school are consistently small in magnitude and none are significantly different from zero so there is clear evidence that the program did not impact on this activity. The results for the Targeted Wage Subsidies component provide the strongest evidence of positive program impacts. In particular, after controlling for demographic and individual characteristics, the difference-in-differences estimates indicate a positive impact of 1.2 to 2 months, depending on the base year. Each of these estimates is significantly greater than zero; they also pass the specification test of not being significantly different (at the 5 percent level) from each other for alternative choices of the base year. However, the longitudinal estimates which include pre-program activity levels as controls result in a different conclusion — that the program had a small positive (+0.6 months) but statistically insignificant impact on time spent working. It is not possible with the available data to determine which of these two sets of estimates is more credible. Because the difference-in-differences estimates pass the "base year" specification test these estimates are preferred. Thus the Targeted Wage Subsidies program component appears to have had a positive impact of approximately 1.2 to 2.0 months on time spent working during 1998, but this conclusion is not supported by all the evidence. The evidence relating to time spent unemployed is weaker, with all the longitudinal estimates being negative but most not being significantly different from zero. There is also some evidence that Targeted Wage Subsidies reduced time spent in school, but this again is mixed. The results for the Employment Assistance Services and Purchase of training components are shown in Tables 9.1.5 and 9.1.6 respectively. Both of these components appear to have had no significant impact on time spent in the three labour market activities.
9.2 EarningsThis section analyzes the impact of EBSM on earnings, using both survey-based data on post-program earnings and administrative data on pre-program earnings. The survey asked respondents how much money they earned before deductions from all jobs in 1998, and this is used as the measure of post-program earnings for EBSM participants and non-participants. The administrative data provide information on gross earnings for the preprogram years 1992 through 1996. Because the survey-based and administrative data on earnings may differ for a variety of reasons some caution is required in interpreting the results of the analysis of earnings. However, as noted in the previous section, combining survey-based and administrative information will not bias the results as long as any differences between the two sources of data affect both the participants and non-participants to a similar extent. In these circumstances, the differences associated with the alternative earnings measures are removed by having a comparison group of non-participants. In order to separate the earnings behavior associated with EBSM participation from the possible impact of the program on subsequent earnings, the focus is on those participants who completed EBSM in 1997. After removing those individuals who did not respond to the earnings question in the survey, this leaves a sample of 311 participants and 284 non-participants. The results reported in Table 9.2.1 are based on this sample and subsets of the participant sample for the various program components. Earnings is the most commonly used measure of success associated with employment and training programs because it combines both the effects of employability (weeks worked per year, hours worked per week) and the "price" of labour (weekly or hourly wage rate). For this reason the analysis includes those who report zero earnings. Thus results are for all the observations available on participants and non-participants, and are not conditional on the subset who report positive earnings. Average annual earnings of EBSM participants was about $ 670 higher than that of non-participants in 1998, although the difference is not statistically significant (mean difference of $ 669; see equation 9.2.1). However, once observed demographic and individual characteristics are controlled, the regression-adjusted mean difference is negative and approximately $1,500 smaller (see equation 9.2.2). This indicates that EBSM selected (or was selected by) individuals with observable characteristics that made them likely to earn less than the comparison group of non-participants. This indication is confirmed by inspection of the administrative data on pre-program earnings: those who became enrolled in the ESBM program consistently earned less during the years 1992 through 1996 than did those who did not participate in EBSM. Thus there is evidence of non-random selection into the program. EBSM participants have observable (and possibly also unobservable) characteristics that result in lower earnings than non-participants even in the absence of the program. Estimates of program impact that do not take into account this non-random selection — such as the mean difference in post-program earnings (equation 9.2.1) and the regression-adjusted mean difference in post-program earnings (equation 9.2.2) — are likely to be biased. Equations 9.2.3 to 9.2.12 report difference-in-differences estimates of program impact for alternative choices of the base year (1992 to 1996). The simple difference-in-differences estimates are appropriate when selection into the program is influenced by unobserved factors that also influence earnings. These unobserved factors may vary across individuals but are assumed to be constant over time. In these circumstances, taking pre-versus post-program differences removes the influence of these unobserved factors, thus providing estimates that are free from selection bias. The difference-in-differences estimates with demographic controls take account of both observed and unobserved differences between participants and non-participants. These regression-adjusted difference-in-differences estimates are thus more general and the analysis will therefore focus on these. For EBSM as a whole, the regression-adjusted difference-in-differences estimates are all positive but none is significantly different from zero. The magnitude of the estimates ranges from $1,084 using 1996 as the base year to $2,403 using 1995 as base year. Note that apart from the 1996 base year estimate, the range of estimates is reasonably narrow: $1,826 (1992 base year) to $2,403 (1995 base year). A common finding in evaluating employment and training programs is that the earnings of program participants "dip" in the year prior to entry into the program (i.e., 1996 in the case of EBSM). This decline in earnings in the period prior to the program is not surprising; it simply indicates that those who apply for or who are selected for employment and training programs enter these programs because they have unusually low earnings due to job loss or other employment related adverse outcomes. For this reason, earnings in the period immediately prior to entry into the program are not representative of the "normal earnings" of these individuals. Accordingly, longitudinal estimates of program impact generally avoid using the data from the period just prior to program entry. When the comparison group is chosen from large nationally representative surveys, the earnings of the non-participants generally do not display this decline in the period immediately prior to the program. This outcome is to be expected because the sample of non-participants is not drawn from a group that has experienced a poor employment-related outcome. In the analysis of EBSM, it is fortunate that the comparison group is drawn from regular EI recipients, many of whom, like the participant group, have entered EI because of an adverse outcome such as job loss. Thus the earnings of comparison group members are expected to also be unusually low relative to their historical pattern in the period just prior to the EI spell. This is in fact the case: the earnings of both the participant and non-participant groups decline noticeably in 1996 relative to their average levels over the 1992-1995 period. Because there would generally be a tendency for earnings of both groups to return to their usual levels even in the absence of the program, it is advisable to place less weight on the difference-in-differences estimates for the pre-program year 1996 than for earlier years 1992 to 1995. As a consequence, the principal focus will be on the estimates using the base years 1992 to 1995.  View TABLE 9.2.1 - Estimates of the Impact of EBSM and Components on Annual Earnings As discussed previously, an important test of the key assumption which underlies the difference-in-differences estimates — that the unobserved factors which influence both participation in the program and the earnings of participants are constant over time — is that the estimates for different choices of the base year should not be statistically different from each other. Both the simple difference-in-differences estimates for the base years 1992 to 1995 and the regression-adjusted difference-in-differences estimates for the same years pass this specification test. The more general regression-adjusted estimates range from $1,826 (1992 base year) to $2,403 (1995 base year), and these estimates are all within one standard error of each other (see the standard errors reported in equations 9.2.6, 9.2.8, 9.2.10, and 9.2.12 in Table 9.2.1). That the difference-in-differences estimates pass this specification test increases confidence in these estimates of program impact. The longitudinal estimates that include pre-program earnings as controls (equations 9.2.13 to 9.2.15) are consistently negative and fall in the range of -$283 to -$467. However, none of these estimates is significantly different from zero, so they are consistent with the regression-adjusted difference-in-differences estimates that also find no significant impact of EBSM on earnings. Note that although these estimates control for observed differences in pre-program earnings between participants and non-participants, they do not take account of unobserved factors that influence both participation and earnings and will generally be biased estimates of program impact in the presence of such factors. The fact that the difference-in-differences estimates pass the specification test of being invariant with respect to the base year suggests the presence of such unobserved influences on participation and earnings. For this reason, the evidence provides more support for the difference-in-differences specification than for the specification using pre-program earnings as controls. In summary, both sets of longitudinal estimates — those based on the regression-adjusted difference-in-differences specification and those based on the specification including preprogram earnings as controls — support the conclusion that EBSM did not have a significant impact on earnings in the year following program completion. The difference-in-differences estimates are consistently positive and indicate an impact of approximately + $2,000, but these impacts are imprecisely estimated and one cannot reject the hypothesis that they are not significantly different from zero. Similarly the regressions which include pre-program earnings as controls are consistently negative but do not differ significantly from zero. The results reported in Table 9.2.1 provide an impact estimate that is common to all participants in the program. In order to investigate whether the intervention was more effective for some types of participants than for others, the analysis also estimated each of the equations reported in Table 9.2.1 allowing the estimated EBSM impacts to differ by gender, visible minority status and disability status. The results suggest that there are not significant differences in program impact by these characteristics. Specifically, estimated impacts are generally (but not always) higher for females than for males, but the differences between females and males are never statistically significant. Estimated impacts are consistently higher for the disabled than for non-disabled, but none of these differences is statistically significant. With respect to visible minorities, the estimated impacts are smaller than for the other participants, and in the majority of cases these differences are not statistically significant. Thus it is concluded that there is no clear evidence that the impacts of the program on earnings (which are themselves not significantly different from zero) differ by gender, visible minority status or disability status. Table 9.2.1 also reports estimates of the impact of the program on earnings for the various program components. There are large differences across the program components in the earnings of participants in the year following program completion (see equation 9.2.1). Those in the Self-Employment, Job Creation Partnerships, Targeted Wage Subsidies and Employment Assistance Services components earned less than their counterparts in the non-participant group during 1998, with the largest gap being Job Creation Partnerships participants whose earnings averaged approximately $7,500 below those of non-participants. In contrast, the Purchase of training participants earned $3,186 more than non-participants in 1998. These summary statistics suggest that there may have been large differences across the program components in the impact of EBSM on earnings. In order to determine whether these differences in average post-program outcomes can be attributed to the program, the analysis turns to the longitudinal estimates of program impact. For reasons discussed previously, the researchers have the most confidence in the regression-adjusted difference-in-differences estimates for the base years 1992 to 1995 (equations 9.2.6, 9.2.8, 9.2.10, and 9.2.12). For each of the components Self-Employment, Job Creation Partnerships, Targeted Wage Subsidies and Employment Assistance Services, the estimated impacts based on the longitudinal estimators are generally negative; however, because the sample sizes are quite small most of the estimates are very imprecise, with standard errors often larger than the estimated impact coefficients. Thus the only statement that can be made with confidence about the impact of these four program components is that they do not appear to have significantly reduced earnings of participants relative to what their earnings would have been in the absence of the program. The one exception is the Purchase of training component. The regression-adjusted difference-in-differences estimates are consistently positive and fall in the range of $3,206 (1994 base year) to $3,673 (1992 base year). None of the estimates for the base years 1992 to 1995 inclusive is statistically significantly different from each other, and three of the four estimates are significantly different from zero at the 10 percent level of significance, with the fourth being on the margin of significance at that level of significance. Thus the difference-in-differences estimates for the Purchase of training component pass the specification test of the assumptions underlying that type of longitudinal estimator. As was the case for the estimates for EBSM as a whole, the longitudinal estimates which simply include pre-program earnings as controls are smaller in magnitude (ranging from $483 to $699) and are not significantly different from zero. For the reasons discussed previously with respect to the EBSM estimates, the difference-in-differences estimates are superior. Accordingly there is some evidence that the Purchase of training option raised the earnings of Purchase of training participants by approximately $3,500 in the year following the program. As a final step in the analysis of earnings, the evaluators examined whether those who contributed to the cost of their intervention did better in terms of post-program earnings than those who didn't. In the case of the Purchase of training component, the analysis tested whether those who paid for some of their training had estimated earnings impacts that were different than those who did not contribute to the cost of their training. The difference-in-differences estimates (with demographic controls) for the base years 1992 to 1995 are consistently larger for those who contributed to the cost of their training than for Purchase of training participants who did not so contribute. However, none of these estimates is significant at the 5 percent level, and only two (those for the 1995 and 1994 base years) are significant at the 10 percent level. Although the estimates are all positive, they are too imprecise to reach a firm conclusion about the impact of contributing to the cost of training. Similarly, for the Self-Employment component the evaluators investigated whether those who contributed to the initial capital investment in their business did better in terms of subsequent earnings.41 Because all the Self-Employment participants who completed the program during 1997 contributed a positive amount to the business, the analysis examined whether there was any difference between those who contributed less than the median amount of $9,000 and those who contributed more than the median amount. No systematic differences were found between these two subsets of the Self-Employment component. The estimated earnings impacts are sometimes smaller for those contributing above the median amount and sometimes larger, and the differences between those with above and below median contributions were not significant for any of the choices of base year. 9.3 Employment Insurance BenefitsIn this section impact on use of Employment Insurance is investigated, using administrative data on paid weeks of EI benefits and total EI benefits received. These data are available on an annual basis from 1992 to 1998. In order to distinguish between any effects of participation in the program on EI receipt during the program itself and any impacts of the program on subsequent behavior, the analysis focuses on those participants who completed EBSM during 1997. For these individuals, the administrative data for 1998 provide some early indication of the possible impacts of the program on reliance on EI. Tables 9.3.1 and 9.3.2 provide estimates of the impact of EBSM and its components on two measures of EI receipt: (1) Paid Weeks of EI Benefits and (2) Total EI Benefits. In the post-program year 1998, EBSM participants received 0.6 weeks more EI benefits than did non-participants (equation 9.3.1) although the difference was not statistically significant. However, those who participated in EBSM received, on average, EI benefits for two to three weeks more than did non-participants during the period prior to EBSM (1992 to 1996). Accordingly, most of the longitudinal estimates of program impact, which take account of pre-program differences in weeks of EI benefits, suggest that EBSM may have reduced weeks of EI benefit receipt. The regression-adjusted difference-in-differences estimates are, however, not fully conclusive because they range from -0.3 weeks (and not significantly different from zero) using 1992 as the base year to -2.5 weeks using 1994 as the base year. The difference-in-differences estimates do not pass the specification test of being invariant with respect to the choice of base year. Thus there is some evidence that EBSM may have reduced weeks of EI receipt, but one could also interpret this evidence as suggesting that the program had no significant impact on weeks of EI benefits. The estimated impacts on total EI benefits are similarly inconclusive. The regression-adjusted difference-in-differences estimates range from + $13 (and not significantly different from zero) using 1992 as the base year to - $530 using 1994 as the base year. Again these estimates do not pass the base year specification test. Some of the longitudinal estimates suggest that EBSM reduced EI benefits received by $400 to $500 per year, while others suggest that the program did not have a significant impact on EI benefits. It is not possible on the basis of the available information to choose between these alternative conclusions. However, one clear conclusion does follow from the evidence in Tables 9.3.1 and 9.3.2: EBSM does not appear to have increased reliance on EI, as measured by weeks of EI benefits or total EI benefits received. This finding is important because the survey-based evidence on post-program EI use at a point in time (see section 9.5) suggests that EBSM may have increased reliance on EI. Because the analysis in this section is based on EI use by participants and non-participants before and after the program, these results are more robust than in those obtained in that section. As indicated in Tables 9.3.1 and 9.3.2, there are large differences across program components in the estimated impacts of the program on EI use. The Self-Employment component appears to have been particularly effective in reducing EI weeks and total benefits received. All the estimated impacts on weeks of EI are negative and fall in the range of 7 to 14 weeks. Similarly, the estimated impacts on total EI benefits are consistently negative and lie in the range of $2,000 to $4,000. Thus there is considerable evidence that the Self-Employment component had a substantial impact on EI use. There is also some, albeit less conclusive, evidence that the Job Creation Partnerships component reduced reliance on EI during 1998. Keep in mind, though, that work under Job Creation Partnerships is not insurable, and this alone could account for decreased post-program use of EI. The regression-adjusted difference-in-differences estimates of the impact of Job Creation Partnerships on weeks of EI benefits are consistently negative and fall in the range of —3.0 to —7.1 weeks; however, not all these estimates are significantly different from zero. Similarly, all the longitudinal estimates of the impact of Job Creation Partnerships on total EI benefits are negative, although not all are significantly different from zero. Most of the estimated impacts of the Purchase of training component on EI use are also negative, suggesting that if anything this program component acted to reduce EI weeks and benefits. However, these estimates are smaller than those obtained for Self-Employment and Job Creation Partnerships and several are not significantly different from zero. Thus the Purchase of training component may have reduced EI use to a modest degree, but the evidence supporting this finding is not conclusive. One could also conclude from the evidence in Tables 9.3.1 and 9.3.2 that the Purchase of training component had no significant impact on reliance on EI. Finally, the Targeted Wage Subsidies and Employment Assistance Services components appear to have had no impact on EI use in the year following the completion of the program. View TABLE 9.3.1 - Estimates of the Impact of EBSM and Components on Paid Weeks of EI Benefits View TABLE 9.3.2 - Estimates of the Impact of EBSM and Components on Total EI Benefits In summary, there is some (weak) evidence that EBSM reduced EI use, measured in terms of both weeks of benefits and total benefits received. To the extent that this reduction occurred, it came principally from the Self-Employment component that had a large impact on the use of EI, and to a lesser extent from the Job Creation Partnerships component. The Purchase of training component may also have contributed to a reduction in EI receipt. The Targeted Wage Subsidies and Employment Assistance Services components appear to have had no impact on this outcome. 9.4 Use of Social AssistanceThis section examines the impact of the program on social assistance receipt to determine whether there is evidence that the program helped reduce reliance on social assistance. The survey asked participants and non-participants whether they received social assistance during 1998, as well as the number of months of social assistance receipt. The administrative data provide information on income from social assistance for each year from 1992 to 1997. Combining the administrative and survey data yields a variable that measures whether or not the individual received social assistance benefits sometime during the year; this variable is constructed from the administrative data for the years 1992 to 1996 and from the survey data for 1998. Because the analysis combines administrative and survey data, some caution is required in interpreting the results. In particular, it is generally found to be the case that survey respondents tend to under-report both whether or not they received social assistance and the amount of social assistance income received. However, this tendency will not bias the results providing that program participants and non-participants under-report social assistance receipt to the same degree. For some of the survey respondents, their extent of reliance on social assistance during 1998 combines the effects of participating in the program with any post-program impacts of EBSM. As in the previous section, in order to obtain estimates of EBSM impacts that are not contaminated in this way the analysis is restricted to those participants who completed EBSM in 1997. This yields a sample of 557 participants and 499 non-participants. The estimates for this group thus provide evidence on reliance on social assistance during the year after program completion. EBSM participants were more likely to receive social assistance following the program than were the non-participants (11.3 percent versus 5.4 percent, a difference of 5.9 percentage points). However, EBSM participants were also more likely to receive social assistance benefits prior to the program. For example, the difference in social assistance recipiency rates was 5.7 percentage points in 1996 and 4.3 percentage points in 1995. Clearly there is evidence that the EBSM tended to select participants who had a much higher likelihood of needing income support in the form of social assistance than was the case for non-participants.42 Table 9.4.1 reports estimates of program impact for EBSM as a whole as well as for the five program components. Because the dependent variable is an indicator or limited dependent variable (SAR98 = 1 if individual received social assistance in 1998; SAR98 = 0 otherwise) a probit specification is employed.43 Because the parameter estimates themselves are difficult to interpret, the table reports the marginal probabilities. These can be interpreted as the estimated effect of changing from a value of 0 for the participation dummy (i.e., a non-participant) to a value of 1 for this dummy variable (i.e., a participant). Thus, for example, in the simplest specification of a probit model with no controls, the estimated marginal probability of 0.05 corresponds to an increase in the likelihood of social assistance receipt of 5 percentage points, similar to the unweighted mean difference between participants and non-participants of 5.9 percentage points (see above). Note, however, that the estimated marginal probability (difference between participants and non-participants) declines to 4 percentage points once individual and demographic factors are controlled (equation 9.4.2). Most of the estimated program impacts become essentially zero once pre-program levels of social assistance receipt are controlled. With the exception of the estimates using 1992 as the base year, the difference-in-differences estimates are close to zero in magnitude and none are significantly different from zero. Note also that the difference-in-differences estimates from the years 1993 to 1996 pass the specification test of being invariant with respect to the choice of base year. Accordingly the researchers have more confidence in these estimates than in those based on the 1992 base year or those using probit models which control for pre-program levels of social assistance receipt. Thus EBSM had no impact on social assistance receipt. In this context, it is worth noting that a simple comparison of social assistance usage in the post-program year 1998 would show the EBSM participants with a level of SA receipt more than double that of the comparison group, a difference that might be attributed in part to the program. The analysis suggests that such an attribution would be a mistake. The results in Table 9.4.1 indicate that participation in EBSM did not reduce reliance on social assistance. This conclusion also holds for most of the components. For Self-Employment, Targeted Wage Subsidies, and Employment Assistance Services none of the impacts based on longitudinal estimates is statistically significant. In the case of Job Creation Partnerships, the cross-sectional estimates using 1998 data alone (equations 9.4.1 and 9.4.2) do indicate a positive and significant effect associated with participation in Job Creation Partnerships (i.e., suggesting that participation in Job Creation Partnerships increased reliance on social assistance). However, this result disappears once pre-program levels of social assistance receipt are controlled for using the difference-indifferences methodology. These difference-in-differences estimates also pass the base year specification tests, and accordingly are preferable to the probit model estimates that control for pre-program levels of social assistance receipt. Thus Job Creation Partnerships — like most of the other EBSM components — did not increase (or decrease) reliance on social assistance. View TABLE 9.4.1 - Estimates of the Impact of EBSM and Components on Social Assistance Receipt The one component that may have resulted in an increase in social assistance receipt is Purchase of training. The longitudinal estimates of program impact are consistently positive, and range from an impact of 0.01 (i.e., one percentage point) which is not significantly different from zero to 0.08 and significantly greater than zero at the 1 percent level of significance. The difference-in-differences estimates do not pass the base year specification tests in this case. The evidence is clearly mixed, and one could conclude either that Purchase of training had no impact on social assistance receipt (i.e., using the information from the most recent pre-program years, 1994, 1995 and 1996) or that it had an impact of raising social assistance receipt by 6 to 8 percentage points (i.e., using the base year estimates from 1992 and 1993). The survey also asked respondents about their current activities, including whether or not the individual is currently on social assistance. Information regarding current activities is analyzed next. 9.5 Current ActivitiesThe survey asked participants and non-participants about their current activities, including working on a paid job, self-employed, looking for a job, upgrading one's education, in a job training program, on social assistance and on employment insurance. The responses to these questions are examined in this section. Because the information relates to post-program activities for all participants and non-participants, the analysis includes the responses for all individuals in the sample, rather than for those who completed EBSM in 1997 as in the previous sections in this chapter. Thus for this section there are 1,033 observations on EBSM participants and 499 observations on non-participants. Although being able to examine the current activities of all the survey respondents is a desirable feature of this section, an offsetting disadvantage is that there is no comparable information on pre-program activities. Specifically, the survey question on current activities refers to activities at a point in time (the date of the survey), whereas the information available for the period prior to the program relates to activities during a calendar year. Administrative data are used as much as possible in this section in order to take account of observable pre-program differences between the participants and non-participants. However, in the absence of point-in-time measures of activity levels prior to the program, difference-in-differences estimators of program impact are not possible. Accordingly, the range of model specifications that can be investigated in this section is more limited than that available in previous sections. The analysis combines the various current activities into five main groups: EMPLOYED (includes working in a paid job and self-employed), UNEMPLOYED (looking for a job), EDUCATION/TRAINING (upgrading one's education or in a job training program), ON SOCIAL ASSISTANCE, and ON EI. For the Self-Employment component the researchers also investigate the impact of the program on the likelihood of being self-employed. Each of the dependent variables is defined in indicator (or "dummy") form; that is, EMPLOYED = 1 if the individual is currently employed in a paid job or self-employed and EMPLOYED = 0 otherwise, and similarly for SELF-EMPLOYED, UNEMPLOYED, EDUCATION/TRAINING, ON SOCIAL ASSISTANCE, and ON EI. Because each of the dependent variable is an indicator or (0,1) variable, all the models use the probit specification. All the estimated impact effects have been converted to marginal probabilities for ease of interpretation. The results are presented in Tables 9.5.1 to 9.5.6. In each table, the first two equations use only the post-program cross-sectional data and the remaining equations control for pre-program activity levels using the administrative data. As discussed below, some of these pre-program controls are quite crude and thus the associated results should be treated with caution. Equation 9.5.1 indicates that, at the survey date, EBSM participants were approximately six percentage points less likely to be employed than non-participants. This difference of 6 to 7 percentage points remains when individual and demographic characteristics as well as for pre-program employment levels are controlled. The pre-program controls, however, are relatively crude. Information on gross earnings for the years 1992 to 1996 is used to construct an indicator variable for those who were employed sometime during the year (i.e., reporting positive gross earnings) versus not employed during the year (gross earnings of zero). As the results in Table 9.5.1 indicate, adding these pre-program employment indicator controls essentially leaves the cross-sectional estimates unchanged. Clearly the EBSM participants are somewhat less likely to be employed than the non-participants. These differences appear to be principally driven by the Purchase of training component, with differences of a similar magnitude but not statistically significant being found for the Job Creation Partnerships and Employment Assistance Services components.
Although the results in Table 9.5.1 are consistent across the various specifications, they do not provide convincing evidence that the program reduced the likelihood of employment among EBSM participants. The differences between the two groups could reflect unobserved factors that affect both EBSM participation and the likelihood of employment. Table 9.5.2 reports comparable results for the impact of the Self-Employment component on the likelihood of being self-employed as of the survey date. There is an enormous difference between the incidence of self-employment among Self-Employment participants and that among non-participants. This difference of 66 to 69 percentage points remains when demographic factors and for pre-program levels of self-employment activity are controlled. The administrative data reports self-employment income for each of the years 1992 to 1996, and this information is used to construct a variable indicating whether the individual was self-employed (i.e. received positive self-employment income) during the year in question. During the years 1992-1996, approximately 6 to 7 percent of those who became EBSM participants were self-employed by this measure; this self-employment rate was quite stable over the 1992-96 period. The self-employment rate among non-participants was in the 8 to 10 percent range, and was also quite stable over time. By these indicators there is no evidence that the Self-Employment component tended to select (or be selected by) individuals who were more likely to be self-employed even in the absence of the program. Indeed, the non-participants appear to be slightly more likely to be self-employed in the absence of the program than do the Self-Employment participants. Thus there is some evidence to suggest that at least some of the large differences between Self-Employment participants and non-participants can be attributed to the program. Indeed, given the pre-program incidence of self-employment among the two groups, it would not be unreasonable to conclude that much of the 66 - 69 percent differential in self-employment rates after the program can be attributed to the intervention.
Table 9.5.3 presents results on unemployment status at the time of the survey. After controlling for demographic factors, EBSM participants were about 10 percentage points more likely to be looking for a job than non-participants. This difference between participants and non-participants declines slightly (to about 9 percentage points) when pre-program levels of unemployment are controlled for using weeks on EI as a proxy for weeks of unemployment. Clearly there is no evidence that EBSM helped to reduce the incidence of unemployment among participants. Because there is no good measure of job search activity for the pre-program period, a firm conclusion from this evidence is not possible. The results suggest that if the program had any effect on unemployment it was to increase job search activity rather than decrease it. The program components which are associated with a significantly higher level of job search activity among participants than non-participants are Job Creation Partnerships, Employment Assistance Services and Purchase of training. View TABLE 9.5.3 - Estimates of the Impact of EBSM and Components on Current Status: Unemployed For the activities of "Upgrading your education" and "In a job training program" there are no proxy measures for pre-program activity. Thus Table 9.5.4 reports results using the post-program cross-sectional data alone. After controlling for individual and demographic factors, the probability that EBSM are engaged in education or training is about 4 percentage points higher than is the case for non-participants. This difference arises from participants in the Purchase of training component; for the other program components there are no significant differences between the two groups. This greater propensity to be engaged in education or training among the Purchase of training participants could be an impact of the program or could reflect unobserved factors that influence both the likelihood of being engaged in education or training and the likelihood of participating in the Purchase of training program.
For the final two activities (social assistance and employment insurance receipt) there are good measures of pre-program activity levels from the administrative data. Still difference-in-differences estimates are not possible because the pre-program information relates to social assistance and EI receipt during the calendar year whereas the post-program measure is based on the time of the survey. However, the analysis is able to control for observed differences in social assistance and EI use in the 1992 to 1996 period in addition to taking account of demographic and individual characteristics. Turning first to social assistance receipt, the EBSM participants were slightly more likely to be on social assistance at the time of the survey, and this difference (of approximately one percentage point) remains after controlling for individual characteristics. However, once pre-program levels of social assistance receipt are controlled, the differences between EBSM participants and non-participants become essentially zero and statistically insignificant. This result also holds for each of the Self-Employment, Targeted Wage Subsidies, Employment Assistance Services and Purchase of training components. However, participants in the Job Creation Partnerships component remain about two percentage points more likely to be on social assistance than non-participants after taking account of pre-program levels of social assistance receipt. In summary, there is no evidence from the information on activities at the time of the survey that EBSM reduced the reliance on social assistance. This finding supports the conclusion reached in section 9.4 which was based on social assistance receipt during the calendar year 1998. The results for EBSM as a whole also hold for the Self-Employment, Targeted Wage Subsidies, Employment Assistance Services and Purchase of training components. For the Job Creation Partnerships component the program may have increased the likelihood of being on social assistance by a modest amount (raising the probability by approximately 0.02). This result is not consistent with that found earlier (in section 9.4) where it was concluded that the Job Creation Partnerships component — as was the case for other program components — did not alter the likelihood of relying on social assistance. Thus there is mixed evidence on this issue. Nonetheless it should be noted that the differences in findings are not large, with the results in section 9.4 indicating a zero impact on social assistance receipt and those in this section indicating a small positive impact. The difference in results is probably attributable to the differences in pre-program information, with that in section 9.4 being better than that available for the analysis in this section.
Although EBSM does not appear to have had any impact on social assistance receipt, the results in Table 9.5.6 indicate that the program may have increased reliance on EI. The differences in EI receipt as of the date of the survey between participants and non-participants are moderately large (about 10 percentage points) and these decline only slightly after controlling for individual characteristics and pre-program levels of EI incidence. This result is being driven principally by the Purchase of training component, with the estimated impacts of the other components being small and not statistically significant. Thus the Purchase of training component appears to have increased EI incidence by approximately 12 percentage points, while the other components had no significant impact on reliance on EI. The point-in-time estimates are at odds with the longitudinal findings that EBSM may have modestly reduced EI dependency (section 9.3). Estimates based on the longitudinal data are preferred because they control for preprogram differences.
9.6 HighlightsIt should be noted that the impact results presented here are preliminary only and that more definitive results will be presented in a summative evaluation when more time after program completion will have elapsed. The impact analysis showed the following for those participants who completed the program in 1997:
Lack of pre-program data make the following conclusions less robust because the differences between the two groups could reflect unobserved factors that affect both EBSM participation and the outcome:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[To Top]](/web/20060127000430im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}
![[Previous Page]](/web/20060127000430im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127000430im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127000430im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}