|
|
     |
||
 |
     |
 |
|
This chapter presents our econometric analysis of the impacts of the Compass program. Because there are several outcomes of interest, the chapter proceeds in stages, examining outcome measures in the same sequence as the previous chapter. For most of the outcomes we obtain three types of estimates of program impact. The first are simple linear regression models with and without controls for observable factors. Explanatory variables used in these regressions are as follows: (i) basic demographic and personal characteristics: age, educational attainment, gender, and marital status; (ii) additional demographic and personal characteristics: visible minority status, presence of children 0 to 5 years, 6 to 11 years, 12 to 17 years, and 18 years and over, and need for child care; (iii) pre-program information on labour market activities, using the fraction of time devoted to the activity in 1992 and in 1993. These linear regression models thus control for observable factors, including pre-program levels of the outcomes in question, which may account for differences between the behavior of participants and non-participants, as well as observable factors which may account for non-random selection into the Compass program. The additional two types of estimators reported in this chapter are based on alternative models of non-random selection into the program which depend on unobservable factors. One approach uses instrumental variable estimators to account for the possible endogeneity of program participation due to non-random selection. The instruments used are age, presence of children under 5 years and receipt of social assistance in the two years prior to Compass. A prior investigation of program participation indicated that these variables are significant determinants of participation in Compass. Whether or not these are appropriate instruments, however, is questionable. The method of instrumental variables will produce consistent estimates of program impact if the chosen instruments are correlated with participation in the program but uncorrelated with the outcome variable of interest. In general (and specifically in this evaluation) it is difficult to find variables that meet these requirements. The second approach uses longitudinal "difference-in-differences" estimators which take advantage of the fact that the survey obtained information on time spent on these activities by each respondent in 1992 and 1993 (prior to the introduction of Compass) as well as 1996. These estimators provide unbiased estimates of the impact of the program if selection into the program is correlated with unobserved person-specific factors that also influence the time spent on these activities by individuals. 7.1 Recent Labour Market History This section analyzes the relationship between participating in Compass and the time spent in three main activities: work, in school and unemployment. The survey of participants and non-participants asked about time devoted during each of the years 1992 to 1996 to the following labour market activities: in school and not working; in school and working; working and not in school; and not working and not in school. Because only a small proportion of time is devoted to "in school and working," we have combined these four activities into three: working (which includes both working/not in school and working/in school), in school (which is restricted to in school and not working), and unemployed (not working and not in school). None of the findings are altered by treating the "in school and working" activity separately, or combining this category with "in school and not working" rather than the combination used here. The focus of our analysis is the time devoted to these activities in 1996. Because the survey was carried out in October and November, the number of months devoted to all four activities was not the same for all survey respondents; to account for these differences, we analyze the fraction of the total period devoted to each activity. This method also takes account of the (small) number of cases in which the total number of months in the years 1992 to 1995 devoted to the four activities did not sum to 12. For many survey respondents, the time devoted to these activities in 1996 combines the effects of participation in Compass during 1996 with any post-program impacts of Compass. In order to obtain an estimate of the impact of Compass which is not contaminated in this way, we analyze the subset of individuals who completed or were referred to the program prior to 1996. The estimates for this group thus provide evidence on the impacts of Compass on labour market activities during the year (or in a small number of cases more than one year) after program completion. Table 7.1.1 reports the estimated program impacts on the three activities based on the alternative specifications. We report the estimates from a variety of specifications in order to illustrate the sensitivity (or lack of sensitivity) of the findings to alternative specification choices. All estimates use only the subset of the sample of participants who completed the program before 1996 and the non-participants who were referred to Compass before 1996. As noted in the previous chapter, prior to Compass non-participants spent about 4 to 6 percent more of the year working than did those who subsequently became Compass participants. However, in both 1995 and 1996 Compass participants spent a greater fraction of the year working than did non-participants; this difference was especially large (about 21 percentage points) in 1996. This suggests that the program may have had a positive effect on the proportion of time spent working. The estimates in Table 7.1.1 are generally consistent with this expectation. The estimates which control only for observable differences between participants and non-participants indicate that 12 to 14 percent more of the year is spent working in the year following participation in the program. The IV estimates are also positive, but very imprecisely estimated and not significantly different from zero. This imprecision is not unexpected, and is probably due to the difficulty of separately identifying the influences on participation in the program from the influences on the outcomes associated with program participation. For this reason, we do not regard the IV estimates as being credible estimates of program impact. The difference-in-differences estimators suggest larger impacts - in the range of 16 to 27 percent, depending on the choice of base year. Although this is a wide range, the estimates are not in fact significantly different from each other (at the 5% level of significance). Nor do the difference-in-differences estimates differ significantly from those obtained from linear regression with various choices of controls. Thus the credible estimates do indicate that the program increased the proportion of time spent working, at least in the short run, by more than 10 percent. Prior to Compass, those who became participants spent 8 to 10% more of their time in school than did those who became non-participants. By 1996, differences between participants and non-participants in this dimension had essentially disappeared. However, the estimates reported in Table 7.1.1 differ in their assessment of the impact of the program. The longitudinal difference-in-differences estimates suggest that the program reduced the time spent in school by 9 to 10 percentage points. The assumptions under which the difference-in-differences estimators provide unbiased estimates of program impact imply that the estimated impacts should not be sensitive to the choice of base year; the estimates in Table 7.1.1 pass this specification test, increasing our confidence in these estimates. The other specifications also yield a negative impact on time spent in school, but these estimates are not statistically significantly different from zero. None of the specifications suggest that the program increased time spent in school. Table 7.1.1 Estimates of the Impact of Compass on Time Spent Working, In School and Unemployed
Turning to the impacts on time spent neither working nor in school, the linear regression models indicate a reduction in time spent unemployed in the range of 10 to 12 percentage points; these estimated impacts are not sensitive to the set of variables included to control for other influences. The IV estimates are again very imprecise. In this case the longitudinal estimates are sensitive to the choice of base year, suggesting no significant impact when 1992 is used as the base year but a substantial impact when 1993 is used for that purpose. Because the difference-in-differences estimates do not pass the specification test of invariance to the choice of base year, the linear regression model estimates appear the most credible. None of the specifications suggest that Compass had the effect of increasing time spent unemployed. In summary, the evidence reported in this section indicates that the Compass program tended to increase participants' proportion of time spent working, decrease their time spent unemployed, and decrease or leave unchanged their time spent in school. These conclusions apply to the short run impacts of the program, approximately the year following completion of/referral to Compass. Table 7.1.2 reports estimated impacts by program option for the WEO and TTO options. (There are not sufficient observations to obtain estimates for the EDO option.) For the TTO option, there is clear evidence of a positive impact on time spent working and a negative impact on time spent unemployed. Based on the specification with full demographic and pre-program controls, these estimated impacts are modestly larger in magnitude (although not statistically significantly different from) than those for the full sample, as shown in Table 7.1.1. For the WEO option, the evidence of a positive impact on working time and negative impact on unemployment is weaker. Based on the specification with full demographic and pre-program controls, the magnitudes of the estimated impacts on time spent working and unemployed are smaller than their counterparts for the TTO option, and not significantly different from zero. However, the much lower precision of the WEO estimates could be due to the fact that the sample sizes are about half those available for the TTO option. Table 7.1.2 Estimates of the Impact of Compass on Time Spent Working, In School and Unemployed by Program Option
7.2 Use of Social Assistance A primary objective of the Compass program was to reduce reliance on social assistance. In this section we examine the impacts of the program on social assistance receipt using information obtained from the survey of participants and non-participants which asked respondents about the extent of their use of social assistance over the period 1993 to 1996. Further analysis of the impacts of the program on social assistance receipt is also carried out in a later section (see section 7.4) which uses information on current activities. Two measures of reliance on social assistance are analyzed in this section. The first and simpler is based on whether or not the individual received social assistance (SA) sometime during the year. The second is based on the number of months of SA receipt during the year. SA Receipt During the Year Table 7.2.1(a) reports the proportion of Compass participants and non-participants that received SA sometime during the year, for the years 1993 to 1996. The first part of the table shows the breakdown for all participants and non-participants. In 1993 a greater fraction of non-participants received SA than was the case for participants; however, in both 1994 and 1995 the proportions of participants and non-participants on SA were almost identical. In 1996, the fraction of non-participants receiving SA rose relative to the previous year, while that of participants fell, resulting in a difference of 16 percentage points (.67 versus .83). However, it would be inappropriate to regard this difference as an estimate of the impact of the program on behavior for two reasons. First, many Compass participants were still enrolled in 1996, and for these individuals the lower incidence of SA may simply reflect the fact that receipt of SA is less likely while participating in Compass. Second, this difference does not control for other factors that may influence SA receipt. To account for the enrolment effect, the second part of the table shows the proportions receiving SA during the year for those participants who completed the Compass program prior to 1996 and those non-participants referred to Compass prior to 1996. Note that for this group, there are not significant differences in SA receipt in 1993 (i.e. prior to the introduction of Compass) or 1994. However, a significant difference emerges in 1995, the year in which most participants were enrolled in Compass, and this differential widens further in 1996, after the completion of the program. The differential of 23 percentage points suggests that Compass may have reduced reliance on SA. However, this estimate does not control for other factors that may influence participation in Compass and SA receipt. 7.2.1 (a) Proportion of Compass Participants and Non-participants on Social Assistance
7.2.1 (b) Months on Social Assistance During the Year
Table 7.2.2 reports a variety of estimates of the impact of the Compass program on the likelihood of receiving SA during the year. In order to isolate the possible effects of being enrolled in the program from the impacts of the program on behavior, all estimates are based on the subset of the sample who completed or were referred to Compass prior to 1996. The estimates fall into three categories. The first group (linear regression models) are appropriate estimates of program impact if selection into the program is random or is non-random and depends on variables which are observable and therefore can be directly controlled for in the statistical analysis. The second group (longitudinal fixed effects estimators, or difference-in-differences estimates) are appropriate if selection into the Compass program depends on person-specific factors which are unobservable (to the researcher) but which are constant over time; the influence of these unobserved "fixed effects" is removed by taking differences between pre-program and post-program observations on participants and non-participants. The third group (instrumental variables estimators) are appropriate if participation in the program and the impacts of the program are jointly determined by various observable and unobservable factors. The estimated impact with no controls (equation 7.2.1) corresponds to the mean difference in SA receipt between participants and non-participants in the post-program period (1996). This would be an unbiased estimate of the impact of Compass if selection into the program had been randomly determined. Adding the basic controls for age, gender, marital status and education (equation 7.2.2) has little effect on the estimated impact, and none of these controls exert a statistically significant influence on SA receipt (only marital status borders on significance). However, controlling for prior use of SA (receipt of SA in 1993, prior to the introduction of Compass) raises the estimated impact from .23 to .30. Almost identical results (not reported) are obtained when one controls for prior use of SA in both 1993 and 1994. Including in the regression equation both prior use of SA and the basic controls produces an estimate of .29 while including additional controls for visible minority status, presence of children aged 0 to 2, 2 to 5, 6 to 11, 12 to 17 and 18 or above, and need for child care lowers the estimated impact modestly to .26. Thus these estimates fall into a range of .23 to .30. The longitudinal "difference-in-differences" estimators also fall in this range, giving an estimated impact of .22 when 1993 is used as the base (pre-program) year and .26 when 1994 is used as the base year. The use of 1994 as a pre-program year may result in some bias (toward producing a larger estimated impact) because some of those in the participant group entered the program in that year. For this reason, the 1993 base year estimate is favoured, although the two estimates do not differ significantly from each other. A specification test of the "fixed effects" model is that the estimates should not differ significantly according to the base year chosen; these estimates thus pass this specification test. The instrumental variables estimates allow for the possible endogeneity of program participation by jointly modelling SA receipt and program participation in a two equation simultaneous equations model. A number of such models were estimated; the reported estimates (equation 7.2.6) includes as instrumental variables age, presence of children under 5 years of age, and pre-program use of SA as measured by the baseline survey. This model, as well as variants of this model, produces estimates of program impact which are larger but very imprecise. Thus these estimates do not differ significantly from those in the .22 to .30 range, the range of the other estimates in Table 7.2.2. We conclude that there is some evidence that the Compass program reduced reliance on social assistance by 22 to 30 percentage points, thus reducing the proportion of individuals in this population receiving SA in 1996 from about 73% to about 50%. This estimated impact is short run in nature, applying in most cases to the first year after completion of Compass. Table 7.2.2 Estimates of Compass Impact on Social Assistance Receipt During the Year
Months on SA during the year Table 7.2.1(b) reports months of social assistance receipt over the 1993-1996 period for two groups: (i) all participants and non-participants, and (ii) participants who completed Compass prior to 1996 and non-participants referred to Compass prior to 1996. As discussed previously, the second group is of principal interest for estimating the impacts of the program, and is used for all the estimated impacts discussed below. Participants completing Compass before 1996 had slightly fewer months of SA receipt than comparable non-participants in both 1993 and 1994, although these differences were not statistically significant. However, in both 1995 and 1996 Compass participants had significantly fewer months on SA than comparable non-participants. Table 7.2.3 reports various estimates of program impact, corresponding to alternative assumptions about the factors affecting months of SA receipt and the factors affecting participation or non-participation in the program. The simplest estimate of -2.5 months (equation 7.2.9, no controls case) corresponds to the mean difference in months of SA receipt between participants and non-participants. As discussed previously, this estimate would be unbiased if assignment to the program had been randomly determined, in which case there would be no need to estimate more complex models. Controlling for a variety of observable influences, including the extent of SA receipt prior to the program, produces estimates which range from -2.1 months to -2.8 months. These estimates are all significantly different from zero (indicating that the program had a significant impact, according to these estimated equations) and do not differ significantly from each other. The difference-in-differences estimates, which as discussed previously are appropriate if selection into the program is based on unobserved person-specific factors which are constant over time, produce very similar estimates of the impact of the Compass program on months of SA receipt. These estimates also do not differ significantly from each other, and thus are (in a statistical sense) invariant to the choice of base year. The instrumental variables models produced much larger estimates of program impact. These estimates were also found to be quite sensitive to the choice of instruments for participation in the program. Thus these estimates do not appear sufficiently robust to be regarded as credible estimates of the impact of Compass. Note, however, that because the IV estimates were always larger than the other estimates reported in Table 7.2.3, these estimates do not contradict the conclusion that the program appears to have had a significant effect on reliance on SA, at least in the short run. In summary, there is evidence that in the short run the program reduced the number of months of SA receipt during the year by 2.1 to 2.8 months. Table 7.2.3 Estimates of Compass Impact on Months of Social Assistance Receipt During the Year
To conclude this section, we report in Table 7.2.4 the estimated impacts by program option for three of the models estimated in Tables 7.2.2 and 7.2.3. (The number of observations for the EDO option is too small to permit controlling for other influences on social assistance receipt.) The estimates for the TTO option are larger than those for WEO, although the TTO and WEO impacts are generally not significantly different from each other. The magnitudes of the estimated impacts for EDO (without controls) are similar to those to the other two options, but very imprecisely estimated due to the small number of observations on this option. Table 7.2.4 Estimates of Compass Impact on Social Assistance by Program Option
7.3 Use of UI This section examines the impact of the program on use of the UI program. The analysis is very similar to that of SA receipt in the previous section, with the following differences: (i) information on UI receipt is based on administrative data rather than survey data, (ii) four years of pre-program information is available, 1990-93 inclusive, and (iii) for 1996, the only year for which post-program data are available, information is available for the first six months of the year. Table 7.3.1 presents summary statistics on weeks of UI receipt during the year for two groups: all participants and non-participants, and those participants/non-participants who completed the Compass program/were referred to Compass prior to 1996. Analysis of the impact of Compass on UI receipt is restricted to the latter group. As indicated in Table 7.3.1, for those who completed/were referred to Compass prior to 1996, the differences between participants and non-participants were generally small in the pre-program period (1990-93), and in none of the years was the difference statistically significantly different from zero. For the full sample, differences between participants and non-participants in UI receipt were also small. During the program, weeks of UI receipt fell more for participants than non-participants, while in 1996 a positive differential in UI weeks opened up. This greater use of UI by participants compared to non-participants is especially pronounced for the group who completed/were referred to Compass before 1996. The substantial gap in 1996 suggests that Compass may have increased UI receipt among participants. Table 7.3.1 Weeks of UI Receipt During the Year, 1990-1996
Table 7.3.2 presents estimates of program impact based on various models which correspond closely to the models used to analyze the impact on SA receipt. These models produce a range of estimates from 3.9 to 5.7 weeks for the first six months of 1996. The estimates which control for selection into the program based on unobservable factors (the difference-in-differences and IV estimates) fall in the range 3.9 to 5.4 weeks, while the estimates which do not control for selection into the program or control for selection based on observable factors alone produce estimates in a slightly higher range (4.5 to 5.7 weeks). Nonetheless, not one of these estimates is significantly different from any of the others, so all point to the conclusion that the Compass program significantly increased UI receipt among program participants. Based on the first six months of 1996, the estimated impact is 7.8 to 11.4 weeks on an annual basis (assuming that the first six months of the year is representative of the entire year). Table 7.3.2 Estimates of the Impact of Compass on Weeks of UI Receipt
When the analysis of UI impact is carried out separately for each program option (see Table 7.3.3), the largest estimated impacts are those associated with the TTO option. These are in the 6.0 to 6.7 weeks range, equivalent to 12 to 13.4 weeks on an annual basis. Table 7.3.3 Estimates Impact on Weeks of UI Receipt by Program Option
7.4 Current Situation (at the point of the survey) The survey obtained information on the current activities of participants and non-participants. This information is analyzed in this section under four principal types of activities: (i) employment (both paid employment and self-employment) (ii) unemployment (searching for work) (iii) upgrading skills through education or training, and (iv) receiving income assistance in the form of social assistance (SA) or unemployment insurance (UI). In addition to examining the impacts of Compass on these four principal activities, we also analyze the impacts on the component activities: paid employment and self-employment in the case of employment, education and training in the case of skills upgrading, and SA and UI in the case of income support. Because of the number of activities analyzed and the associated large volume of estimated impacts, only the principal findings are reported in the tables in this section. Table 7.4.1 reports estimates of the impact of Compass on employment, as well as separate estimates of the impact on self-employment (for the EDO component) and paid employment. The first three columns of estimates use the full sample of survey respondents, and the second three columns report estimates for those who completed or were referred to Compass prior to 1996. These latter estimates thus provide evidence on possible program impacts 10 months or more after completion of the program. For the full sample, the estimated impacts based on linear regression models indicate that the program increased the proportion of participants engaged in employment by 19 percentage points, whether or not one controls for basic demographic factors (age, gender, educational attainment, marital status) and additional personal characteristics (visible minority status, presence of children aged 0 to 5, 6 to 17, and 18 and over, and need for child care). Instrumental variable estimates of program impact were also obtained; these attempt to take account of the possible endogeneity of participation in Compass associated with non-random selection of participants and non-participants. These IV estimates were generally very imprecise (note the large standard errors associated with these estimates) and also unstable (in particular, sensitive to the choice of instruments for program participation). For these reasons we will focus on the linear regression model estimates in what follows. When attention is limited to those who completed or were referred to Compass prior to 1996, the estimated program impacts are only slightly smaller - in the range of 16 to 18 percentage points. With the smaller sample size, these estimates are also somewhat less precise, but nonetheless are statistically significant at the 1% level. The breakdown between self-employment and paid employment suggests that the effect of Compass was on paid employment; the estimated impacts on self-employment (from EDO) are small and not significantly different from zero. In summary, the evidence suggests that participation in Compass had the effect of increasing the proportion of the target population who were engaged in paid employment. This effect seems to persist for at least 10 months following completion of the program. These estimated effects are not sensitive to controlling for observable factors that may differ between participants and non-participants because of the non-random selection into the program. The principal qualification to this conclusion is that we have not been able to obtain credible estimates of program impact which account for possible unobserved factors which could result in selection bias (i.e. unobserved factors which may be related to both participation in Compass and to employment status following Compass). The estimates which attempt to take into account these unobserved factors using the available (post-program) data are too imprecise and unstable to regard as plausible estimates of program impact. Table 7.4.1 Estimates Impact of Compass on Employment
A similar analysis was carried out on the following three activities: looking for work, upgrading education, and undertaking training. The analysis was also carried out on the combined activity of upgrading skills (education and/or training). No significant differences between Compass participants and non-participants were found for these activities. Table 7.4.2 reports the main findings of the analysis of the impacts on reliance on income support programs. This examination of current activities thus complements the previous analyzes of SA and UI receipt discussed in sections 7.2 and 7.3 respectively. In contrast to these previous analyzes which employed administrative data on participants and non-participants before and after the program, the assessment in this section is based on the survey of participants and non-participants, and thus is restricted to cross-sectional data on post-program activities. For the full sample of survey respondents, Compass participation is associated with a decrease in reliance on SA and an increase in use of UI. The magnitudes of the estimated impacts are affected only modestly by controlling for observable differences between participants and non-participants. The estimated reduction in the proportion using SA of 22 percentage points is larger than the estimated increase in the proportion using UI of 12 to 14 percentage points, thus indicating that the net impact is reduced reliance on income support of about 10 percentage points. The IV estimates suggest much larger impacts, but the precision of these estimates is also much lower. These estimated impacts could reflect, in part, the fact that Compass placement provided participants with sufficient work experience to qualify them for UI. As a consequence, participants would be more likely to be receiving UI rather than SA immediately after the program. This effect could be transitory in nature, and would thus not necessarily represent a lasting effect of the program. In order to examine this possibility, we also report in Table 7.4.2 the estimated effects for those who completed or were referred to Compass prior to 1996. The estimated impacts on SA receipt are now lower, in the 12 to 15 percentage point range versus 22 percent for the full sample. Nonetheless, these estimated impacts remain statistically significant. The UI impacts are also lower for this group, and are generally not significantly different from zero. The combined impact is now estimated to be a reduction in reliance on income support of 7 to 8 percentage points, but this effect is only marginally significant (significant at the 15% but not the 10% level of significance). Table 7.4.2 Estimates Impact of Compass on Use of Income Support
In summary, this analysis of current activities suggests that Compass participation makes SA receipt less likely and UI receipt more likely in the very short run, the period immediately following the program. Because the reduction in the proportion receiving SA is larger than the increase in the proportion receiving UI, the net effect is a reduction in reliance on income support. However, these effects dissipate during the period following the program. As a consequence, if one examines participants and non-participants approximately 10 months or more after program completion, the estimated (negative) impact on SA receipt is lower (but still significantly different from zero), the positive impact on UI receipt is also lower (and no longer significantly different from zero), and the combined effect is also lower and in the same direction as before (i.e. reduced reliance on income support) but not significantly different from zero. In order to provide additional insights into the impacts of Compass on current activities, we also carried out the analysis by program option. Table 7.4.3 reports the principal findings. Again, results are reported separately for the full sample of survey respondents and for those who completed or were referred to Compass prior to 1996. The latter group provides some evidence on possible short term (as opposed to very short term) effects of the intervention. The full sample evidence suggests that both WEO and TTO had similar impacts on the proportion of the population employed, in the range of 18 to 20 percentage points. These estimates are significantly different from zero at the 1% level. However, when the analysis is restricted to those who completed Compass 10 or more months prior to the survey, the estimated impacts of the WEO option are much larger - 26 to 29 percentage points - than those of the TTO option - 13 percentage points. These findings suggest that the WEO option has more lasting effects on the likelihood of employment than is the case with the TTO option. In both the very short run (i.e. for the full sample of survey respondents) and the short run (those who completed Compass 10 or more months prior to the survey), the TTO option has a larger negative impact on SA receipt and a larger positive effect on UI receipt than is the case with the WEO option. However, for the full sample the net effect (which is in the direction of reduced reliance on income support) is identical at 10 percentage points. For those surveyed 10 or more months after Compass, the net effect is also in the direction of reduced reliance on income support, and is somewhat larger for WEO than for TTO; however, for both options the net effect is not statistically significant. Nonetheless, both options significantly reduce SA receipt in the short run, with the estimated effect being somewhat larger for the WEO option than for TTO. Table 7.4.3 Estimates Impacts on Current Activities by Program Option (a) Estimates based on full sample of survey respondents
Table 7.4.3 Estimates Impacts on Current Activities by Program Option (b) Estimates based on respondents completing or referred to Compass before 1996
The estimated effects of the EDO option are based on very small samples (55 observations for the full sample of respondents, and 13 observations for those completing prior to 1996). Thus the precision of the estimates is low. Nonetheless, several features of these impact estimates are noteworthy. First, the magnitudes of the estimated effects on employment are similar to those for the WEO and TTO options, albeit not statistically significant. Second, this option is estimated to reduce reliance on both SA and UI, unlike the WEO and TTO options which reduced SA use but had the opposite effect on UI use. This finding accords with expectations in that self-employment does not (in general) qualify individuals for UI, in contrast to the case of paid employment. As a consequence of the similar (in magnitude) impact on SA receipt and the negative impact on UI receipt, the net effects on reliance on income support are largest for this EDO option, and remarkably (in spite of the small sample sizes) are significantly different from zero. In summary, the main findings obtained from the analysis of program options are the following: (i) in the short run, WEO had a larger impact on increased employment than did TTO and EDO, though all options tended to increase employment (albeit the estimated impacts are not always significantly greater than zero); (ii) both WEO and TTO reduced reliance on SA in the short run with the estimated effect being slightly larger for WEO; (iii) both WEO and TTO resulted in increased use of UI in the short run, with TTO having a larger effect on increased UI use and the WEO effect not being significantly different from zero (this is partly a consequence of the longer work period subsidized under TTO, which qualified a greater percentage for UI); (iv) the net effect of WEO and TTO on reliance on income is estimated as being in the direction of reduced reliance on income support in the short run, is larger for WEO than for TTO, but in both cases is not significantly different from zero; (v) the sample sizes for EDO are extremely small, and thus the estimated impacts are imprecise; nonetheless, the estimates do suggest that EDO had a larger positive impact on employment than did WEO and TTO, and a larger negative impact on reliance on income support than did WEO and TTO; this latter difference arises principally because EDO reduces both SA and UI receipt. 7.5 Earnings This section analyzes the impact of Compass on earnings, using the information on annual employment earnings which is available from HRDC UI files up to 1995 and from the client survey for 1996. Table 7.5.1 reports the average annual employment earnings of Compass participants and non-participants, together with the difference in the average earnings of these two groups, over the 1990-1996 period. These data indicate that those who became Compass participants had significantly lower employment earnings prior to Compass (i.e. during the 1990 to 1994 period) than those who became non-participants. During 1995 and 1996, when some participants were enrolled in the program and others had completed their Compass placement, participants had significantly higher employment earnings, on average, than non-participants. Table 7.5.1 Annual Earnings, 1990-1996
In order to separate the earnings behavior of the Compass placement itself from the possible impact of the program on subsequent earnings, Table 7.5.1 also reports average employment earnings of participants who completed their Compass placement prior to 1996 and non-participants who were referred to Compass prior to 1996. For this group, the difference in 1996 earnings provides some indication of the possible impact of the program on short term labour market earnings. As was the case for the full group of participants and non-participants, earnings of those who became Compass participants were significantly lower than those who became non-participants prior to the Compass program, and significantly exceeded the earnings of non-participants during 1995, when most participants were enrolled in Compass. During 1996, when participants had completed their Compass placement, earnings of participants remained above those of non-participants although the difference in average earnings is not statistically significant. Because participants began and completed their Compass placement in different years, we have also tabulated the employment earnings of participants and non-participants according to the years prior to, during, and after Compass. (For non-participants these correspond to years prior to, during and after referral to Compass.) These data are shown in Table 7.5.2. Again, those who became participants had significantly lower earnings prior to Compass, significantly higher earnings during Compass, and earnings which did not differ significantly from those of non-participants after Compass. Although these data suggest that Compass may have had an impact on employment earnings of participants, econometric analysis does not support this conclusion, even when controlling for earnings prior to Compass. Table 7.5.3 reports the estimates obtained from a variety of models of earnings using the data on earnings before, during and after Compass (i.e. the data reported in Table 7.5.2). Analysis of annual earnings on a calendar year basis (the data reported in Table 7.5.1) yields the same conclusion. Table 7.5.2 Annual Earnings Before and After Compass Participation/Referral
Most of the estimated impacts reported in Table 7.5.3 are positive; furthermore, as expected from the behavior of the raw data (which show that earnings of Compass participants were significantly lower than those of non-participants prior to the program), controlling for prior earnings tends to raise the estimated impact. However, none of these estimates is significantly different from zero. The inability to obtain more precise estimates of the impact of the program on earnings derives principally from the fact that there are relatively few observations on earnings of both participants and non-participants in 1996, and thus little information on post-program earnings. In addition, the estimates which require pre-program information (i.e. those that control for earnings prior to Compass, and the longitudinal difference-in-differences estimates) are very imprecise because there are only small number of individuals for whom information is available on their earnings before and after Compass. Table 7.5.3 Estimates of the Impact of Compass on Earnings
In summary, although the average behavior of the participants and non-participants suggests that Compass may have had a positive short run impact on earnings, there is not sufficient data available on post-program earnings to be able to reach this conclusion. 7.6 Attitude Change Both the baseline survey and the post-program survey asked Compass participants and non-participants about their attitudes towards life, work and education and training. This section examines the changes which occurred in these measured attitudes over the period of the program, and whether there were any differences between participants and non-participants in these changes in attitudes. Table 7.6.1 summarizes the analysis of two types of attitude change. The first set of questions asked about the level of general satisfaction with such factors as: social life, family life, education received, work done, and life in general. The second set of questions asked respondents: "In your opinion how likely is it that: (a) in the longer term you will maintain steady employment; and (b) in the longer term you will be on social assistance. Both sets of questions used a 5 point scale, with 1 being "extremely dissatisfied" and 5 being "extremely satisfied" for the questions on level of satisfaction and 1 being "not likely" and 5 being "very likely" for the questions on longer term outlook. From the responses to the baseline and post-program surveys we have constructed measures of attitude change by taking the difference between the level of satisfaction after Compass and the level of satisfaction registered prior to Compass. This measure of increased or decreased satisfaction is available for each individual who responded to both surveys. The highest possible score is thus 4, corresponding to someone who was "extremely dissatisfied" at the time of the baseline survey and "extremely satisfied" after the program. Similarly, the lowest possible score is -4. Examination of these measures of attitude change indicate that the range of values is generally from -4 to +4, the average values are generally positive for both participants and non-participants (one exception is the measure of satisfaction with "the work you have done in your life" which is modestly less than zero for non-participants), and the average values are generally small, indicating that for the participant and non-participant groups overall, there was little improvement or worsening in attitudes over the period. We have constructed similar measures of changes in long term outlook, so that a positive value corresponds to someone who believes that maintaining steady employment in the longer term has become more likely. A positive value for social assistance outlook corresponds to someone who thinks it has become more likely that they will be on SA in the longer term. Again, these measures have a possible range from -4 to +4. Examination of the data indicates that views about the change in the long term do in fact range from -4 to +4 in the data, so that substantial changes in both directions have occurred for some individuals. However, the average changes have been small, with outlooks about maintaining steady employment in the long term improving slightly for participants and worsening modestly for non-participants. The likelihood of being on social assistance in the longer term increases, on average, for both participants and non-participants, but the magnitude of the increase is somewhat larger for non-participants. The estimates reported in Table 7.6.1 indicate that there is not a significant difference between participants and non-participants in the change in the level of satisfaction toward social life with friends and relatives, education received, and life in general. This finding holds whether or not one controls for various factors that could account for differences between Compass participants and non-participants. However, there was a modest decline in participants' satisfaction with family life (relative to the change for non-participants) and a modest increase in participants' satisfaction with "the work you have done in your life". These estimated effects associated with Compass participation are approximately equal in size, the decline in satisfaction with family life being about .20 (i.e. about 1/50 of 1 point on a 5 point scale) and the increased satisfaction with work accomplished being approximately equal in magnitude in the other direction. The changes in the long term outlook do not differ significantly between participants and non-participants. Thus it appears that the program had no discernible impact on individuals' beliefs about the likelihood of maintaining steady employment or being on social assistance in the longer term. Table 7.6.1 Analysis of the Impact of Compass on Attitudes and Beliefs
7.7 Opportunity Fund Concerning the impact of the Opportunity Fund, a number of models of employment, earnings and SA receipt (months) were estimated. In all three analyses, attention was confined to those referred/completing prior to 1996 so as not to confound program impacts and effects of ongoing placement. There were no significant effects for all three analyses, with or without controls46. The absence of any significant effects is not surprising, as the sample sizes are quite small. There were only 176 participants who were also opportunity funds recipients, and only 22 opportunity fund recipients who were not participants. When we narrow down to those who completed prior to 1996 and to those who responded to the survey (needed for some of the analyses), there are very few observations. To conclude, the statistical analysis finds no evidence of significant impacts of the opportunity fund, but this should not be taken to imply that there were no impacts. Rather there are too few observations on post-program outcomes to be able to identify whatever impacts there may have been. 7.8 Conclusion At least in the short run, Nova Scotia Compass has been successful in reaching its primary objective: to reduce reliance on social assistance. Use of social assistance fell by over 20% for participants as compared to non-participants. Partially offsetting this was an increase of around 15% in the use of UI, brought about in part because program participation helped clients qualify for UI. The net impact is reduced reliance on income support of about 10 percentage points. There is some indication this effect diminishes with time, however. Compass also led to an increased proportion of participants engaged in employment, on the order of 18 percentage points. No significant impact on earnings was uncovered, though this is likely because there were too few cases with available post-program data.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[To Top]](/web/20060127010543im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}
![[Previous Page]](/web/20060127010543im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127010543im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127010543im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}