|
|
     |
||
 |
     |
 |
|
Evidence is presented in this chapter on several measures of EBSM success to date based on multiple lines of evidence, including focus groups with participants and employers, key informant interviews, administrative data, and surveys of participants and non-participants. First, we consider the degree to which EBSMs have met accountability targets set out in the Canada/Prince Edward Island Labour Market Development Agreement (LMDA). Second, the results of descriptive analysis of EBSM outcomes are then presented. Finally, some of the EBSM outcomes revealed through the descriptive analysis could be explained by pre-existing differences in the characteristics of participants and non-participants and not necessarily by the EBSMs themselves, we present the results of multivariate analysis which is able to control for these differences. Though early in the process, it is important to monitor and assess how the EBSMs are working and provide early warning signals if expectations are not being met. This information will allow LMDA management to make changes to the EBSMs that would facilitate their ultimate success. It is important to note, however, that EBSM participation may not result immediately in jobs, but may be expected to lead to employment and other outcomes in the future, beyond the scope of the current evaluation. The future summative evaluation will provide the opportunity to identify incremental impacts and results. The findings follow:
6.1 Attainment of Results TargetsTo meet the objective of accountability for the EBSMs, the Canada/PEI LMDA, like all LMDAs with provinces and territories, specified results targets in three areas: active EI claimants served, returns to work, and unpaid EI benefits. As set out in the Agreement, these primary result measures are to be monitored and adjusted annually based on the past year's performance and changing circumstances. The purpose of setting the targets and checking to see if they are being met each year is to determine the extent to which the EBSMs are successful in meeting basic objectives of integrating EI clients into the workforce. Divergence from targets could signal the need for adjusting the EBSMs to better meet the needs of clients. A component of this evaluation is to consider the extent to which the accountability targets are being met.33 In this section, the results of the assessment exercise are summarized, while Appendix C presents a detailed discussion of the methodology and the results of the analysis. For the number of EBSM participants returning to work target, computations based on the administrative data indicate that the 1998/99 return to work target (2,000) was exceeded by almost 50 percent, reaching a level of 2,902 returns to work. In 1998/99, there were over $4.6 million in unpaid EI benefits, exceeding the target ($4 million) by over 12 percent. It is noteworthy that between 1997/98 and 1998/99, the return-to-work target was increased while the unpaid benefits target was reduced. These changes likely partly due to the fact that, in the previous fiscal year, returns to work exceeded the target while unpaid EI benefits were well short of it. This change in targets indicates a principle of the Canada/PEI LMDA is being followed, whereby these targets are to be adjusted each year to reflect changing circumstances, particularly as experience and familiarity grow with time. The latter might also be reflected in the fact that returns to work rose considerably between the two fiscal years, indicating rising EBSM effectiveness. However, this also could be due to a growing economy providing increasing job opportunities. Finally, other computations based on comparisons between the administrative data and the evaluation survey indicate that the administrative data do not capture all actual returns to work. Some of these differences can be partly attributable to the different datasets and methodologies used to compute the results. Once again, the interested reader is referred to Appendix C for a discussion of the different data sets and methodologies. 6.2 Impact on Participants: Descriptive AnalysisIn this section, we present descriptive findings relating to the impacts of six EBSMs: Self-Employment (SE), Targeted Wage Subsidies (TWS), Job Creation Partnerships (JCPs), Employment Assistance Services (EAS), Training and Feepayers.34 Two approaches to measuring impacts are used in this section: (1) clients' own subjective ratings of the importance of the help they received in obtaining employment; and (2) objective measures of labour-market outcomes as revealed by participants' post-intervention status in a number of areas, including employment and attitudes, as compared to non-participants. It is important to note that, in these simple (bivariate) descriptive analyses, observed differences in outcomes between participants and the comparison group cannot account for differences in the characteristics of the two groups. For example, if participants had greater education or were more highly motivated than non-participants, this could explain their possibly more favourable outcomes, irrespective of the role of the intervention. This is the reason that the descriptive analysis was followed by multivariate analyses, which can control for the pre-existing differences; the results of this multivariate analysis are presented in Section 6.3. Still, results of the descriptive analysis are presented here in order to provide basic information on a number of outcomes, overall and for various population sub-groups. Due to difficulties in interpreting the results of descriptive analyses, the results of a number of other labour-market outcomes, such as joblessness, job-search behaviour and utilization of income support, were examined in the multivariate modelling and are presented in Section 6.3. The interested reader is referred to Appendix E for descriptive results pertaining to these labour-market outcomes. We start with participants' own assessment of EBSM importance and go on to objective observations of participants' post-intervention status. 6.2.1 Rated Importance of AssistanceSurvey respondents were asked to indicate the extent to which they thought the assistance they received from Human Resources Development Canada (HRDC) was important in helping them to get a job. Twenty-two percent of participant survey respondents who were employed in the post-program period felt that the program was very important (responded with a 6 or 7 on a 7-point scale). Sixteen percent rated the HRDC assistance as somewhat important (responded with a 3 to 5 on a 7-point scale) and 61 percent rated the assistance as not important (responded with a 1 or 2 on a 7-point scale) in helping them to get a job. Training and EAS respondents were the most likely to feel that the employment program was very important in helping them find a job (29 and 23 percent, respectively), whereas TWS participants were the least likely to feel this way (12 percent). It should be noted that this question was addressed only to those respondents who had a job in the post-intervention period and to respondents who were not continuers (i.e., Self-Employment participants who continued operating their businesses or wage subsidy participants who were hired on by the host employer in the post-program period). If we assume that these continuers would have provided a response of "very important" to this question had it been asked of them, the proportion of respondents who rated the HRDC assistance as very important rises dramatically among Self-Employment (from 16 to 78 percent), TWS (from 12 to 70 percent) and JCP participants (from 19 to 44 percent) (Exhibit 6.1). 
While participant ratings of the importance of HRDC assistance are modest, a comparison of participant and comparison group survey responses reveals that, overall, participants are more likely than comparison group respondents to rate the services they received as very important in helping them to get a job (Exhibit 6.2). This implies an advantage for participants in LMDA programs relative to non-participants (note that participants were asked about HRDC services whereas non-participants were asked about employment services in general). Among EBSM participants, reachbacks were somewhat more likely than claimants to rate the HRDC assistance as very important. An examination of participant demographics did not reveal any significant sub-group differences. 
Participants who continued in their wage subsidy jobs were also asked specifically about the importance of their wage subsidy program or job creation project in helping them to obtain the job with their program employer (Exhibit 6.3). JCP participants were more likely than TWS participants to rate their employment program as very important in helping them to get this job (68 versus 42 percent, respectively). 
Qualitative evidence on impacts was also obtained from focus groups with clients and employers. Focus groups with clients revealed a number of perceived impacts that LMDA programs have had to date. To begin, the acquisition of skills through training has been instrumental in allowing some participants to find employment. Even for those who had not found employment at the time of the focus groups, a number mentioned that the programs had contributed to an improved sense of self-confidence, self-esteem and optimism that they will find employment because their job prospects are better. These findings are somewhat consistent with results of the Pan-Canadian Formative Evaluation which found that the perceived impacts of EBSMs were primarily related to skills acquisition and, even more importantly, to work experience — not necessarily jobs. This study also found that EBSMs had had an impact on client attitudes through increased self-confidence. Self-reported positive benefits of wage subsidy programs include a salary increase for one respondent, as well as the acquisition of employment for a number of others. One factor that may contribute to post-intervention unemployment, noted by employers and participants alike, was that often employers do not keep employees on after the 12-week subsidy. Employers felt that they are often unable to do so because of financial restrictions. 6.2.2 Employment/Labour Market OutcomesIn this section we present survey evidence on employment status and retention. It is important to reiterate that observed differences in outcomes between participants and the comparison group as presented in this section do not control for the pre-existing characteristics of the different groups. Once again, due to problems with interpretation (i.e., the inability to control for pre-existing differences between participants and comparison group members), the descriptive results for a number of employment outcomes, such as employment rates and employment stability, are not presented in this section. These employment outcomes were analyzed through multivariate modelling and are presented in Section 6.3. Again, the interested reader is referred to Appendix E for the descriptive results pertaining to these employment outcomes. It should be pointed out before proceeding that a different pattern of results was observed between reachbacks and active EI claimants for many of the employment outcome measures (as shown here and in Appendix E). Specifically, reachbacks appeared to have more positive outcomes than active EI claimants. A possible explanation for this is a pre-existing strong labour force attachment among reachbacks which would mean greater employment "outcomes" for this group. In most cases here, the different findings between reachbacks and claimants, as well as between participants and the comparison (non-participant) group, can be attributed to positive employment outcomes specifically among comparison group reachbacks. This group would have received no form of assistance, whereas all of the participants and the comparison group claimants would have received some form of assistance, either through income assistance (EI), employment programming (EBSMs), or both. It may be that comparison group reachbacks' apparent lack of use of any form of assistance denotes an already strong labour force attachment and, consequently and apparently, more positive results in terms of employment outcomes. While existing data do not indicate significant differences in profiles between comparison group reachbacks relative to the other groups, it is likely that these data do not capture all differences between the groups that could account for the more positive employment outcomes observed for the reachback comparison group. i) Pre- /Post-Employment Status The impact of EBSMs on participants' employment status was measured in two ways. First, we compared participants' employment status in the week prior to the intervention (or reference date) to participants' employment status at the time of the survey. This measure provides a sense of the absolute change in employment status between these two points in times. Generally, this measure shows positive shifts in employment from the pre- to post-intervention periods for all EBSMs and larger shifts relative to the comparison group. Second, participants' employment status in the first week following the intervention and at the time of the survey was compared. This measure provides a sense of the extent to which employment status outcomes persist following an intervention. Descriptive analyses for this comparison are presented in Appendix E. As shown in Exhibit 6.4, there was a larger overall increase in employment from the pre-to post-program periods among participants relative to the comparison group and participants' post-program employment rates were slightly higher than those of the comparison group for both claimants (56 versus 47 percent, respectively) and reachbacks (57 versus 51 percent, respectively). It is important to note however, that the comparison group had a stronger attachment to the labour force prior to the reference date, thus the smaller observed increases in employment rates for this group were at least partly attributable to a ceiling effect caused by higher rates of pre-reference date employment. Unemployment rates were roughly equivalent for both participants and the comparison group overall but participants exhibited considerable declines in unemployment rates (55 to 29 percent and 64 to 32 percent) compared to the comparison group (30 to 34 percent and 38 to 32 percent).
Positive shifts in employment were observed for all the EBSMs under study as well (Exhibit 6.5). All groups at least doubled their employment rate and the largest positive shift in overall employment occurred for self-employment (SE) participants (19 to 83 percent). Unemployment was also much lower at the time of the survey, dropping considerably for all EBSMs, but particularly for SE.  View Exhibit 6.5 ii) Retention A more direct measure of the contribution of EBSMs to positive employment outcomes is the extent to which wage subsidy participants were hired on by their host employers following the completion of the wage subsidy. Only participants who completed their wage subsidy were asked this question. As shown in Exhibit 6.6, the majority of wage subsidy participants who completed the full period of their subsidy were hired on by their host employer (55 percent), although considerably more TWS participants were hired on than JCP participants (65 versus 30 percent, respectively). This latter finding is not surprising if we consider the nature of these employment programs (i.e., TWS is geared toward job placement whereas JCP is designed simply for work experience). TWS participants were also more likely to have been hired into the same job they had during the wage subsidy (86 versus 73 percent of JCP participants). Little difference existed in the extent to which claimants and reachbacks were hired on after their program; however, claimants were slightly more likely to have been hired into the same job they had during the wage subsidy (88 versus 80 percent, respectively). Rates of retention also varied according to client demographics (not shown). The incidence of participants being retained in full-time year-round jobs rose with education level, whereas retention rates for full-time seasonal jobs rose with age and declined with education. With respect to the types of jobs into which wage subsidy participants were hired by their host employers, TWS participants were four times as likely to have been hired into year-round full-time positions than JCP participants (43 versus 11 percent), whereas JCP participants were somewhat more likely to have been hired into full-time seasonal (47 versus 35 percent) or casual and contract positions (21 versus two percent). Again, these findings are not unexpected, given the nature of the programs. Differences in the types of post-program jobs participants held with their host employers were also observed in relation to clients' claimant status. While reachback clients were more likely to hold year-round full-time jobs (57 versus 30 percent), claimants were more likely to hold full-time seasonal (40 versus 26 percent) or part-time seasonal positions (12 versus five percent). The future summative evaluation will provide the opportunity to identify incremental impacts and results.
iii) Job Search Activity Those who reported actively searching for work in the post-program period were asked to specify the job search methods they used while looking for work. As shown in Exhibit 6.7, the types of job search activity were somewhat similar across all participant groups. The most commonly cited job search methods were sending in resumes or applications (64 percent), checking job banks (64 percent), personal networks to friends and family (56 percent), and personal visits to employers (52 percent). A number of sub-group differences were also observed for participants in different programs:
Other notable sub-group differences emerged for participants and the comparison group. Overall, participants were more likely than comparison group respondents to send applications and resumes, whereas comparison group respondents were more likely than participants to check the newspaper. Furthermore, claimants in both the participant (55 versus 49 percent) and comparison groups (51 versus 45 percent) were more likely than reachbacks to make personal visits to employers. View Exhibit 6.7 iv) Interest in Entering the Labour Force and Willingness to Move Participants' motivations to enter the labour force were measured through their rated interest in entering the labour force in the next 12 months and their willingness to travel for employment. Of those respondents who were jobless at the time of the survey, the vast majority of participants rated themselves as very interested (responded with a 6 or 7 on a 7-point scale) in entering the labour force in the next 12 months (92 percent) (Exhibit 6.8). Little variation existed in the extent to which participants in different EBSMs were interested in entering the labour force, although SE participants were least likely to be very interested (81 percent) and Training participants were most likely to be very interested (95 percent). Participants were more likely to be interested in entering the work force than comparison group respondents. 
Similar to rated interest in entering the labour force, participants were somewhat more likely to indicate that they would be willing to travel for the right job (responded with a 5, 6 or 7 on a 7-point scale) than comparison groups respondents (70 versus 59 percent, respectively),35 and were willing to do so for a lower average hourly wage ($12.50 versus $14.10, respectively) (Exhibit 6.9). Together, these findings imply that participants are more motivated to enter the labour force than comparison group respondents. Sub-group differences were observed according to client demographic characteristics (not shown). Willingness to travel was highest among males, younger respondents and single respondents. The minimum hourly wage to travel for a job was higher among men than women and rose as a function of the respondent's level of education. 
6.2.3 Summary of Descriptive Analysis of Program ImpactsOverall, the descriptive analysis of program impacts reveals some advantages for EBSM participants in terms of employment. Qualitative evidence from the focus groups suggests favourable employment outcomes were perceived to have occurred for participants, notably in terms of indirect employment measures, such as positive skill impacts and increased self-confidence. Survey evidence suggests the perceived impact of EBSMs on participants' post-intervention employment status is more modest, although EBSMs had a more positive impact on employment relative to services accessed by comparison group members. However, it is only through multivariate analysis, which takes into consideration the characteristics of participants which may pre-dispose them to successful outcomes, that definitive statements about program impacts can be made. This will be done in the next section. For now, the following points summarize the findings for different outcome indicators, based on the descriptive analysis.
6.3 Multivariate Modelling ResultsIn the preceding section, participants in six EBSMs (EAS, SE, TWS, JCP, Training, and Feepayers) were compared to non-participants (the comparison group) in terms of a large number of post-intervention outcomes. As mentioned, however, the results from this simple descriptive analysis may yield a biased estimate of the impact of EBSM participation because of pre-existing socio-demographic and labour-market differences between non-participants and participants, favouring the latter. For example, the fact that one intervention out-performs another may have more to do more with the fact that its participants are highly educated and motivated relative to non-participants and participants in other interventions, than it has to do with the intervention itself. To ensure that differences in measured outcomes were not simply the effect of these pre-existing differences, therefore, multivariate modelling analyses were conducted. In these analyses, we control for differences between participants and non-participants, so that a "net" effect of EBSM participation can be measured. The idea behind multivariate analysis is to use a statistical technique to explain a particular outcome (for example, post-program employment) in terms of a set of factors. The "set" of factors used in the analysis includes both the variables of interest (in this case, participation in EBSM interventions) and a number of other "control" variables (for example, background characteristics of participants) which might also explain the differences in the employment outcome. We also add what is called a "Heckman Correction" factor or "Inverse Mill's Ratio," which controls for any unobservable characteristics that distinguish participants from non-participants. (See Appendix C section 4 for a full explanation of the methodological approach used.) In addition, because the models indicated that particular control variables influence the outcomes, we ran the models separately for segments defined by gender, age, claimant status, prior employment status, and rural/urban status. Models for the latter were run because rural-urban differences were of particular interest to the evaluation committee. The purpose of this exercise was to observe how paired segments differed in terms of how the interventions impact on outcomes, for example, how men and women differ in terms of being employed following an intervention. The specific variables used in the analysis are as follows. The outcomes are a set of eleven employment, earnings and income-support measures, many of which were examined above in the descriptive analysis. These outcomes reflect the basic increased employment and reduced income-support dependence objectives of the Canada/PEI LMDA and include seasonal employment as an outcome, which is particularly relevant in the PEI context. The intervention variables in the models cover participation in the five EBSMs of EAS, SE, TWS, JCP, and Training and Feepayers (TFP) combined.36 The control variables capture the time since the intervention; socio-economic and labour-market characteristics of individuals such as sex, age, education, residence, past labour market history, EI claimant status, and past use of income support; and use of assistance services such as self-serve products, counselling and action plans — all of which are variables that could account for differences in outcomes between participants and non-participants over and above the impact of the interventions. See Appendix F for a complete list of the variables used in the analysis and their means and frequencies, indicating the differences between participants and non-participants. In this section, a summary of the results of the multivariate modelling analysis is presented, including the segmented analysis. The summary is in three parts. We provide, first, the overall findings in each main outcome area, then a summary of the intervention-by-intervention results, and finally a summary of the characteristics of participants who tend to benefit from each intervention. Please see Appendices G to J under separate cover. In the presentation and discussion of the results, a number of important points should be borne in mind. First, lower in the regression analysis we mention only variables having a "statistically significant" impact on the outcome variable. Significance is measured at the five percent level, which means that we are 95 percent confident of the result presented. Second, in this formative evaluation, the impacts of interventions have been measured only over a short period of time (one year or less). Truer measures of these outcomes are obtainable over the longer term, a need that to some extent will be addressed in the summative evaluation. Third, for this reason as well, we attempt to point out only patterns of findings rather than focus on specific outcomes for specific groups and characteristics of participants. Presenting all the detail would not only obscure the main findings, but would also not be useful at this early stage in the process when the focus is on design, delivery and implementation issues. We do, however, discuss specific differences between rural and urban participants because of the above mentioned interest in this issue among committee members. Note that we also mention specific participant characteristics that appear related to success so as to guide program officers in targeting of their assistance efforts. 6.3.1 FindingsThe analysis was organized and the results are presented according to the four main intervention outcomes: employment, job search, earnings, and income support use. Where applicable, multivariate results are compared to the results of the descriptive analysis from the previous section and to the national results presented in the pan-Canadian EBSM formative evaluation report. In the exhibits presented in this sub-section, we show only the results for the EBSM interventions, indicating the direction of the impact each EBSM had on each outcome where the measured impact was significant in the models. In addition, based on the results presented in the appendices, we comment on the significant impacts of the control variables among all participants, as well as the differences in program impacts between paired segments (e.g., males compared to females) where patterns arise, while emphasizing rural/urban differences. i) Employment Outcomes In Exhibit 6.10, we present the impacts of the EBSMs for five (post-intervention) employment outcomes: the likelihood of being currently employed, the likelihood of being currently full-time employed, the likelihood of being currently seasonally employed, the likelihood of being employed for three consecutive months, and the percentage of weeks employed since the intervention. Non-significant impacts are indicated by "NS" in the exhibits. The results show that SE led to positive employment outcomes for participants compared to non-participants for most employment outcomes. The likelihood of seasonal employment, a particularly important outcome from the perspective of the Island, was reduced only by participation in SE. Additionally, TWS and TFP had a positive effect on the likelihood of three consecutive months of employment. Other EBSMs had negative employment effects: JCP reduced the chances of current employment while EAS reduced the percentage of weeks working. These results run counter to results from the descriptive analysis, which had indicated positive employment outcomes for participants compared to non-participants in a wide set of interventions. Controlling for differences between participants and non-participants, as was done here, has eliminated the apparent positive employment effects of the several interventions identified in the descriptive analysis. It would seem that participants in a number of interventions were predisposed to realize greater employment gains than non-participants, even before participating in the EBSM interventions.
The complete results according to different groups (or segments) of individuals presented in Appendix G indicate few patterns in employment impacts of EBSM interventions with respect to differences between paired segments. For example, in comparing males and females, depending on the intervention and employment outcome, sometimes it would be male participants who were affected by a particular intervention and sometimes it would be female participants. With respect to specific urban-rural differences in employment outcomes of the interventions, they are the following:
The analysis also identified certain client characteristics contributing to employment success beyond the role played by the interventions (see Appendix G). These "success factors" include having a post-secondary education, earning over $20,000 in the year before the intervention and being employed one year before the intervention. Negative factors are being less than 45 years of age and being in a minority group. A factor contributing to sustained employment (i.e., as evidenced by an increase in full-time employment, three consecutive months of employment, or a greater proportion of weeks employed) is the length of time since the intervention. The descriptive analysis in the previous section showed that, even over the relatively short period of time between the intervention and the survey, the percentage of EBSM participants who were unemployed fell, while that of those in full-time, seasonal jobs increased. This result lends support to the notion that some interventions (such as EAS and TFP) may have longer post-intervention "gestation" periods that exceed the time horizon of this evaluation. This interpretation is further supported by focus-group responses which indicated that participants were realizing skill gains, which were seen as contributing to positive employment outcomes down the road. Similarly, at the national level, the pan-Canadian EBSM evaluation report did not report information on employment impacts per se, undoubtedly owing to the short-term nature of the evaluation period. Instead, only skills and job-experience impacts were observed, which were seen as contributing to employment outcomes in the future. ii) Job Search In Exhibit 6.11, we present the intervention results from modelling job-search intensity, defined as the number of weeks in job search as a percentage of the weeks since the intervention while jobless. The results indicate that participation in SE reduced job search, a finding that is true for most sub-groups of the population. That EAS, which includes job-search assistance, did not increase job-search intensity is somewhat surprising. As indicated in the complete results presented in column 1 of Exhibit H.1 of Appendix H, factors associated with job-search intensity while jobless can be identified. A large number of personal characteristics increased the intensity of job search, including being male, having a pre-intervention interest in being trained, starting a business and entering the labour market, having received 105 or more weeks of EI before the intervention, and having used self-serve employment assistance products. Looking at differences within pairs of segments, all groups experienced reduced job search intensity following participation in SE, except claimants, the not-employed and urban residents. Only urban residents' job search intensity was negatively affected by JCP and TWS. View Exhibit 6.11 iii) Earnings Enhanced earnings is another goal of the Canada/PEI LMDA. Exhibit 6.12 presents the modelling results for the EBSM interventions for three earnings outcomes: current weekly earnings, absolute change in weekly earnings from before the intervention to the time of the survey, and percentage change in weekly earnings over that period. The earlier descriptive analysis identified no relationship between EBSM participation and earnings. Note, however, that after controlling for other factors affecting earnings, the multivariate analysis reveals that TWS, JCP and EAS lead to negative earnings outcomes. Additionally, current weekly earnings and percentage change in earnings were reduced by participation in SE in a number of segments. In general, it is rural not urban residents who experience negative earnings outcomes following their interventions, particularly EAS, TWS and JCP. Negative earnings outcomes may be due to the fact that, as shown in the descriptive results, participants often change employment status. This suggests that participants may be changing careers and finding themselves near the "bottom of the career ladder" following their intervention. Once again, time will tell if positive earnings outcomes will eventually occur and possibly show up in the summative evaluation. Finally, the complete modelling results presented in Appendix I indicate that having a post-secondary education, being younger, being male, and having three or more job separations contribute significantly to positive earnings outcomes. Also, earning $10,000 or more in the year prior to the intervention increases current weekly earnings and percentage change in earnings, while earning $30,000 or more reduces the absolute change in earnings. This finding implies again, that participants are moving to entry-level jobs following their intervention.
iv) Income Support Use In addition to the goals of sustained employment and increased earnings for EBSM participants, an objective of the Canada/PEI LMDA is to reduce dependence on Employment Insurance (EI) and Social Assistance (SA). In Exhibit 6.13, we present results for outcomes in these areas. Note that a negative result is really the sought-after outcome here. We first discuss participants' post-intervention income-support use relative to non-participants. Then we discuss the effects of prior use of income support on post-intervention use, which are indicative of the extent to which income support use is being reduced. We start with comparisons of EBSM participants' use of EI. The results in Exhibit 6.13, column 1, indicate that only SE participants had lower EI use compared to non-participants (the comparison group). This result is contrary to the descriptive evidence which showed that participants in all interventions had lower EI use. Once again, controlling for differences between participants and non-participants has eliminated the apparent advantage for participants. However, the segmented results in Exhibit J.1 in Appendix I do show that interventions reduced EI use for specific segments, as follows:
Finally, the analysis also found that TWS increased EI use for rural participants alone. As for the effects of the control variables, the complete results presented in Appendix J also indicate that having a post-secondary education reduces post-intervention EI use, as does having pre-intervention interest in starting a business and earnings of $20,000 or more. On the other hand, speaking a language other than French or English, having more than two job separations, and having received more than 105 weeks of EI in the five-year pre-intervention period increased the percentage of post-intervention weeks on EI. The latter implies no change in EI use among heavy users of EI. Also, the longer the time since the intervention, the greater is the percentage of weeks on EI, implying discouragement over time. Turning to post-intervention SA use, Exhibit 6.13, column 2 indicates that no intervention had an impact, suggesting that the EBSMs may have difficulty in reducing use of an income support mechanism (SA) not tied to the labour market (as EI is). Indeed, the segmented analysis (Exhibit J.2 in Appendix J) indicates that EAS acted to increase SA use for younger, employed and urban segments. With respect to effects of the control variables, having a post-secondary education leads to lower likelihood of using SA, as with EI use. Being married also reduces the likelihood of SA use, but being in a minority group reduces it, as does using SA before the intervention.
Turning to reliance on public income support over time, we showed above that pre-intervention use of SA and heavy pre-intervention use of EI were strong determinants of post-intervention use of the respective income support mechanisms. But this finding was based on the full sample including non-participants. Therefore, we focused on just EBSM participants to measure the impact of pre-intervention use of income support on post-intervention use. The results (not shown) indicate that pre-intervention users of EI and of SA were more likely than non-users or low-users to continue to use the respective income-support mechanisms in the post-intervention period. Among EBSM participants, then, there has been no reduction in income support dependence, at least within the short-term context of this formative evaluation. Conversely, the pan-Canadian EBSM formative evaluation report found that, nationally, EBSM participation did appear to reduce EI dependence in the short term. 6.3.2 Summary by InterventionTo provide more detail on outcomes on an intervention-by-intervention basis, the modelling results have been summarized in Exhibit 6.14. In each cell, there are two lines of information:
A summary of the findings, based on the exhibit, follows:
View Exhibit 6.14
6.3.3 Client Profile SummaryIn this section, we summarize the impacts of each EBSM intervention, this time emphasizing the characteristics of participants who tend to profit from their participation in the intervention. The material here is based on the results of the segmented analysis summarized in Exhibit 6.14 and presented in detail in Appendices G to J. For each intervention, where there are differences in outcomes within pairs of segments, the specific characteristics of clients benefiting from their participation in the intervention appear in italics. For participation in SE, there are no real differences in intervention impacts on employment outcomes within pairs of segments, except that the employed one year before tend to do better than the not employed, and rural participants do better than urban participants. Other characteristics are associated with specific employment outcomes; for example, younger participants in SE do better than older participants with respect to seasonal employment; and reachback participants do better than claimants with respect to current employment; but the opposite is true with respect to three consecutive months of employment. No population groups derive any gains in job-search intensity or earnings or reduction in SA use as a result of their participation in SE. SE acts to reduce the relative length of post-intervention EI spells for male, female, younger, claimant, reachback, employed, not employed, and rural participants. For participation in TWS, the ideal characteristics vary according to outcome measure and intervention. In general, though, those employed one year before participating do better than those not employed. Other ideal characteristics include being an urban resident or a reachback for current employment; being older, a reachback and living in an urban setting for full-time employment; being younger and living in a rural setting for seasonal employment; being anything but a reachback, not employed and living in an urban setting for three consecutive months of employment; and being older and a reachback for percentage of weeks employed. No population group benefits from TWS participation with respect to job search, earnings and income-support outcomes. For participation in JCP, being a claimant, not employed and a rural resident benefits a participant from the standpoint of reduced chances of seasonal employment. Being a claimant also benefits participants in terms of three consecutive months of employment. These are the only positive employment results for JCP. No population group experiences increased job-search intensity, positive earnings outcomes or reduced SA use from JCP participation. For reduced post-intervention EI spells, ideal characteristics for JCP participants are being male, older, an active claimant and an urban resident. For participation in TFP, the extent to which participants experience positive employment outcomes again varies by outcome measure and client characteristic. For example, for full-time employment, males do better than females; the employed do better than the not employed; and urban participants do better than rural participants. For seasonal employment, younger participants do better than older participants; claimants, better than reachbacks; employed participants, better than not employed participants, and rural participants, better than urban ones. And for the chances of three consecutive months of employment, ideal characteristics include being female, younger, a claimant, and employed one year before. No positive outcomes of TFP participation were observed for job search intensity, earnings, and SA use. Finally, TFP leads to reduced post-intervention EI spells for male and urban participants. For participation in EAS, no population group enjoys positive employment outcomes, except urban participants in terms of current employment. Nor did groups participating in EAS see post-intervention job-search intensity or earnings increase. Finally, male, female, younger, and urban EAS participants enjoy declines in their post-intervention EI spells, but no group participating in EAS saw reduced SA use. 6.4 Overall Chapter SummaryEvidence on several measures of EBSM success to date was presented in this chapter. Though it is early in the implementation process (year one), it was deemed important to monitor and assess how the EBSMs are working to provide early warning signals of expectations not being met and to provide some insight into potential evaluation issues and design for the summative evaluation phase. There are three main findings of the analysis presented here. First, with respect to accountability results targets set for the Canada/PEI Labour Market Development Agreement, this evaluation found that in 1998/99 targets were exceeded in two areas: EBSM participants returning to work and unpaid EI benefits resulting from a return to work before the end of the EI claim. While this is indicative of EBSM success to date, this could also possibly reflect targets that may be set too low or an economy that is improving. It was also found that the return-to-work indicator in HRDC administrative data may under-estimate actual returns to work, implying the need for improvement in the information systems and procedures used to track and monitor participants' progress. Second, this evaluation has shown that it is important to control for differences in characteristics of EBSM participants and non-participants. These differences could contribute to success beyond the influence of the interventions themselves. Whereas results of the descriptive analysis presented early in this chapter indicated a range of positive employment and other outcomes for participants in most EBSMs compared to non-participants, the multivariate analysis, which controlled for differences between participants and non-participants, revealed that it was mainly Self-Employment (SE), and to a lesser extent Targeted Wage Subsidies (TWS) and Training Feepayer (TFP) participants, who benefited from EBSM participation to date in terms of employment gains and reduced EI use. Third, it was determined that it may be too early to make definitive statements about the effectiveness of the EBSMs. Qualitative evidence gathered in this evaluation did indicate some job gains, but, more importantly, real gains in skills and confidence, which may be expected in the long term to materialize in jobs. This is supported by the survey evidence gathered for this evaluation indicating that the chances of job gains rose with the time since the intervention. The fact that the multivariate analysis pointed to job gains mainly for SE participants and to some extent TWS and TFP participants does not necessarily mean that the other EBSMs are ineffective. Instead, it means that it may take longer to see the payoff from these EBSMs, beyond the scope of this formative evaluation.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[To Top]](/web/20060127015404im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}
![[Previous Page]](/web/20060127015404im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127015404im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127015404im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}