Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

This chapter assesses short-term impacts and outcomes for participants. Recall that to be eligible, participants need to be unemployed and may be on EI Part I, if an active claimant, or have exhausted either a regular or maternity/paternity claim in the past.

A. Assuming More Responsibility

The EBSMs encourage clients to assume more responsibility for the following under the Employment Insurance Act:

• Identifying employment needs.

• Sharing in the cost of assistance, if appropriate.

• Increasing employment.

• Reducing their dependency on EI/IA.

As could be expected, views varied widely from area to area and from agency to agency within an area regarding achievement of this goal. Factors that affect views include the type of client dealt with (e.g., whether the agency deals primarily with clients who are not job-ready), the prevailing economic conditions and the availability of jobs, especially entry-level or lower-paying jobs.

Where there are jobs available, EBSMs are seen as reducing dependency. Where there are no long-term jobs, EBSMs are seen as being less successful in securing employment. However, they are still seen as making clients more marketable and better able to market themselves.

Action plans are seen as assisting or even forcing clients to take more responsibility for their employment. Those clients who are case-managed generally have a RTWAP. The plan may be prepared by themselves, alone or with the help of a counsellor, or prepared by the agency. In most cases, however prepared, the client and counsellor are required to sign off on the plan. In most cases, it appears that the plan is used to measure and track progress.

Some interviewees note that many clients, especially those receiving training or self-employment assistance, have contributed to the cost of their assistance; some have done so in a significant way and incurred large debts with no assurance of work in the future. Other interviewees note that clients either do not contribute (especially mutual clients) or contribute nominal amounts or make in-kind contributions only.

We asked questions in our participant and comparison group member surveys designed to identify if there was a difference between the level of responsibility of the two groups related to labour market issues. Differences between the two groups may be attributable to participation since the two groups were drawn to be statistically similar in observable characteristics other than participation. Note, however, that any differences may be due to other unobservable factors. For example, we cannot rule out the possibility that some hidden factor, such as motivation, is attributable both to participation and to any difference between participants' and comparison group members' responses to these questions. Findings are indicative but not definitive of an attributable impact of the EBSMs.

Overall, participants seem slightly more positive than comparison group members on questions related to the level of responsibility individuals have toward identifying their employment needs. They may take more responsibility for identifying these needs.

There is no difference between the two groups on the question asked about sharing in the cost of assistance. There are no significant differences between participants and comparison group members in terms of their willingness to pay for a training course. As we have seen earlier, both groups are also similar in terms of the proportion paying at least part of their costs of training. However, comparison group members are far more likely to pay the full cost of training.

Findings across the two groups suggest both share similar levels of responsibility for increasing employment. Questions common to the participant and comparison group questionnaires probed for factors which might be related to a heightened responsibility for increasing employment. Responses suggest participants' and comparison group members' views are similar across these statements.

Participants may be more responsible for reducing their dependence on EI/IA through their involvement with an EBSM. Participants' responses were slightly higher than comparison group members on a question asked to assess individuals' responsibility toward reducing their dependence on EI and IA. The regression analysis confirms that participation in an EBSM affects the importance placed on being independent of EI or IA. Again, however, we cannot rule out some unobserved factor, such as motivation, which may affect both participation and responsibility.

B. Discontinuation

Of the participants surveyed who were not still involved in their EBSMs, 16% report discontinuing their participation in the EBSM before it was completed. Almost one-half of the reasons for discontinuation were positive-to get a job. In 24% of cases, discontinuation is a negative result of the program-the withdrawal of funds or dissatisfaction. Of the participants who discontinued their EBSMs, almost one-half (48%) report doing so because they found employment. Other reasons why their programs were not completed include funding reductions (15%), maternity leaves (8%), lack of program satisfaction (7%), family responsibilities or medical reasons (6%), asked to leave (4%), difficulties with employer (4%), employer could not afford (2%) or other reasons (5%).

C. Satisfaction

A majority of the participants whose EBSM involvement was finished at the time of the survey give their EBSMs a positive satisfaction rating. More than three-quarters (79%) rate their EBSM as "5" or more on a seven-point scale where one means "very unsatisfied" and seven means "very satisfied." The average rating given to the EBSMs by these participants is 5.6. The average rating for participants taking TP and SE was 5.9 and 5.8, respectively.

Using a seven-point scale where one means "very unuseful" and seven means "very useful," participants rated the usefulness of their EBSM to them in obtaining full-time employment. The average was 3.9. TWS was highest rated at 4.9 and SE and TP were rated at 4.2 and 3.8, respectively.

Participants also gave their EBSMs positive ratings in terms of their usefulness in six areas. On average, these participants rated their EBSMs as being most useful in terms of building their confidence (5.0), focusing their career goals (4.9) and learning how to do a certain job (4.6). They were also considered to be useful, though to a lesser extent on average, at helping participants develop job skills such as managing time and organizational skills (4.0), obtain full-time employment (3.9) and improve or reach their earning potential (3.9).

D. Attitudes to Work

Most of the participants who had finished their EBSM report that their attitudes toward work and the future have become more positive (59%) or stayed the same (32%) since their EBSM ended.

E. Further Training

Twenty-three per cent of participants have taken further training or courses since their EBSM ended. Almost one-half of these participants (48%) mention taking this training as a result of their involvement in their EBSM.

F. Employment

Seventy-five per cent of participants who had completed or discontinued their EBSM or who were self-serve only clients have been employed subsequent to their involvement with the LMDA. For active claimants and reach-back clients, the percentage who have worked in their post-EBSM period was 75% and 77%, respectively. For members of the comparison group, 69% had worked after the date when the participant they were chosen to represent was scheduled to complete their intervention. For comparison group members chosen to represent active claimants, 70% had worked; and for those chosen to represent reach-back clients, 66% had worked.

Exhibit 4 compares the characteristics of jobs held by participants and comparison group members in the post-EBSM period. Most jobs held by participants (84%) or comparison group members (91%) were with employers. The exception was for participants of SE where 64% of jobs in the post-program period were in business for themselves. Excluding participants under SE, 92% of jobs for participants are with employers-comparable to the comparison group findings.

As Exhibit 4 illustrates, the job experiences of participant and comparison group members are highly similar. Notable exceptions are the income from employment and employment status. These are explored in more detail in Exhibit 5 for participants who were current claimants and the reach-back clients and the comparison group individuals who correspond to them. (Note that other characteristics presented in Exhibit 4 do not vary appreciably by client status.)

Exhibit 5 Selected Characteristics Of Post-EBSM Period
	Participants			Comparison Group
	Total	Current Claimants	Reach- Back	Total	Current Claimants	Reach- Back
Working At Some Time	75%	75%	77%	69%	70%	66%
Average Income Earned In Typical Week	$466	$466	$464	$538	$555	$485
Currently Working	65%	65%	67%	55%	55%	54%

As noted in Exhibit 5, participants are more likely to work in the post-EBSM period-75% compared to 69% for comparison group members. Reach-back clients are most likely to have worked (77%) while the group chosen to compare to them are least likely to have worked (66%)³ . This observation also holds for the percentage employed at the time of the survey. However, those comparison group members who do work tend to have higher average weekly earnings than participants who work.

Exhibit 6 compares the pre-EBSM period experience for participant and comparison group members. The pre-EBSM period is defined as the year preceding the start date of the EBSM for participants. For comparison group members, it is the year prior to the start date of the intervention for their matched participant. The exhibit illustrates data for all participants, active claimants and reach-back clients, as well as data for comparable groups from the comparison survey.

As the exhibit illustrates, our comparison group well matches participants in terms of their pre-EBSM labour market experience-especially for active claimants. However, what is striking is the reported labour market experience of reach-back clients. Fifty-seven per cent of participants who are reach-back clients report working in the year prior to their EBSM. Jobs are full time (average 39 hours per week), most are for an employer (84%) and lasted on average 15 months.

Comparison of Exhibits 4, 5 and 6 suggest:

• Participants did better than comparison group members in the post-EBSM period compared to the pre-EBSM period in terms of the percentage working. Reach-back participants had the largest improvement from 57% to 77%. Comparison group members, who were chosen to represent reach-back participants, experienced a drop in the percentage employed from 75% to 66%, in a similar period. Current claimant participant employment dropped from 78% to 75% between the pre- and post-EBSM periods. For comparison group members chosen to represent current claimants, the employment rate dropped from 78% to 70%.

• Results are not as unequivocal in terms of income earned in a typical week for those who are working. Reach-back participants have increased their earnings relative to the comparison group members. Reach-back participants gained $20, on average, compared to a loss of $110 for the reach-back comparison group between the pre- and post-EBSM periods. Current claimant participants, on the other hand, lost $59 compared to a gain of $60 for the relevant comparison group members⁴ .

We also assessed specific employment impacts for some EBSMs through our survey:

• Fifty-five per cent of participants ending TWS interventions continued to work with their TWS employer.

• Eighty-six per cent of participants ending SE interventions started their own businesses. Of those who had started businesses, 96% were continuing to run their business when surveyed and 25% had hired one or more employees to work with them.

G. Dependence on EI and IA in Post-EBSM Period

Exhibit 7 provides a summary table of the post-EBSM dependence on EI and IA of participants and comparison group members. In the post-EBSM period, 40% of participants and 37% of comparison group members had received EI. The figure for participants will include those still on claim when their EBSM ended and may include those whose EI was "extended" (EI Part II income support) through participation in EBSMs after the one selected for our survey. Note that 50% of participants who were current claimants received EI in the post-EBSM period, compared to 40% of comparison group members selected to be like current claimants.

Nine per cent of participants and 5% of comparison group members received IA in the post-EBSM period. The principal reason identified for being on EI or IA was a lack of work available. Seasonal layoff as a reason mentioned was higher for our comparison group members.

The percentage on EI was lower for participants (10%) than comparison group members (12%) at the time of the survey. Months on EI and benefit rates were comparable across the two groups. The only outlier is the current rate on EI for TWS participants which stands at 25% compared to 9% for all EBSMs. A possible explanation is that time spent on a TWS increases EI eligibility.

The percentage on IA at the time of the survey remained higher for participants. Average IA amounts were similar across the two groups. The average number of months on IA was less for participants compared to comparison group members on IA at the time of the survey.

H. Short-Term Incremental Impacts on Earnings, Hours Worked and Dependence on Income Support

We used regression analyses to identify whether participation in an EBSM had a significant effect in the short-term on the experience of participants in terms of their:

• Earnings per year.

• Hours worked per year.

• EI benefits received per year.

• IA benefits received per year.

This analysis uses data for a matched comparison group of non-participants to predict what would have happened to participants had their interventions not occurred. Individuals for the comparison group were chosen to reflect the characteristics of participants (determined through administrative sources) other than participation. The regression models compared the actual and predicted experiences, the difference reflecting the effect of participation in an EBSM.

These models corrected for selection bias. It is possible that some unknown factors, such as motivation, may be related both to participation and to outcomes. It is this relationship that creates selection bias. However, the modelling approach that we used was designed to correct for the effects of such bias..

A number of demographic and other variables were used to control for remaining differences between participants and comparison group members, including pre-EBSM values of some of the outcome variables.

These are tough tests for the EBSMs. Typically, EBSMs remove the participant from the labour market for a period of time. Over this period, their matched comparison group member remains in the labour market, may be able to get a job and begin to increase their employment earnings. Participants may need to catch up to comparison group members when their EBSMs are completed. Regression analyses are generally reserved for summative evaluations where the longer term analysis provides for a fairer test of the EBSM impacts.

Exhibit 8 presents significant findings from the short-term regression analysis related to the above outcome variables. Separate models were used for all EBSMs, all EBSMs excluding EAS, each separate EBSM, active claimants and reach-back clients. For each model, and outcome variable which are significant, we list the estimated value for the variable and the level of significance⁵ of the estimate.

Across all EBSMs, the annual EI benefits received by participants are estimated to have increased by $663 as a result of participation in an EBSM. For current claimants, EI benefits received on an annual basis increased through participation by $1,203. For reach-back clients, EI benefits received decreased by $798 compared to what would have happened in the absence of the intervention. For employment benefit measures only (EAS is excluded), there are no significant impacts on EI benefits through participation.

Participation did not result in significant changes in earnings, hours worked or IA benefits on an annualized basis across all EBSMs or all employment benefits. Note, however, that hours worked increased significantly for reach-back clients. Recall that the post-EBSM period is relatively short in the formative evaluation (typically less than one year). Participants have likely not reached their full labour market potential so soon after the intervention. The intervention also may have effectively removed the participant from the labour market compared to what would have happened in the absence of the program. It may take some time for participants to recover from this labour market absence. (The summative evaluation can provide evidence of the longer-term impact of participation.) Because fewer observations are included in the models for individual EBSMs, it is less likely that these models are able to detect a significant change when one exists. Significant changes by individual program were:

• An increase in EI benefits received by EAS participants of $1,058 per year. (Recall EAS may provide access to other EBSMs. Participants who are current claimants may have their EI "extended" (EI Part II income support) through participation in other EBSMs.)

• A decrease in annual EI benefits received by SE participants of $1,060. (Recall that self-employment does not result in insurable hours.)

• An increase in hours worked per year by SE participants of 132 hours.

I. Impact of Negotiated Financial Assistance

Participants are encouraged to contribute to the cost of their RTWAP, if appropriate, through NFA. Sharing in the cost of training may bring positive impacts in terms of attitudes, labour market impacts and dependence on income support. This section assesses such potential impacts. Note that the number of participants paying all of their training cost is small. As a result, we merge this group with those paying part of their costs of training.

1. Attitudes Toward Work

Participants were asked their degree of agreement or disagreement with a number of statements related to attitudes toward work. The proportion that individuals paid of their training costs is a strong indicator of their willingness to pay for training to stay off government support. Those who paid at least part of their training costs are more likely to think they have a number of good qualities. They are also much more likely to turn down a good paying job if they had to move to get it. Training may be seen as a way to avoid moving to get a good paying job. Those unwilling to move may be more willing to pay for training. Those who paid at least part of the costs of their training place more importance on being independent from EI and IA.

2. Labour Market Impacts

Participants who have paid at least part of their training costs were slightly less likely to be employed in the period after training. Seventy per cent of those who paid at least part of their costs of training were employed in the post-EBSM period compared to 76% who paid none of the costs of training. At the time of the survey, 62% of participants who paid at least part of the training costs were employed compared to 65% who paid none.

3. Dependence On EI And IA

The proportion on EI at some time following their training for those who paid at least part or none of their cost of training were 41% and 29%, respectively. The proportion on EI at the time of the survey was 6% for both groups.

Similarly, 14% of those who paid at least part and 7% of those who paid none of their cost of training were on IA at some point after their training. The proportion on IA at the time of the survey was 12% and 3% for those who paid at least part or none of their cost of training, respectively.

As a result, paying at least part of the costs of training does not appear to reduce the dependence on EI and IA.

J. Outcomes in Terms of Savings to the Employment Insurance Fund

In areas where there are no jobs, TWS may find short-term employment for participants who then become eligible for EI benefits again in the future. This may account for the high proportion of TWS clients on EI in the post-EBSM period (53%) and at the time of survey (25%).

Footnotes

3	On further inspection, we found that 42% of the reach-back comparison group had worked throughout the EBSM period. This compares to 23% of reach-back clients who claimed their post-EBSM job had started prior to their intervention start date. Forty-four per cent of the latter, who worked, worked for themselves. These participants may not have technically been unemployed. However, because our comparison group also includes those who were employed during the EBSM period, we are able to determine the incremental impact of participation through our regression analysis.
4	A loss in income is sometimes observed for participants in the short term. Participation results in their absence from the labour market. During this absence, comparison group members may find a job and begin to increase their earnings. When participants return to the labour market, their earnings may be lower than comparison group members. Participant earnings must "catch up" to earnings of those who have remained in the labour market. However, over the longer term, the rate of increase in wages for participants may be greater than for comparison group members. Earnings in the medium and long term may be greater for participants. This issue will be explored by the summative evaluation.
5	The level of significance identifies the risk or probability that an estimated result, which is judged to be significant, is in fact not different from the actual result. For a result to be considered significant, a maximum risk of 5% is generally accepted. Smaller levels of significance represent less risk and therefore greater certainty that the stated result is significant. We report the level of significance for all results identified as significant.