Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

This chapter presents outcome evaluation findings for the Edmonton Destinations ITCY. Gross program effects are reported first, followed by incremental impacts.

2.1 Gross Effects

(a) Client Satisfaction

Table 2 presents information taken from Exit A (n = 138) and Exit B (n = 123) surveys relating to clients' experience with their training.

Table 2 - Client Satisfaction with Training

With a mean satisfaction rating of 8.1 out of 10, Edmonton clients were highly satisfied with the program. It is interesting, however, that of those who had taken another training program in the past, only 62% felt the Destinations program was better.

Relatively few clients (17%) had a problem during training that they needed help with. All of these clients felt the agency tried to help, and all were satisfied with the help they received. The majority of clients (72%) indicated having a role model¹¹ while going through the program.

Essentially all clients received incentive money from the agency, and the majority (77%) felt the money was enough to meet their needs. It is interesting, however, that only 37% of clients indicated they would have quit the program had they stopped receiving the incentive, possibly suggesting the incentive was not absolutely necessary in all cases.¹²

The majority of clients were satisfied with specific agency services:

• support received while training off site (84%);
• agency efforts to help them find a job (93%); and
• support received while adjusting to the job (89%).

Most (80%) were also satisfied with their first job as a place to start their career.

Of those contacted who left the program before completing their training, the majority (65%) indicated dropping out on their own rather than being asked to leave by the agency. The main reasons for dropping out include:

• wasn't learning anything (21%);
• had transportation problems (21%);
• got a job (21%);
• needed money, had to get a job (17%);
• had family problems (17%).

(b) Skills Acquired

Table 3 presents information about the skills clients indicated learning in the program.

Table 3 - Program Impacts – Skills Acquired

A greater proportion of clients learned life skills and job skills than learned academic skills. This result is not surprising in view of the fact that academic skills were not stressed in the program.¹³ In terms of job finding skills, significantly more learned how to look for a job than how to go into business for themselves. Again, entrepreneurial skills were not stressed in the program.

It took just under 2 weeks on average for Destinations clients to obtain their first job after training. Of those who were working at the time of the Exit B survey, the majority (76%) indicated they were using skills they had learned in the program. It is interesting, however, that 40% felt they could have gotten their current job without the training they received, suggesting that many may have been working at low-skilled/entry level jobs and/or jobs unrelated to their training.

(c) Changes in Lifestyle and Attitudes

Table 4 shows that the majority of clients (83%) attributed improvements in their attitude/motivation to the training program. Some clients also noted improvements in their family relationships and health.

Table 4 - Program Impacts – Lifestyle Improvements

Tables 5 provides information about changes in client attitudes by comparing pre-program information (Baseline) with information collected at various points after Baseline (Follow-up). The results suggest that attitudes did not change for the better after the program. In fact, clients' attitudes about life¹⁴ appeared to have deteriorated in the year after Baseline.

Table 5 - Program Impacts – Changes in Attitude (pre/post)

Table 6 shows that clients' involvement in recreation/social activities did not appear to change significantly after the program.

Table 6 - Program Impacts – Changes in Activities (pre/post)

(d) Employment Trends

Figure 3 shows the status of Edmonton clients at Baseline and at various points after Baseline as reported on Follow-up surveys. The results show a marked improvement in the proportion of clients employed¹⁵ over time, levelling off at approximately 65%, while the proportion unemployed¹⁶ declines correspondingly to about 20%. The proportions in school/training levels off at approximately 20%. The proportion on SFI stays relatively stable at roughly the level it was at Baseline (i.e., slightly under 20%).

Figure 3 - PG Status from Baseline to Month 18

The main reasons clients gave for being unemployed when contacted on follow-up include:

• lack of experience (24%);
• lack of education (19%);
• no jobs available (19%);
• no transportation (10%).

The first two reasons are somewhat surprising in view of the fact the clients just went through a training program.

(e) Income Trends

Figure 4 shows the average weekly income of Edmonton clients at various points after Baseline. The trend is clearly towards an increase in income over time. This can be attributed to steady increases over time in the proportion employed, the average hours worked per week and the average hourly wage earned (see Table 7 on the following page).

Figure 4 - PG Average Weekly Income

Table 7 - Trends in Employment and Income

(f) Most Successful Participants

The average weekly income for all program participants at 12 months (including those not working) was $143. Certain subgroups did better than others. The characteristics most strongly correlated to better incomes include:

• > Grade 8 education: $150
• ever had previous paid employment: $156
• household not on SFI (when client < age 16): $169
• aged 19 – 21: $170
• male: $171
• employed in year before Baseline: $173
• had a criminal record: $176

Except for the last characteristic (i.e., criminal record), it appears the program was not fully able to remove the traditional barriers to employment faced by many of the clients (i.e., younger, female, poor education, poor work history, previous SFI history, etc.).

Table 8 provides the characteristics of the jobs held 12 months from Baseline by clients who received different levels of training in the program.¹⁷ The results suggest that those who trained longer had better outcomes in terms of employability (i.e., proportion employed) and quality of employment (i.e., average weekly income, permanent job, opportunity for advancement).

Table 8 - Impact of Length of Training on Employment

2.2 Incremental Effects

While Figures 3 and 4 indicate that Program Group participants enjoyed steady improvement over time in their employability and employment incomes, it is necessary to contrast their results with those of the Comparison Group to determine the extent to which these improvements can be attributed to the ITCY training they received.

(a) Employment Characteristics

Table 9 compares the characteristics of the jobs held by PG and CG members 12 months from Baseline. The proportion of PG members working is greater (61% vs. 48%). For those who were employed, however, CG members appeared to fare slightly better in terms of weekly income from employment, although fewer found jobs providing benefits and fewer were satisfied with their hours/hourly rate. The findings suggest that PG members did not find jobs that were substantially better than those found by CG members.

Table9 - PG-CG Comparison of Employment Characteristics

(b) Employability

Figure 5 provides longitudinal data comparing the proportion of PG and CG participants employed at various points after Baseline.

Figure 5 - PG and CG Employment Status from Baseline to Month 18

The results clearly show that, like PG members, CG members also exhibited employability improvements over time. This finding was not unexpected, since the job market in the province was relatively buoyant at the time of the study. Furthermore, CG members were motivated to find employment and had applied to the ITCY in order to obtain training to help them find a job.¹⁸ The trend lines suggest, however, that once graduating from the ITCY, a greater proportion of the PG cohort was able to maintain employment. A 10% to 20% difference between the groups is sustained over time. This is demonstrated in Figure 6 on the following page that compares adjusted¹⁹ employment rates averaged over the 9 to 15 month period from Baseline.²⁰ The mean difference is 17%.

Figure 6 Adjusted PG and CG Employment Rates (mean of means for months 9, 12 and 15)

(c) Income

Figure 7 on the following page shows the adjusted incremental gains in weekly income of the PG cohort over time.²¹ The figure clearly demonstrates that the PG cohort had higher employment earnings than the CG cohort. This is further demonstrated in Figure 8 which compares adjusted weekly incomes for the two groups averaged over the 9 to 15 month period from Baseline. The PG earned an average of $63 more per week than the CG over this period.

Figure 7 Adjusted Incremental Gains in Average Weekly Income

Figure 8 Adjusted Average Weekly Income (mean of means for months 9, 12 and 15)

Other Outcomes

SFI Involvement

Table 10 provides information comparing PG and CG members in terms of the percentage who collected SFI and the average monthly benefit amount each cohort collected before and after Baseline.

Table 10 - SFI Claim Activity

It is interesting that the proportion of PG members collecting SFI jumped considerably after Baseline (i.e., from 18.5% to 32.9%), ²⁴ yet the average monthly benefit amount decreased. For the CG, both the proportion collecting and the average amount collected increased after Baseline. These findings suggest PG members may have become less reliant on SFI while the CG became more reliant, resulting in "savings" to the Province of approximately $9.85 per month per PG participant.²⁵

EI Involvement

Table 11 compares PG and CG members in terms of the percentage of the cohort who collected EI and the average amount each cohort collected before and after Baseline.

Table 11 EI Claim Activity²⁶ (August/93 — June/97)²⁷

It appears that PG members increased their involvement with EI when considering the proportion collecting in the post-Baseline period. The average amount collected per claim went down considerably, although the average amount of EI collected per year per PG member stayed about the same. CG members appeared to reduce their involvement in EI, both in terms of the proportion who claimed and the average amount collected.

Police Involvement

Table 12 compares PG and CG members in Edmonton in terms of their involvement with the Edmonton Police Service (EPS) over time. The results show a slight mitigation in police involvement for PG members over time compared to CG members.

Table 12 - Edmonton Police Service Activity

Footnotes

10	This data is drawn from Early Exit surveys which were not completed with every client who left before completing their training. The data may not agree with agency-provided statistics.
11	Clients were asked, "While you were going through the program, did anyone serve as a role model for you, someone who took you under their wing or inspired you to try harder?" The role model could have been an agency staff member or someone at the employer's job site.
12	The Interim Evaluation found that the availability of incentives helped to attract youth to the project.
13	The agency had difficulty with its computer-assisted learning system which was ultimately phased out of use.
14	These include: social life with friends; family life; education they have received; jobs they have had; their life in general.
15	Clients were designated employed if they were working at the time of the interview or if they indicated having been employed during periods for which data was collected historically.
16	Unemployed is defined as not working and not in school or training.
17	Phase 1 of training included the in-house life skills component only. Phase 2 included Phase 1 training plus some training at NAIT. Phase 3 training included Phases 1 and 2 and job skill training at an employer's job site.
18	It was found that only 8 of the 68 CG members went on to take alternative training within 9 months after Baseline. Only 3 of these individuals were employed at 12 months.
19	See page 7 for discussion on controlling for bias between PG and CG.
20	A repeated measures design was used to compute a mean of means for data periods 9, 12 and 15.
21	Incremental gains were derived by subtracting the mean weekly income for CG members from the mean for PG members. The means were calculated including those individuals with 0 income (i.e., not working) such that the means represent the weekly incomes averaged over the entire cohort rather than only those who were working at that point in time.
22	Includes those collecting for 1 or more months during the period.
23	The monthly average is calculated for the total cohort, including those who did not collect at all or did not collect every month.
24	This can partially be explained by the fact that the post-Baseline period tracked was somewhat longer than the pre-Baseline period (i.e., 18 months vs. 12 months).
25	Integrated Training Centres for Youth The difference between the PG change (-$3.85) and the CG change ($6.00).
26	For PG and CG members who provided Social Insurance Numbers.
27	Data used in the table was drawn from an EI database compiled by HRDC using only 1 week of claim activity from each month's historical claim data. Consequently, the statistics for percentage collecting EI and the average collected per year would not reflect actual claim activity for either cohort and should be used with discretion in making comparisons between cohorts.
28	The average is calculated for the total cohort, including those who did not collect any EI during the 47 month period from August, 1993 to June, 1997. The number of months before and after Baseline varies for each client and has been factored into the calculation to ensure the average is properly normalized.
29	Self-reported by youth on Baseline Survey. It is difficult to reconcile the relatively large proportion of PG/CG members who reported having a criminal record at Baseline (completed from May, 1995 to October, 1996), with the relatively small proportion who appeared on the EPS database. Further study would be required to assess the accuracy of the statistics presented.
30	% of individuals either accused or suspected of committing a crime in the given year.
31	Adjusted to approximate 12 months.