Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Integrated Training Centres for Youth (ITCY):
Outcome Evaluation
Strategic Initiatives Program

Final Report

Prepared for:
Evaluation and Data Development
Strategic Policy
Human Resources Development Canada
and
Alberta Advanced Education & Career Development

December 1998

SP-AH075E-12-98

The Integrated Training Centres for Youth (ITCY) pilot project is the result of a joint federal-provincial Strategic Initiative funded and managed by Human Resources Development Canada and the Alberta Departments of Advanced Education & Career Development and Family & Social Services.

This document reports the results of an outcome evaluation of ITCY conducted in 1996/1997 by the Alberta Management Group, in affiliation with T.D. Weiden and Associates. The consultants were directed by an Evaluation Steering Committee comprised of representatives from the federal and provincial partners.

The consultants would like to express their gratitude to program staff and participants; and to federal, provincial and community stakeholders who provided important information about program operations.

We would also like to express our appreciation to the following members of the Evaluation Steering Committee for their input and guidance during the course of the study.

Mr. Jay Alam
Evaluation and Data Development Branch
Human Resources Development Canada (Ottawa)

Mr. Ralph Bellstedt
Research Economic Services Branch
Human Resources Development Canada (Edmonton)

Mr. Don Gardener
Labour Market Services
Alberta Advanced Education & Career Development

Mr. Rudy Hoehn
Income and Employment Services Branch
Alberta Family & Social Services

Mr. Don Westlake
Evaluation Branch
Alberta Advanced Education & Career Development

The evaluation of the Integrated Training Centres for Youth pilot project was governed by a set of evaluation questions established by program sponsors at the time the pilot was launched. The evaluation plan called for both a Process Evaluation and an Outcome Evaluation to be conducted.

The Process Evaluation component was completed in March of 1996. The results of the Process Evaluation are contained in a Technical Report entitled, Integrated Training Centres for Youth: A Process Evaluation, and a companion Final Report prepared to summarise the main findings, conclusions and recommendations for purposes of wider dissemination.

The Outcome Evaluation component was conducted in two phases:

1. Qualitative Investigation — involving the collection and analysis of qualitative information taken from interviews with clients, project staff and other government and community stakeholders familiar with the ITCY programs.

2. Quantitative Investigation — involving the collection and analysis of quantitative information taken from Baseline, Exit and Follow-up surveys of program participants and members of a comparison group.

Qualitative results are contained in a report entitled, Integrated Training Centres for Youth — Interim Evaluation Report: Qualitative Findings, prepared in May of 1997.

This report presents detailed results of the quantitative investigation of outcomes. Major qualitative findings from the Interim Report are also brought forward in drawing conclusions relating to the major outcome evaluation issues investigated and in reporting overall "Lessons Learned" from the ITCY pilot project.

In 1994, Human Resources Development Canada (HRDC) launched the Strategic Initiatives (SI) program. In Alberta, it was agreed that SI would fund an Integrated Training for Youth pilot project in partnership with the provincial departments of Family & Social Services (F&SS;) and Advanced Education & Career Development (AE&CD;).

The pilot project was initiated to test the value of customised counselling, training and work site interventions for young individuals who are at risk of long term dependence on public income support. Tenders were called in early 1995 for agencies to establish Integrated Training Centres for Youth (ITCYs) which were to incorporate certain features of a service delivery model developed by the Center for Employment Training in the United States. Contracts were awarded to the following 3 agencies:

1. Career High — a program of the Chinook School Division to provide training to youth at two sites in Red Deer and Innisfail;

2. Destinations — a partnership between two training consultants and the Northern Alberta Institute of Technology to provide training to youth in Edmonton;

3. Fifth on Fifth Youth Services — a program of the Lethbridge Youth Foundation to provide integrated training to youth in Lethbridge and area.

The three ITCYs targeted youth aged 16 to 20 who had dropped out of school and were at risk of long-term dependence on social allowance. Clients were to receive integrated life, job and academic skill training coupled with job placement, job maintenance, financial incentives and other support services necessary to complete their training and gain meaningful employment.

Methodology

The Outcome Evaluation was intended to document program impacts on training participants, and determine the extent to which the intervention led to incremental benefits. The evaluation measured impacts using both a pre-post designs for program participants as well as a comparison group design using non-participating program applicants.

Qualitative information was collected through interviews with clients, program staff and other government and community stakeholders. ¹ Quantitative data was collected over a period of nearly 2 years through a series of survey instruments delivered to Program

Group and Comparison Group members at different stages of the intervention (Baseline, Exit, Follow-up). The quantitative results were consolidated into quarterly periods to compensate for low monthly response rates. The data was also adjusted using covariate regression techniques to compensate for selection bias between the Program and Comparison Groups. Bias between responders and non-responders was not considered significant.

Evaluation Conclusions

The following conclusions have been drawn regarding major evaluation issues established to guide the study:

• Participant Satisfaction — Clients rated the programs quite highly and were generally satisfied with the services and support they received. Employers were also generally satisfied with their role and with the clients they trained or hired.

• Reasons for Discontinuation — The main reasons participants discontinued training include: poor motivation/lack of interest; dissatisfaction with the program, having transportation barriers; getting a job; going back to school; and having family/financial problems.

• Impact of Incentives — The provision of financial incentives was instrumental in attracting many clients to the training. The incentive was adequate for most participants; however, it was not needed in some cases, and in other cases was not enough to meet clients' needs and encourage/enable them to finish their training.

• Impact on Home and Family Life — The ITCY training had a small impact on clients' home and family life.

• Preparation for Self-Sufficiency — The ITCYs biggest impact appears to have been on clients' attitude, motivation and self-esteem. Most clients also gained useful work experience and occupational skills, as well as job finding and life skills. The programs were not as successful in providing academic skills and a career plan.

• Achievement of Self-Sufficiency — Clients showed a steady increase over time in the proportion employed and in their weekly incomes. However, the ITCY programs appeared to have only a modest impact on clients' self-sufficiency, as measured by incremental gains in employment and income relative to the Comparison Group.

• Removal of Barriers — The longer clients trained, the better their employment outcomes tended to be; however, the ITCYs were not entirely successful in mitigating the effect of certain barriers to employment facing many clients (e.g., poor motivation, lack of transportation, low education, poor work history/lack of experience).

• Cost Effectiveness — The Integrated Training model is valued by case managers who believe it to be a more efficient and effective way to deliver services to highly barriered individuals. The partnerships forged by the agencies (e.g., with employers, with other community service providers) helped to bring more services to the client without increasing program costs.

• While results indicate that program participants fared somewhat better than the Comparison Group, it is doubtful that the incremental economic benefits would outweigh the higher costs associated with the training model. On follow-up, many clients indicated not using or needing the skills they learned in their current jobs. In addition, the jobs obtained by program participants were not substantially better than jobs obtained by the comparison group. These findings call into question the "value added" by the occupational skill training component of the programs.

Lessons Learned

Based on the results from the interim and final Outcome Evaluations, the following lessons have been learned from the Integrated Training Centres for Youth pilot project:

• There is a need for employment-focused services for youth that have dropped out of school. Existing services typically focus on adults, or provide alternative schooling for youth not coping well in the traditional educational system.

• Community stakeholders support the concept of Integrated Training. They view it as a more holistic, efficient and effective way to provide the range of services needed by youth at risk.

• Employers are not loath to become involved as trainers of youth at risk, and are prepared to take on a mentorship role that goes beyond the usual training-on-the-job function.

• Partnerships can be developed which enhance the services provided:

• The pilot projects suffered no major jurisdictional obstacles. On the contrary, the partnership between the Federal and Provincial government resulted in more creative and comprehensive programming than might have been mounted by either partner alone.

• The ITCYs are able to form partnerships with employers and other community service providers which reduce duplication and increase access to complementary services which otherwise would need to be provided in house.

• Incentives help to attract youth to training and to reduce their barriers to training. Incentives also provide an inducement for employers to take on youth to train, and for clients to stay in training for extended periods.

• The provision of incentives is problematic. It is difficult to implement a system based strictly on need. Also, projects, which provide incentives, can expect higher than normal dropout rates since the money will attract a certain proportion of participants who are not committed to pursuing long-term employment.

• Although clients were highly satisfied with their experience with Integrated Training, the programs were not fully successful in mitigating some of the barriers youth at risk face in gaining access to the job market.

• Integrated Training is a relatively expensive employment intervention. Substantial economic benefits must be realised in order for the government to obtain payback over a reasonable time horizon. The pilot projects did not appear to result in the necessary level of incremental benefits.

Recommendations are provided in regard to the conditions under which the ITCY pilot projects could be expanded or replicated to address problems facing young people in other areas.

1. Funders sponsoring employment-focused training programs involving youth at risk should carefully consider both the age range being targeted and the nature of the outcomes expected for the youth being served.

   The funding partners noted that the younger participants (16-18 years of age) generally had a lack of awareness about what is required to enter the "world of work".

   In fact it became apparent that a decision by a 17 year-old to return to school had to be considered "successful" in all but the strictest definition of the program.

2. Funders should consider extending the length of contracts involving "Integrated Training" interventions.

   The continuation of the program, with modifications, was achieved for a period of time in Edmonton, until funding realities intervened.

3. Funders sponsoring "Integrated Training" interventions should ensure broad consultation occurs with community stakeholders and service providers to avoid duplication of services and prevent perceptions of competition for clients between agencies.

   The "politics" between NGOs who compete for (government and other) funds is a reality not limited to ITCY, and is beyond the scope of our intervention.

4. Funders should experiment further with the use of financial incentives to enhance their potential to remove barriers and to encourage participants to complete their training.

   There were lessons learned from the incentive "non-model", and some funding formulas, with "room to manoeuvre" would have been preferable.

5. Funders sponsoring programs targeting youth "at risk" should ensure an effective assessment process is in place to determine level of risk, as well as clear eligibility criteria which establish the type of client to be accepted.

   There was no clear definition in the ITCY selection criteria as to what it meant for a youth to be "a risk". In subsequent programming efforts, employment barriers will need to be better identified.

6. Funders should ensure that programs, which contain an occupational skill training component, are well-articulated with the labour market to enhance the value for money received.

   The placement and follow-up component of ITCY generally did not reflect results significantly better than those achieved in previous traditional employment supports. A sharper focus on client selection, and more realistic employment situations will need to be targeted in future initiatives.

1.1 Background

In 1994, the Government of Canada announced its intention to renew and revitalise the country's social security system to create an environment that better rewards effort and offers incentives to work. To this end, Human Resources Development Canada (HRDC) launched the Strategic Initiatives (SI) program to provide a funding mechanism for the federal government to work in partnership with provincial and territorial governments to test new and innovative approaches in high priority areas of employment, education and income security. Projects supported by SI were funded on a 50/50 basis between the federal and provincial/territorial governments.

Negotiations took place between HRDC and the Alberta government departments of Family & Social Services (F&SS;) and Advanced Education & Career Development (AE&CD;) to identify projects that would be eligible for SI funding. Negotiations led to an agreement to fund the Integrated Training Centres for Youth (ITCY) pilot project.

Tenders were called in early 1995 for agencies to establish ITCYs. Three proposals were accepted, and contracts were negotiated with the following agencies:

1. Career High — a program of the Chinook School Division to provide training to youth at two sites in Red Deer and Innisfail;
2. Destinations — a partnership between Hennig Research and Consulting, Skye Consulting and the Northern Alberta Institute of Technology (NAIT) to provide training to youth in Edmonton;
3. Fifth on Fifth Youth Services — a program of the Lethbridge Youth Foundation to provide integrated training to youth in Lethbridge and area.

The focus of the ITCY pilot project was on youth that had dropped out of school and were having difficulty achieving significant labour force attachment. Youth interested in attending an ITCY had to meet basic Strategic Initiatives eligibility requirements:

• age 16 to 20;
• have less than grade 12 education;
• out of school for a minimum of 3 months with no intentions of returning;
• unemployed or underemployed;
• motivated to work.

The pilot project would test the value of customised training and work site interventions for youth that were at risk of long- term dependence on government support, in order to help them make a successful transition to employment.

The ITCYs began accepting clients in the spring and summer of 1995. A process evaluation commenced in May of 1995 under the supervision of an Evaluation Steering Committee made up of representatives from the three sponsoring departments. The final report was submitted in January 1996. HRDC published the final report entitled: Integrated Training Centres for Youth: A Process Evaluation in June of 1996.

As part of the contract for the process evaluation, the consultants were responsible for designing an outcome evaluation framework (Workplan for an Outcome Evaluation of the Integrated Training Centres for Youth. January 1996) complete with procedures for the selection/assignment of a comparison group, along with the forms and procedures needed to collect outcome data.

The consultants began work on the outcome evaluation in October 1996. An Interim Report was submitted to the department of AE&CD; in May 1997. The report consists of qualitative findings from interviews with a variety of ITCY stakeholders, including agency staff and clients, employers, government representatives and community agencies involved with youth. Key results from the Interim Report have been brought forward into this report to assist in drawing final conclusions (see Chapter 4.0).

1.2 Program description

AE&CD; wanted the ITCY pilot project to incorporate certain features of integrated training based on a model developed by the Center for Employment Training in the United States, for example:

• open entry/exit;
• individualised, self-paced training;
• intensive, hands-on, competency-based, job-specific skill training;
• academic skill training linked directly to occupational skills;
• industry and counselling-oriented instructors;
• integrated support services (financial incentives, mentoring, counselling, job placement, job maintenance).

The program emphasis was on integrating practical job and life management skills with ongoing coaching and support services tailored and sequenced to the individual needs of each participant.

Figure 1 below provides an overview of how clients typically access services at an ITCY. Various components of ITCY programming are described in Appendix A.²

1.3 Methodology

The outcome evaluation methodology combines two different designs:

• Applicant-Based Design — the comparison of variables between clients who received the Integrated Training (IT) intervention and other youth who did not. Clients were designated members of the Program Group (PG) if they received 2 or more weeks of training. Clients who applied to the program but did not complete at least 2 weeks of training were designated members of the Comparison Group (CG).

• Pre-Post Design — the measurement of changes in a set of "Baseline" attitudinal variables for both PG and CG clients.

Data for the outcome evaluation was collected primarily through a series of survey instruments (Appendix B) ³ delivered at different stages of the intervention:

• Baseline Survey — administered to PG and CG members at intake to capture family/employment history, demographic data and attitudes that might be affected by the intervention.

• Exit Surveys — administered to PG members upon completion of key stages of the intervention to identify services received client satisfaction and changes in baseline attitudinal variables. Exit A was to be administered after completion of training, and Exit B after a 4 month job maintenance period.⁴

• Follow-up Surveys — administered to PG and CG members at several points after intake to document employment status (Follow-up A) and changes in baseline attitudinal variables (Follow-up B).

Comparison Group

The original intention, as documented in the Outcome Evaluation Workplan, was to create an experimental "control group" in Edmonton whose members would be very similar to those in the PG. For example, it was originally proposed that eligible clients be assigned randomly to the two groups. A number of practical limitations arose which prevented implementing the experimental design as proposed. Project sponsors agreed to an alternative approach wherein the CG would be comprised of a range of youth, all "at risk"⁵ and otherwise eligible for the IT intervention, but not necessarily equivalent to those who eventually formed the PG. (See Appendix C for further discussion of changes to original experimental design.)

The conceptual flow of youth to the Program and Comparison Groups in Edmonton is outlined in Figure 2 below. The schematic also fits the Red Deer program, although procedures for administering the Baseline Survey were somewhat different. Also, Red Deer did not have a waiting list.

Data Collection

Client intake and job training commenced in July 1995. Clients whose intake was later than the October 1996 cut-off date were excluded from the outcome study. AE&CD; hired screeners/trackers (1 each for Edmonton & Red Deer) to administer all the data collection instruments.⁶ Although the workplan for the outcome evaluation outlined a detailed schedule for data collection, the trackers were not able to adhere to the schedule and, therefore, a significant amount of data was collected historically.⁷

Because of the delay in tracking CG members after their initial contact with the tracker, and in contacting PG clients after they had left the training program, many could not be located for follow-up interviews and response rates for individual months were low. In order to increase the representativeness of the measurement periods, a consolidation process was used to maximise the effective response rate. The data was consolidated at points 3, 6, 9, 12, 15, and 18 months from baseline. If data for a given month (e.g., month 3) was not available, the consolidation procedure used the data for the month prior (e.g., month 2). If data for that month was also not available, then data for the month following (e.g., month 4) was used.

Attitude measures were similarly grouped into 2 post-intervention periods:

1. Months 3 - 8 from baseline which would measure changes in attitudes that may be attributed to the intervention;
2. Months 11 - 13 from baseline which would measure changes and sustainability in attitudes 1 year from baseline.

Table 1 documents the response rates for each of the measurement instruments and their associated data periods.

Table 1 - Response Rates

Bias Between Responders and Non-Responders

Responders and non-responders were found to be very similar in both Edmonton and Red Deer. They were significantly different at the .01 level ⁸ on only 2 attitudinal variables. For example, in Edmonton responders were more likely than non-responders to indicate they had a lot of support around them. In Red Deer, responders were more likely to feel they had the skills to get a job. No correction for bias between responders and non-responders was considered necessary, and the results reported for the PG sample are considered representative of those who took the training.

Bias Between Program Group and Comparison Group

When non-random assignment of subjects to program and comparison groups is not feasible, as in this study, selection bias (due both to self-selection and program selection factors) presents a considerable challenge. More specifically, the problem of selection bias occurs when some determinant of earnings is correlated with one or more variables not associated with whether a person received training.

Two classes of variables must be considered: measured and unmeasured. Measured variables present the least difficult problem, in that standard statistical procedures (e.g., analysis of covariance) can account for their impact through multivariate regression techniques. Unmeasured variables pose a more difficult problem, and have been the subject of much discussion and analysis, particularly among econometricians investigating the impact of training programs.⁹

Two clusters of variables known to influence earnings, and which therefore present a potential source of selection bias, are demographic (e.g., age, race, gender, education, and prior work experience and wages) and motivational or attitudinal variables. Bell et al. have shown that the unmeasured components of these clusters can be reasonably dealt with through the use of non-participating program applicants (e.g., screen-outs and no-shows) as comparison subjects. These authors argue and successfully demonstrate that non-participating program applicants provide a reasonable alternative to random assignment in controlling for unmeasured components of selection bias in the evaluation of training programs, particularly when coupled with standard regression techniques to control for measured demographic variables. It is argued that unmeasured motivational/attitudinal variables are, a priori, controlled for in large part by using subjects who applied to the program but did not participate.

The argument by Bell et al. is made stronger if comparison subjects neither self-select out of the program nor are selected out by program staff. Those in the present study who were on the waiting list but were not invited to participate due to lack of space (Waiting List-Not Invited) fit this category. These subjects represent about 40% of the Edmonton CG, but none, unfortunately, of the Red Deer CG. In Edmonton, very few clients were selected out by program staff, mitigating possible bias from program selection factors.

The argument is also strengthening when motivational and attitudinal variables are explicitly measured at baseline, as was done in the present study. This enables the incorporation of these variables (along with relevant demographic variables which were also measured in this study), into the vector of covariates used to provide statistical control of variables associated with earnings.

The covariates consistently used to control for bias include:

• age;
• gender;
• education;
• ever had paid employment;
• mobility in 5 years prior to Baseline;
• living arrangement at Baseline;
• visible minority or aboriginal;
• criminal record;
• single parent;
• household history of SFI before client age 16;
• client history with SFI in 3 years prior to Baseline;
• employment history in year prior to Baseline;
• number of factors (out of 10) where a client would be considered "at risk" of needing Government assistance;
• mean attitude scores (work, self and life).

This chapter presents outcome evaluation findings for the Edmonton Destinations ITCY. Gross program effects are reported first, followed by incremental impacts.

2.1 Gross Effects

(a) Client Satisfaction

Table 2 presents information taken from Exit A (n = 138) and Exit B (n = 123) surveys relating to clients' experience with their training.

Table 2 - Client Satisfaction with Training

With a mean satisfaction rating of 8.1 out of 10, Edmonton clients were highly satisfied with the program. It is interesting, however, that of those who had taken another training program in the past, only 62% felt the Destinations program was better.

Relatively few clients (17%) had a problem during training that they needed help with. All of these clients felt the agency tried to help, and all were satisfied with the help they received. The majority of clients (72%) indicated having a role model¹¹ while going through the program.

Essentially all clients received incentive money from the agency, and the majority (77%) felt the money was enough to meet their needs. It is interesting, however, that only 37% of clients indicated they would have quit the program had they stopped receiving the incentive, possibly suggesting the incentive was not absolutely necessary in all cases.¹²

The majority of clients were satisfied with specific agency services:

• support received while training off site (84%);
• agency efforts to help them find a job (93%); and
• support received while adjusting to the job (89%).

Most (80%) were also satisfied with their first job as a place to start their career.

Of those contacted who left the program before completing their training, the majority (65%) indicated dropping out on their own rather than being asked to leave by the agency. The main reasons for dropping out include:

• wasn't learning anything (21%);
• had transportation problems (21%);
• got a job (21%);
• needed money, had to get a job (17%);
• had family problems (17%).

(b) Skills Acquired

Table 3 presents information about the skills clients indicated learning in the program.

Table 3 - Program Impacts – Skills Acquired

A greater proportion of clients learned life skills and job skills than learned academic skills. This result is not surprising in view of the fact that academic skills were not stressed in the program.¹³ In terms of job finding skills, significantly more learned how to look for a job than how to go into business for themselves. Again, entrepreneurial skills were not stressed in the program.

It took just under 2 weeks on average for Destinations clients to obtain their first job after training. Of those who were working at the time of the Exit B survey, the majority (76%) indicated they were using skills they had learned in the program. It is interesting, however, that 40% felt they could have gotten their current job without the training they received, suggesting that many may have been working at low-skilled/entry level jobs and/or jobs unrelated to their training.

(c) Changes in Lifestyle and Attitudes

Table 4 shows that the majority of clients (83%) attributed improvements in their attitude/motivation to the training program. Some clients also noted improvements in their family relationships and health.

Table 4 - Program Impacts – Lifestyle Improvements

Tables 5 provides information about changes in client attitudes by comparing pre-program information (Baseline) with information collected at various points after Baseline (Follow-up). The results suggest that attitudes did not change for the better after the program. In fact, clients' attitudes about life¹⁴ appeared to have deteriorated in the year after Baseline.

Table 5 - Program Impacts – Changes in Attitude (pre/post)

Table 6 shows that clients' involvement in recreation/social activities did not appear to change significantly after the program.

Table 6 - Program Impacts – Changes in Activities (pre/post)

(d) Employment Trends

Figure 3 shows the status of Edmonton clients at Baseline and at various points after Baseline as reported on Follow-up surveys. The results show a marked improvement in the proportion of clients employed¹⁵ over time, levelling off at approximately 65%, while the proportion unemployed¹⁶ declines correspondingly to about 20%. The proportions in school/training levels off at approximately 20%. The proportion on SFI stays relatively stable at roughly the level it was at Baseline (i.e., slightly under 20%).

Figure 3 - PG Status from Baseline to Month 18

The main reasons clients gave for being unemployed when contacted on follow-up include:

• lack of experience (24%);
• lack of education (19%);
• no jobs available (19%);
• no transportation (10%).

The first two reasons are somewhat surprising in view of the fact the clients just went through a training program.

(e) Income Trends

Figure 4 shows the average weekly income of Edmonton clients at various points after Baseline. The trend is clearly towards an increase in income over time. This can be attributed to steady increases over time in the proportion employed, the average hours worked per week and the average hourly wage earned (see Table 7 on the following page).

Figure 4 - PG Average Weekly Income

Table 7 - Trends in Employment and Income

(f) Most Successful Participants

The average weekly income for all program participants at 12 months (including those not working) was $143. Certain subgroups did better than others. The characteristics most strongly correlated to better incomes include:

• > Grade 8 education: $150
• ever had previous paid employment: $156
• household not on SFI (when client < age 16): $169
• aged 19 – 21: $170
• male: $171
• employed in year before Baseline: $173
• had a criminal record: $176

Except for the last characteristic (i.e., criminal record), it appears the program was not fully able to remove the traditional barriers to employment faced by many of the clients (i.e., younger, female, poor education, poor work history, previous SFI history, etc.).

Table 8 provides the characteristics of the jobs held 12 months from Baseline by clients who received different levels of training in the program.¹⁷ The results suggest that those who trained longer had better outcomes in terms of employability (i.e., proportion employed) and quality of employment (i.e., average weekly income, permanent job, opportunity for advancement).

Table 8 - Impact of Length of Training on Employment

2.2 Incremental Effects

While Figures 3 and 4 indicate that Program Group participants enjoyed steady improvement over time in their employability and employment incomes, it is necessary to contrast their results with those of the Comparison Group to determine the extent to which these improvements can be attributed to the ITCY training they received.

(a) Employment Characteristics

Table 9 compares the characteristics of the jobs held by PG and CG members 12 months from Baseline. The proportion of PG members working is greater (61% vs. 48%). For those who were employed, however, CG members appeared to fare slightly better in terms of weekly income from employment, although fewer found jobs providing benefits and fewer were satisfied with their hours/hourly rate. The findings suggest that PG members did not find jobs that were substantially better than those found by CG members.

Table9 - PG-CG Comparison of Employment Characteristics

(b) Employability

Figure 5 provides longitudinal data comparing the proportion of PG and CG participants employed at various points after Baseline.

Figure 5 - PG and CG Employment Status from Baseline to Month 18

The results clearly show that, like PG members, CG members also exhibited employability improvements over time. This finding was not unexpected, since the job market in the province was relatively buoyant at the time of the study. Furthermore, CG members were motivated to find employment and had applied to the ITCY in order to obtain training to help them find a job.¹⁸ The trend lines suggest, however, that once graduating from the ITCY, a greater proportion of the PG cohort was able to maintain employment. A 10% to 20% difference between the groups is sustained over time. This is demonstrated in Figure 6 on the following page that compares adjusted¹⁹ employment rates averaged over the 9 to 15 month period from Baseline.²⁰ The mean difference is 17%.

Figure 6 Adjusted PG and CG Employment Rates (mean of means for months 9, 12 and 15)

(c) Income

Figure 7 on the following page shows the adjusted incremental gains in weekly income of the PG cohort over time.²¹ The figure clearly demonstrates that the PG cohort had higher employment earnings than the CG cohort. This is further demonstrated in Figure 8 which compares adjusted weekly incomes for the two groups averaged over the 9 to 15 month period from Baseline. The PG earned an average of $63 more per week than the CG over this period.

Figure 7 Adjusted Incremental Gains in Average Weekly Income

Figure 8 Adjusted Average Weekly Income (mean of means for months 9, 12 and 15)

Other Outcomes

SFI Involvement

Table 10 provides information comparing PG and CG members in terms of the percentage who collected SFI and the average monthly benefit amount each cohort collected before and after Baseline.

Table 10 - SFI Claim Activity

It is interesting that the proportion of PG members collecting SFI jumped considerably after Baseline (i.e., from 18.5% to 32.9%), ²⁴ yet the average monthly benefit amount decreased. For the CG, both the proportion collecting and the average amount collected increased after Baseline. These findings suggest PG members may have become less reliant on SFI while the CG became more reliant, resulting in "savings" to the Province of approximately $9.85 per month per PG participant.²⁵

EI Involvement

Table 11 compares PG and CG members in terms of the percentage of the cohort who collected EI and the average amount each cohort collected before and after Baseline.

Table 11 EI Claim Activity²⁶ (August/93 — June/97)²⁷

It appears that PG members increased their involvement with EI when considering the proportion collecting in the post-Baseline period. The average amount collected per claim went down considerably, although the average amount of EI collected per year per PG member stayed about the same. CG members appeared to reduce their involvement in EI, both in terms of the proportion who claimed and the average amount collected.

Police Involvement

Table 12 compares PG and CG members in Edmonton in terms of their involvement with the Edmonton Police Service (EPS) over time. The results show a slight mitigation in police involvement for PG members over time compared to CG members.

Table 12 - Edmonton Police Service Activity

This chapter presents outcome evaluation findings for the Career High ITCY in Innisfail/Red Deer. Gross program effects are reported first, followed by incremental impacts.

3.1 Gross Effects

(a) Client Satisfaction

Table 13 - presents information taken from Exit A (n = 88) and Exit B (n = 87) surveys relating to clients' experience with their training.

Table 13 - Client Satisfaction with Training

With a mean satisfaction rating of 8.0 out of 10, Career High clients were highly satisfied with the program. Of those who had taken another training program in the past, the majority (81%) felt the Career High program was better.

Only 15% of clients had a problem during training that they needed help with. Just under half (46%) of these clients felt the agency tried to help them, although all who were helped were satisfied with the help they received. The majority of clients (68%) indicated having a role model while going through the program.

Most clients (87%) received incentive money from the agency. The majority (89%) of these felt the money was enough to meet their needs, although only 26% indicated they would have quit the program had they stopped receiving the incentive, suggesting the incentive was not absolutely necessary in all cases.

Clients were highly satisfied with specific agency services:

• support received while training off site (98%);
• agency efforts to help them find a job (93%); and
• support received while adjusting to the job (96%).

Most (81%) were also satisfied with their first job as a place to start their career.

Of those who left the program before completing their training, just over half (55%) indicated they dropped out on their own. The rest indicated they were asked to leave. The main reasons for dropping out include:

• wasn't learning anything (35%);
• went back to school (35%);
• transportation problems (22%);
• got a job (22%).

(b) Skills Acquired

Table 14 - presents information about the skills clients indicated learning in the program.

Table 14 - Program Impacts — Skills Acquired

A greater proportion of clients learned life skills and job skills than learned academic skills.³³ In terms of job finding skills, significantly more learned how to look for a job than how to go into business for themselves. This is not surprising, as entrepreneurial skills were not stressed in the program.

It took just over 4 weeks on average for Career High clients to obtain their first job after training. Of those who were working at the time of the Exit B survey; only 57% indicated they were using skills they had learned in the program. Furthermore, over half (52%) felt they could have gotten their current job without the training they received, putting into some question the linkage between the training and the nature of the jobs being found.³⁴

(c) Changes in Lifestyle and Attitudes

Table 15 shows that the majority of clients (84%) attributed improvements in their attitude/motivation to the training program. Some clients also noted improvements in their family relationships and health.

Table 15 - Program Impacts — Lifestyle Improvements

Tables 16 provides information about changes in client attitudes by comparing pre-program information (Baseline) with information collected at various points after Baseline (Follow-up). The results suggest that attitudes did not improve appreciably after the program.

Table 16 - Program Impacts — Changes In Attitude (pre/post)

Table 17 suggests that clients' involvement in recreation activities may have increased somewhat after the program.³⁵

Table 17 - Program Impacts - Changes In Activities (pre/post)

(d) Employment Trends

Figure 9 shows the status of Career High clients at Baseline and at various points after Baseline as reported on Follow-up surveys. The results show a marked improvement in the proportion of clients employed over time, levelling off at approximately 70%, while the proportion unemployed declines correspondingly to about 20%. The proportions in school/training levels off at approximately 20%. The proportion on SFI drops to about 10%.

Figure 9 PG Status from Baseline to Month 18

The main reasons clients gave for being unemployed when contacted on follow-up include:

• don't need/want to work (25%);
• no jobs available (24%);
• waiting for job to start (11%);
• pregnant/on maternity leave (11%).

The first reason is somewhat surprising given the ITCYs were expected to screen out clients who were not motivated to work.

(e) Income Trends

Figure 10 shows the average weekly income of Career High clients at various points after Baseline. The trend is clearly towards an increase in income over time. This can be attributed to steady increases over time in the proportion employed, the average hours worked per week and the average hourly wage earned (see Table 18 on the following page).

PG Average Weekly Income

Table 18 - Trends in Employment and Income

(f) Most Successful Participants

The average weekly income for all program participants at 12 months (including those not working) was $132. Certain subgroups did better than others. The characteristics most strongly correlated to better incomes include:³⁶

• ever had previous paid employment: $145
• employed in year before Baseline: $153
• aged 19 – 21: $179

It appears the program was not fully able to remove the traditional barriers to employment faced by many of the clients (i.e., younger, poor work history).

Table 19 provides the characteristics of the jobs held 12 months from Baseline by clients who received different amounts of training in the program. The results suggest that those who trained longer had better outcomes in terms of employability (i.e., proportion employed) and quality of employment (i.e., average weekly income, permanent job, opportunity for advancement).

Table 19 - Impact of Length of Training on Employment

3.2 Incremental Effects

(a) Employment Characteristics

Table 20 compares the characteristics of the jobs held by PG and CG members 12 months from Baseline. The proportion of PG members working is somewhat greater (68% vs. 62%). For those who were employed, however, CG members appeared to fare slightly better in terms of weekly income from employment, although other job characteristics were not noted quite as frequently (i.e., permanent job, job with benefits, opportunity for advancement). The findings suggest that PG members did not find jobs that were substantially better than those found by CG members.

Table 20 - PG-CG Comparison of Employment Characteristics

(b) Employability

Figure 11 on the following page provides longitudinal data comparing the proportion of PG and CG participants employed at various points after Baseline.

The trend lines suggest that a greater proportion of the PG cohort was able to maintain employment over time. This is also shown in Figure 12 which compares adjusted employment rates averaged over the 9 to 15 month period from Baseline. (Note: Because of the relatively small sample sizes, the differences shown in Figures 11 and 12 are not statistically reliable.)

Figure 11 PG and CG Employment Status from Baseline to Month 18

Figure 12 Adjusted PG and CG Employment Rates (mean of means for months 9, 12 and 15)

(c) Income

Figure 13 below shows the adjusted incremental gains in weekly income of the PG cohort over time. The figure demonstrates an inexplicable reversal of the trend at month 12 which compromises the ability to conclude that the PG cohort had higher employment earnings than the CG cohort. This is further demonstrated in Figure 14 on the following page which compares adjusted weekly incomes for the two groups averaged over the 9 to 15 month period from Baseline. In this figure, the PG is shown to have earned an average of $44 less per week than the CG over the period, once making adjustments for bias between the two samples.

Figure 13 Adjusted Incremental Gains in Average Weekly Income

Figure 14 Adjusted Average Weekly Income (mean of means for months 9, 12 and 15)

(d) Other Outcomes

Table 21 provides information comparing PG and CG members in terms of the percentage who collected SFI and the average monthly benefit amount each cohort collected before and after Baseline.

Table 21 - SFI Claim Activity

The CG and PG showed similar increases in the proportion who collected SFI after Baseline. Both also showed a slight decrease in the average monthly benefit collected, although the benefits to the PG declined more, resulting in "savings" to the Province of approximately $4.10 per month per PG participant.

Table 22 compares PG and CG members in terms of the percentage of the cohort who collected EI and the average amount each cohort collected before and after Baseline.

It appears that PG members increased their involvement with EI when considering the proportion collecting in the post-Baseline period. The average amount collected per claim went down considerably, although the average amount of EI collected per year per PG member stayed about the same. CG members had no claim activity before Baseline (see footnote 27 on page 21).

Table 22 - EI Claim Activity (August/93 — June/97)

The Outcome Evaluation was governed by a set of evaluation questions which were used to focus our investigation. In this section we summarize both qualitative findings from the Interim Evaluation report and quantitative findings presented in this report to draw our conclusions relating to the key issues guiding the study.

1. Participant Satisfaction

Conclusion: Participants were generally satisfied with the project.

• Qualitative Findings — Clients were very satisfied with their ITCY training experience,³⁷ expressing that the services they received met their individual needs and that they had come away with valuable skills and experience. Employers were also generally satisfied with their role and with the clients they trained or hired. They indicated feeling like "partners", engaged in a different and deeper way (e.g., mentoring) than in other TOJ-type programs they had been involved in.

• Quantitative Findings — Clients gave the programs high ratings and were generally satisfied with the services and support they received. Relatively few clients indicated needing help with a problem during training, and where the agency tried to help, the clients were satisfied.

2. Reasons for Discontinuation

Conclusion: The main reasons participants discontinued training include: poor motivation/lack of interest; having transportation barriers; getting a job; going back to school; and having family/financial problems.

• Qualitative Findings — Agency staff cited poor motivation/attitude as the main reason clients did not complete. Other reasons included: going back to school, getting a job and having family/health problems. Employers also identified poor motivation/attitude as one reason clients left their training placement, but also noted that many left because there was a poor match between the placement and the client's career interests.

• Quantitative Findings — The main reasons clients gave for dropping out of the program included getting a job and going back to school, and also having transportation, family or financial problems, as well as feeling they were not learning anything useful.

3. Impact of Incentives

Conclusion: The provision of financial incentives was instrumental in attracting many clients to the training. The incentive was adequate for most participants; however, it was not needed in some cases, and in other cases was not enough to meet clients' needs and encourage/enable them to finish their training.

• Qualitative Findings — Both clients and agency staff clearly indicated that the incentive attracted many youth to the project. ³⁸ Many clients found the incentive money provided the financial stability that enabled them to train for an extended period of time. On the other hand, some youth attended "just for the money" and were never strongly committed to employment. At the same time, it was noted that certain clients (e.g., single parents) quit because they were not receiving as much as they could on SFI, or they could earn by working at the same job they were training at. In these cases, the amount was not a strong enough incentive for them to stay on and finish their training. Agencies indicated that they felt constrained to work within the prescribed grid and had difficulty individualizing the amount of incentive for each client. The incentive became more of an entitlement than an amount based on need or progress.

• Quantitative Findings — The majority of clients reported receiving money from the agency. Most said it was enough to meet their needs.³⁹ At the same time, however, many indicated they would have handled their needs in other ways, and the majority would not have quit training had they not received the money, suggesting the incentive was not absolutely necessary in all cases.

4. Impact on Home and Family Life

Conclusion: The ITCY training had a small impact on clients' home and family life.

• Qualitative Findings — Only a few clients indicated their family life, relationships or recreation had changed much as a result of the training. Clients were more likely to indicate they had less time to socialize/recreate because of the training.

• Quantitative Findings — A minority of clients attributed improvements in family relationships, health or recreation to the training program.

5. Preparation for Self-Sufficiency

Conclusion: The ITCYs biggest impact appears to have been on clients' attitude, motivation and self-esteem. Most clients also gained useful work experience and occupational skills, as well as job finding and life skills. The programs were not as successful in providing academic skills and a careerplan.

• Qualitative Findings — Clients typically felt they had made significant strides towards self-sufficiency as a result of the training they received. They were most likely to indicate that the program had made a positive impact on their attitude, motivation, self-esteem and confidence. Others commented on improvements in their job finding skills. Employers were generally satisfied with the clients in terms of their job finding and occupational skills as well as their attitude and motivation, although they felt many clients suffered from low self-esteem and poor life management skills stemming from their dysfunctional upbringing. Employers were less likely to be satisfied with clients in terms of their level of education, seeing this as a barrier to longer-term career development. Employers were least satisfied with clients in terms of their career orientation, noting a lack of career planning and long-term goals.

• Quantitative Findings — The majority of clients noted improvements in their attitude and motivation as a result of the training. The majority also indicated acquiring job finding, occupational and life skills as well as knowledge about employer expectations and the training/ experience required to pursue a long-term career path. Clients were less likely to indicate they acquired academic skills they could use on the job.

6. Achievement of Self-Sufficiency

Conclusion: The ITCY programs appeared to have a modest impact on clients' self-sufficiency, as measured by incremental gains in employment and income relative to the comparison group.

• Qualitative Findings — Employers were quite optimistic about the future prospects for the clients they trained/hired, but still felt a significant number (e.g., those with poor attitude/motivation, negative lifestyles, "welfare" mentality) would probably be on social assistance in the future.

• Quantitative Findings — Clients showed a steady increase over time in the proportion employed and in their weekly incomes. During the same period, however, the comparison group also demonstrated similar (although smaller) gains.⁴⁰ Furthermore, the jobs obtained by program participants were not substantially better than jobs obtained by the comparison group. Consequently, the programs appear to have had only a modest impact on clients' self-sufficiency over what they might have accomplished in the job market on their own.

7. Removal of Barriers

Conclusion: The longer clients trained, the better their employment outcomes tended to be; however, the ITCYs were not entirely successful in mitigating the effect of certain barriers to employment facing many clients (e.g., poor motivation, lack of transportation, low education, poor work history/lack of experience, young, female).

• Qualitative Findings — Staff felt there would always be a segment of the youth population who would have difficulty maintaining employment because they lacked the motivation to work.

• Quantitative Findings — The longer program participants trained, the better their employment outcomes tended to be (i.e., percentage employed, average weekly income). Clients who were not working gave a wide variety of reasons. Two of the more common reasons were:

• not enough experience/education (Destinations);
• don't need/want to work (Career High).

These reasons were somewhat surprising in view of the fact the clients had just gone through a training program focusing, among other things, on skills and motivation. Other reasons included: in school/training (or looking into school/training), lack of jobs, lack of transportation, moving, waiting for job to start, pregnant/on maternity leave.

Characteristics of clients who were least successful in terms of employment income included: younger (15 — 18), female, low education, poor work history, previous SFI history.

8. Cost Effectiveness

Conclusion: The Integrated Training model is valued by case managers who believe it to be a more efficient and effective way to deliver services to highly barriered individuals. While results indicate that program participants fared somewhat better than the control group, it is doubtful that the incremental economic benefits would outweigh the higher costs associated with the training model.

• Qualitative Findings — "Integrated Training", where one agency is able to provide (or broker) a range of services, was seen by stakeholders as a more efficient and effective way to provide employment-oriented services to high risk youth. It was viewed as more seamless and holistic for clients to deal with one agency rather than shuttle from program to program. Referral sources would either have to ignore certain needs because services were not available, or work to move the client from service to service, making case management more expensive.

• The partnerships forged by the agencies (e.g., with employers, with other service providers) also helped to bring more services to the client without increasing program costs. This aspect of the pilot projects was seen to have "raised the bar" in terms of expectations for other contractors. The pilot experience was helpful in negotiating new contracts, pushing agencies to economize, to provide more comprehensive service, and generally to provide greater value for money.

• Quantitative Findings⁴¹ — The ITCY programs were found to have an average training cost of approximately $4,200 per participant.⁴² The modest incremental income gains by the Program Group would not be expected to yield substantial taxation revenue to government.⁴³ The relatively small involvement of study participants found with the Edmonton Police Service and with EI make it difficult to identify "savings" in these areas that can reliably be attributed to the training programs. More comprehensive data was available relating to the involvement of study participants with SFI; however, the savings identified here were relatively small, such that "payback" of training costs could not likely be projected within a reasonable period of time.

  Integrated training is relatively expensive in comparison with other employment interventions (e.g., job clubs, placement programs), due in part to the additional costs of providing occupational skill training. ⁴⁴ The ITCYs were expected to establish strong linkages with industry so that the training would target occupations in demand and provide relevant job skills that would make clients competitive somewhat above the entry level. The survey results indicated that many clients were not using the skills they learned, and felt they could have gotten their current job without the training they received. In addition, the jobs obtained by program participants were not substantially better than jobs obtained by the comparison group. These findings call into question the “value added” by the occupational skill training component of the programs.

9. Lessons Learned

Based on the results from the interim and final Outcome Evaluations, the following lessons have been learned from the Integrated Training Centres for Youth pilot project:

• There is a need for employment-focused services for youth who have dropped out of school. Existing services typically focus on adults, or provide alternative schooling for youth not coping well in the traditional educational system.

• Community stakeholders support the concept of Integrated Training. They view it as a more holistic, efficient and effective way to provide the range of services needed by youth at risk.

• Employers are not loath to become involved as trainers of youth at risk, and are prepared to take on a mentorship role that goes beyond the usual training-on-the-job function.

• Partnerships can be developed which enhance the services provided:

• The pilot projects suffered no major jurisdictional obstacles. On the contrary, the partnership between the Federal and Provincial governments resulted in more creative and comprehensive programming than might have been mounted by either partner alone.

• The ITCYs are able to form partnerships with employers and other community service providers which reduce duplication and increase access to complementary services which otherwise would need to be provided in-house.

• Incentives help to attract youth to training and to reduce their barriers to training. Incentives also provide an inducement for employers to take on youth to train, and for clients to stay in training for extended periods.

• The provision of incentives is problematic. It is difficult to implement a system based strictly on need. Also, projects which provide incentives can expect higher than normal dropout rates since the money will attract a certain proportion of participants who are not really committed to pursue long-term employment.

• Although clients were highly satisfied with their experience with Integrated Training, the programs were not fully successful in mitigating some of the barriers youth at risk face in gaining access to the job market.

• Integrated Training is a relatively expensive employment intervention. Substantial economic benefits must be realized in order for the government to obtain payback over a reasonable time horizon. The pilot projects did not appear to result in the necessary level of incremental benefits.

The final requirement of the study was to comment on the conditions under which the ITCY pilot projects could be expanded or replicated to address problems facing young people in other areas. In this regard, we make the following specific recommendations stemming from process and outcome investigations relating to design, implementation, delivery and results of the pilot project:

1. Funders sponsoring employment-focused training programs involving youth at risk should carefully consider both the age range being targeted and the nature of the outcomes expected for the youth being served.

Rationale: Youth under the age of 18 pose complications for training programs that have employment outcome expectations. Younger participants generally did not fare as well as older participants in terms of employment and income. The agencies found this group more difficult to serve, as they were seen to be less mature, less willing/able to make a commitment towards employment, and less credible/desirable than older participants in the eyes of employers. There was a strong undercurrent of sentiment among stakeholders that "going back to school" is not necessarily a negative outcome for youth, particularly those of younger age. Indeed, it may be short-sighted to ask youth to commit to employment when it risks them abandoning or significantly postponing completion of their high school education.

Programming for younger participants (18 and under) might focus on part-time school with part-time work, particularly since funding from the public education system can be accessed for the educational upgrading component. This would allow program resources to focus on employment development and maintenance. The Career High ITCY was a good example of such a program.

The outcome expectations for older participants might focus more on employment. Programming strategies would emphasize life and job skill training, job placement and maintenance, and provide only the academic skills that are directly needed for the specific career/occupation the participant is training for. The Destinations ITCY was a good example of such a program.

In any case, contracts will need to be clear on the conditions under which returning to school can be counted as a success. Similarly, contracts will need to stipulate the time frame within which employment is expected to be gained by participants after they complete the program.

2. Funders should consider extending the length of contracts involving "Integrated Training" interventions.

Rationale: Integrated Training involves making many service components available to participants (e.g., assessment, counselling, life skills, job skills, academic skills, incentives, job placement, job maintenance, etc.), either directly at the agency, or brokered through partnership arrangements with other service providers in the community. Furthermore, these services are expected to be provided seamlessly, holistically, on an individualized basis, with open entry and exit of participants.

Such programs are complex. It takes time to set up internal resources, curricula and procedures as well as referral and partnership arrangements with other community agencies. It also takes time to fine-tune these operating parameters. Contracts should ideally be established to last a minimum of 2 to 3 years, particularly new or pilot projects which bear an additional developmental burden.

3. Funders sponsoring "Integrated Training" interventions should ensure broad consultation occurs with community stakeholders and service providers to avoid duplication of services and prevent perceptions of competition for clients between agencies.

Rationale: Because Integrated Training is a "full service" strategy, it will very likely comprise service components which are already provided in the community in some form, services which have available capacity and which are of a high standard of quality. This creates the potential for competition (or perceived competition) for clients with other established agencies.

Integrated Training may take a different form in each community as decisions are made whether given service components are to be provided internally, or brokered externally through partnerships and referral agreements with other service providers. Of particular importance is the definition of the target group intended to be served so that referral agents make appropriate referrals and there is no confusion or disagreement between agencies as to which youth are to be served by various programs in the community.

4. Funders should experiment further with the use of financial incentives to enhance their potential to remove barriers and to encourage participants to complete their training.

Rationale: The agencies found it problematic to individualize the amount of incentive payment based on the needs of each participant. A grid was used which, while satisfactory, appears to have paid some participants too much and others too little to meet their needs. Furthermore, the agencies did not use incentives creatively to any significant extent to provide bonuses for participant performance or progress.

The use of a grid makes sense to provide a base level of funding to participants based on their family composition and living situation. Contractors should be allowed and encouraged, however, to make additional ad hoc payments for special needs (e.g., bus passes, baby sitting, damage deposits, moving expenses, work clothing, etc.) to facilitate the ability to address specific barriers that often arise while barriered clients are in training.

The base incentive would likely be set less than the welfare rate, and certainly below the income level at minimum wage. The study found that some clients left training early because they could make more money on welfare or by working. Contractors should therefore be allowed to experiment with providing bonuses or increasing the base incentive rate as clients progress through training to motivate them to attend and to complete their training before looking for employment.

5. Funders sponsoring programs targeting youth "at risk" should ensure an effective assessment process is in place to determine level of risk, as well as clear eligibility criteria which establish the type of client to be accepted.

Rationale: The guidelines for eligibility were quite broad and did not attempt to differentiate or set priorities for different levels of risk. Consequently, a wide range of clients were accepted into the ITCY programs. This was acceptable for purposes of the pilot project, but may need to be refined for future programming focused on the at risk population. For example, the study found a number of risk factors associated with clients who were less successful than others in terms of employment income (e.g., female, low education, poor work history, previous family/personal SFI history). Programs might be designed specifically to target such clients who are considered "most at risk" based on information they provide about their backgrounds at the point of intake/assessment.

In conjunction with Recommendation #1, the assessment process used by such programs needs to more accurately determine a youth's attachment to school and inclination towards work and career. A certain proportion of youth who meet the criteria for being a "high school drop out", nonetheless will return to school during or shortly after completing training.

Furthermore, the provision of incentives tends to attract participants who may not be career focused. The assessment process should ideally screen out such cases at the earliest possible point.

6. Funders should ensure that programs which contain an occupational skill training component are well-articulated with the labour market to enhance the value for money received.

Rationale: Occupational skill training increases the length and cost of training, particularly if financial incentives are being paid to participants during this phase of training. The enhanced skills provided should result in significant incremental benefits if economic payback is to be realized in a reasonable time frame. The study pointed to modest incremental benefits for ITCY participants, but they did not fare substantially better than the comparison group. On follow-up, many clients indicated not using or needing the skills they learned in their current jobs, placing into question the "value added" from the occupational skill training they received.

Participants who receive occupational skill training should expect to find long-term work available in the same field, and should ideally be placed somewhat above the entry level. Otherwise, there is little justification for going beyond more traditional (and less expensive) placement and work experience programming.

Where occupational skill training is provided at employer sites, quality control is essential (e.g., training placements should be governed by a competency-based training plan). Career exploration and planning are also important to ensure that clients are truly interested in working in the field in which they are training and can be properly matched with employer hosts. Otherwise, they may not complete their training or follow through with employment after completion.

Appendix A




Description of Program Procedures

Intake and Assessment

Eligible youth were invited to a scheduled intake session provided there was space available. The first 2 weeks of the program were considered a formal assessment or probationary period where clients' barriers, needs and level of commitment to the program were identified. Clients could be asked to leave the program at this point for a variety of reasons, depending on each agency's rules and ability to address client problems.

In order to decrease disincentives and increase incentives to train, ITCYs received funding to be used as a client training allowance. The incentive was intended to provide for clients' basic needs while in training. Each participant could access up to a maximum of $4,500.00 in incentive dollars over a period of 8 months.

Training

In-house training typically focused on generic transferable skills. The amount and type of training clients received was based on need, and focused on reducing barriers to employment. Occupational training was competency-based and delivered off-site at an employer's job-site in the form of job shadowing (JS), work experience (WE), or training on the job (TOJ) opportunities.⁴⁵ The Edmonton ITCY offered training in specific occupational areas (hospitality/retail/tourism, clerical/ office, construction, automotive) that were identified through labour market research as having the best employment opportunities for this client group. The Red Deer ITCY did not limit clients' choice of occupation to pre-selected industries.⁴⁶ The only constraint was the availability of job-site training opportunities. ITCY clients were able to move back and forth between in-house and off-site training, depending on their individual needs.

Job Placement and Maintenance

The ITCYs tried to arrange job-site training opportunities with employers who would be in a position to hire a client after they had completed their training. If a client could not find employment after training, they may have engaged in another job-site training experience. Each new opportunity for training was intended to build on the last and eventually lead to full-time employment. Once clients were employed, job maintenance and support was provided as necessary for 4 months.

Appendix B - Survey Instruments




Strategic Initiatives
Baseline Survey

---

About This Survey

The information in the attached survey is being collected as part of a nation-wide research study. The study is being done to improve employment programs in Canada.

In Alberta, we are trying to follow young people who have dropped out of school to see what services they use over time, and which services help them to get meaningful, long-term employment.

We are also asking for your Social Insurance Number to help get information from the provincial or federal government about services used and the employment or income of young people who participate in the study.

Your participation in this survey is completely voluntary.

If you refuse to complete the survey, it will not in any way hurt your chances of getting federal or provincial services now or in the future, nor will it have any influence on whether or not you get into this or any other program. The information you provide will be kept strictly confidential and will only be shared with the professional staff at (the Destinations Project/Red Deer Regional Integrated Training Program) 5th on 5th Youth Services should you (participate in that program) attend the Integrated Training Centre for Youth. The results reported from the study will be summarized for groups only (e.g., males, females, etc.) and will not be linked back to you personally.

Any information you can provide is very important to us in the continuing improvement of employment programs in Alberta.

Thank you for your assistance.

---

"See PDF for survey"



Appendix C




Changes to Experimental Design

The original intention was to create an experimental "control group" in Edmonton whose members would be identical to those in the PG. For example, it was originally proposed that eligible clients be assigned randomly to the two groups. Other adjustments were also proposed to further protect against sampling bias:

1. Not transferring to the CG those youth who opted not to participate in IT when invited, or who dropped out or were asked to leave, on the basis that they might not have been appropriate for IT and therefore not appropriate for the CG;
2. Adjusting the CG to exclude its "worst performing" members ⁴⁷ in direct proportion to the number of PG members who dropped out or were asked to leave, on the basis that the PG would gradually become biased in favour of individuals with better outcomes.

A number of practical limitations arose which prevented implementing the experimental design as proposed.

In the first place, Edmonton Destinations balked at the prospect of a random assignment of eligible clients to their program. They were concerned about a possible adverse impact on their image among youth if it were learned clients were being randomly selected for participation. A compromise solution was proposed whereby eligible clients would be placed on a "waiting list" from which they would be invited on a first-come basis to attend IT as spaces became available. Intake to the waiting list would continue in the hope of building a backlog of clients whom the program could not serve. This group who remained on the waiting list would then constitute the CG.

The waiting list approach has a number of drawbacks. For example, timing differences would be introduced in that members of the PG would come from an earlier period, on average, than members of the CG, leading to concerns about seasonal effects if the job market changed over the period in question. It was also feared that being on the waiting list might result in clients "doing nothing" while they waited for admission to IT. Such individuals could be expected to have poorer outcomes than those who received an intervention. It was hoped that a substantial proportion in the CG would enroll in other employment interventions, thereby providing a better baseline against which IT could be evaluated.⁴⁸ These limitations were discussed and accepted by program sponsors.

A sample of at least 300 in the CG was desired to provide for attrition. It became clear in the first few months of program operation that the volume of referrals was lower than had been hoped. Consequently, the waiting list was being drawn down as fast as it was being filled, raising the prospect that there would not be sufficient numbers of youth to form a CG. Two measures were instituted to address this problem:

1. Firstly, it was agreed that PG clients who dropped out, were pushed out or were invited but opted not to come, were to be transferred back to the CG;
2. Secondly, youth coming to the Edmonton Youth Employment Centre seeking services would be actively recruited to participate as members of the CG, with no option that they would be invited to enroll in IT. This recruitment would continue for as long as necessary during and after the program to build up the numbers in the CG. ⁴⁹

These measures necessarily resulted in having to abandon the concept of an experimental control group. Rather, the CG would be comprised of a range of youth, all "at risk" and otherwise eligible for the IT intervention, but not necessarily equivalent to those who eventually formed the PG.

---

Footnotes

1	Qualitative results have been extensively documented in a previous Interim Report. Major findings from the report have been brought forward into this report for the purposes of drawing final evaluation conclusions.
2	A description of the Lethbridge ITCY has been omitted as no outcome data for the program was collected owing to difficulties in obtaining the resources necessary to track clients. The program is described in the Process Evaluation report.
3	The instruments were designed to address issues in a Monitoring & Evaluation Framework prepared by project sponsors dealing with both process and outcome dimensions.
4	The consultants also designed a brief "Early Exit" survey to obtain information from clients who did not complete their training. This information relates more to process, and is therefore of lesser relevance to the outcome evaluation.
5	Youth filling out the Baseline Survey were all screened for risk factors that would have rendered them eligible for IT.
6	Data collection in Edmonton was taken over by a private consulting firm in April 1996.
7	The majority of follow-up employment measures were collected in a timely manner in Red Deer.
8	The .01 level was used because of the large number of baseline variables available for comparison.
9	See, for example: Heckman, J. J. "Sample Selection Bias As A Specification Error". Econometrica, 47 (1979): 163-174; and Bell, S. H., Orr, L.L, Blomquist, J. D. and Cain, G. G. Program Applicants as a Comparison Group in Evaluating Training Programs. Michigan: W. E. Upjohn, 1995.
10	This data is drawn from Early Exit surveys which were not completed with every client who left before completing their training. The data may not agree with agency-provided statistics.
11	Clients were asked, "While you were going through the program, did anyone serve as a role model for you, someone who took you under their wing or inspired you to try harder?" The role model could have been an agency staff member or someone at the employer's job site.
12	The Interim Evaluation found that the availability of incentives helped to attract youth to the project.
13	The agency had difficulty with its computer-assisted learning system which was ultimately phased out of use.
14	These include: social life with friends; family life; education they have received; jobs they have had; their life in general.
15	Clients were designated employed if they were working at the time of the interview or if they indicated having been employed during periods for which data was collected historically.
16	Unemployed is defined as not working and not in school or training.
17	Phase 1 of training included the in-house life skills component only. Phase 2 included Phase 1 training plus some training at NAIT. Phase 3 training included Phases 1 and 2 and job skill training at an employer's job site.
18	It was found that only 8 of the 68 CG members went on to take alternative training within 9 months after Baseline. Only 3 of these individuals were employed at 12 months.
19	See page 7 for discussion on controlling for bias between PG and CG.
20	A repeated measures design was used to compute a mean of means for data periods 9, 12 and 15.
21	Incremental gains were derived by subtracting the mean weekly income for CG members from the mean for PG members. The means were calculated including those individuals with 0 income (i.e., not working) such that the means represent the weekly incomes averaged over the entire cohort rather than only those who were working at that point in time.
22	Includes those collecting for 1 or more months during the period.
23	The monthly average is calculated for the total cohort, including those who did not collect at all or did not collect every month.
24	This can partially be explained by the fact that the post-Baseline period tracked was somewhat longer than the pre-Baseline period (i.e., 18 months vs. 12 months).
25	Integrated Training Centres for Youth The difference between the PG change (-$3.85) and the CG change ($6.00).
26	For PG and CG members who provided Social Insurance Numbers.
27	Data used in the table was drawn from an EI database compiled by HRDC using only 1 week of claim activity from each month's historical claim data. Consequently, the statistics for percentage collecting EI and the average collected per year would not reflect actual claim activity for either cohort and should be used with discretion in making comparisons between cohorts.
28	The average is calculated for the total cohort, including those who did not collect any EI during the 47 month period from August, 1993 to June, 1997. The number of months before and after Baseline varies for each client and has been factored into the calculation to ensure the average is properly normalized.
29	Self-reported by youth on Baseline Survey. It is difficult to reconcile the relatively large proportion of PG/CG members who reported having a criminal record at Baseline (completed from May, 1995 to October, 1996), with the relatively small proportion who appeared on the EPS database. Further study would be required to assess the accuracy of the statistics presented.
30	% of individuals either accused or suspected of committing a crime in the given year.
31	Adjusted to approximate 12 months.
32	This data is drawn from Early Exit surveys which were not completed with every client who left before completing their training. The data may not agree with agency-provided statistics.
33	Academic skills were stressed to a greater extent in the Career High program than at Destinations in Edmonton. However, Career High clients did not note the acquisition of academic skills more frequently than did clients in Edmonton.
34	Rather than focus on a small set of specific occupations to train for, Career High used a strategy involving numerous short-term work experience/training placements in a variety of industries. This would increase the probability that clients might not see a direct relationship between their current job and the placement(s) they had during training.
35	The low response rates for post-Baseline measures makes comparisons difficult.
36	There were income differences associated with other characteristics, however the number of clients with those characteristics were too small for the differences to be reliable (i.e. p-value approaching 0.10).
37	Results from the Lethbridge ITCY are not included in this or subsequent discussions.
38	The incentive was also instrumental in having clients return to Career High in the evenings to work towards their high school diploma even after they were employed.
39	Urban participants were less likely to indicate the incentive was adequate.
40	The incremental gains were found statistically reliable in Edmonton only.
41	A formal cost-benefit analysis was not performed.
42	Based on Provincial financial data. The average is for Destinations and Career High contracts only and includes both operating costs and incentive payments.
43	Incremental taxation is heavily discounted in cost-benefit scenarios because employment gains by one group partially displace/replace employment by other workers.
44	Occupational skill training significantly lengthens the intervention, increasing staff costs as well as incentive payments to participants.
45	In Edmonton, clients also received 2 weeks of training at NAIT which included certified and non-certified courses.
46	The Red Deer ITCY (Innisfail site) was co-located with the School Division’s maintenance shop. Clients could access training in the maintenance shop in the following occupational areas: carpentry, welding, plumbing, shipping/receiving, or general labour.
47	No criteria were proposed, although employment outcomes would be at the heart of a ranking system used to identify a bottom tier to be excluded from the sample.
48	A procedure was introduced whereby CG members were not told when they might get into IT, and were counselled to actively pursue other program options while they waited. CG members who enrolled in another program (of 3 weeks duration or more) were placed at the end of the waiting list.
49	Members of the CG recruited in this manner were ultimately excluded from the study in order to utilize an applicant-based design.

---





Last Modified: 1998-12-01	Important Notices