Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The purpose of a formative evaluation is to look at the early implementation stage of a program to examine whether the program is implemented according to its original plan and design, to look at uptake, delivery process and participants' satisfaction, and to provide early indications of program results.

This formative evaluation covers a selected group of HRDC youth programs under the Youth Employment Strategy (YES). At the request of Treasury Board, it also examines the coordination role of HRDC in the interdepartmental youth strategy, and some interdepartmental aspects of YES.

The formative evaluation will be followed by a summative evaluation to be completed in 2000. The summative evaluation will focus on the employment-related impacts of HRDC's youth internship/work experience programs.

Overview of YES

In February 1997, the Minister of Human Resources Development Canada announced the creation of a horizontal approach to dealing with youth issues, namely the Youth Employment Strategy. The Government of Canada announced this strategy to assist youth in gaining work experience, which would allow them to make the transition from school to work. It is with this in mind that the Government of Canada initially allocated $315 million over three years to YES. These funds were used by 12 federal departments and agencies, which have integrated their programs under seven youth initiatives within the strategy. On December 7th, 1998, YES was renewed and made permanent at $155 million per year.

The integration of youth programs within YES was accomplished by modifying existing programs and by creating new programs. For its part, HRDC integrated its existing youth programs: Youth Service Canada (YSC), Youth Internship Canada (YIC) and Summer Student Job Action (SSJA). As well, the following programs were created: Youth International (YI), Internships in Science and Technology and the YES Information component.

Objectives

The primary objective of the HRDC youth programs within YES is to help youth prepare to get a job; to get and keep a job; to make a successful transition from school to work; and, to undertake this within the conditions that exist at the local level.

Underlying these objectives are four sub-objectives:

a) To address the information needs of youth;
b) To enable youth to gain work experience and to reach their potential as productive members of society through the implementation of internships in various streams;
c) To support youth who are facing multiple barriers to employment through team work experience in community projects; and,
d) To help secondary and post-secondary students prepare for future entry into the labour market by facilitating access to summer employment.

The Evaluation Process

Scope of the Evaluation

By the time the YES strategy was launched in 1997, some of HRDC's youth programs were already in the process of being evaluated or had been recently evaluated. These were the YSC program, the Summer Career Placement (SCP - the largest component of the SSJA) and the Sectoral Youth Internships. Therefore, the focus of this formative evaluation is on the remaining youth interventions of HRDC.

More specifically, this formative evaluation addresses issues related to the rationale of HRDC's youth internship programs and examines design features of a selected group of youth initiatives. It also provides preliminary results on the success of Youth Internship Canada, Youth International, and Internships in Science and Technology in assisting young Canadians in making school-to-work transitions.

In addition, the formative evaluation included some profiling activities on the following components: Human Resource Centres of Canada for Students, Student Business Loan Program, and the Information Component.

Finally, the evaluation examines the coordination role of HRDC in the interdepartmental youth strategy as well as some interdepartmental aspects of YES.

Methodology

The findings derived from the formative evaluation of YES originated from multiple lines of evidence utilizing the following methodologies:

Literature Review. The literature review comprises an extension of earlier work. The literature review included the recent Lessons Learned Report on the Effectiveness of Employment-Related Programs for Youth, prepared in June 1997 by HRDC.

Profile of Youth. A profile was developed that provided a historical, current and projected perspective of the youth cohort in Canada.

Survey of Employers/Sponsors. The Survey of Employers/Sponsors resulted in 1003 completed interviews and provides information on employers/sponsors participating in the Youth Internship Canada (978 employers/sponsors) and Youth International (25 sponsors) programs. The information provided a profile of employers/sponsors; identified their roles; addressed issues associated with program rationale; program design and delivery; satisfaction with the programs; and, program outcomes.

Survey of Youth Internship Canada Participants. The Survey of Participants in Youth Internship Canada resulted in 400 completed interviews and provided information on the characteristics of interns for the fiscal year 1997-98, their experiences in the program, their views on program design and delivery, and immediate program outcomes.

Case Studies and Site Visits. Case studies address several components: specifically, Youth Internship Canada (six site visits), Youth International (four case studies) Human Resource Centres of Canada for Students (three site visits), and the Job Fairs (two case studies) sub-component of the YES Information component.

Document and Administrative Data Review. The review focused on the extent to which appropriate data is being collected to support program monitoring and the evaluation of program impacts of youth internship programs. The systems reviewed included National Employment Service System (NESS), Contact IV, and the Canadian Job Strategy-1 (CJS-1) file.

Key Informant Interviews. Interviews were conducted with individuals representing HRDC, other departments participating in YES, interdepartmental committees, and sector councils. Additionally, interviews were conducted with 43 participants in the Internships in Science and Technology.

Key Findings

Rationale for HRDC Youth Internship Programs under YES

Based on a review of the literature, the discussion of the rationale (or reasons) for youth internship programs under YES was divided into two parts: the rationale for implementing this type of programming; and, the types of young people who were likely to participate in the youth internship programs. The formative evaluation found that there is support in the literature for interventions that carefully choose groups that should participate in active labour-market programs (ALMP) such as Youth Internship Canada. However, the HRDC program documents specified only very general guidelines for the structure and content of internships, as well as broad participation criteria. The broad range of program participants raises the risk that resources may have been spent on youth who may have had a successful school-to-work transition without support from the program. As the literature documents though, careful targeting of ALMP is no simple matter in determining specific targets.

For its youth internship programs under YES, HRDC contributes funds, to private and public-sector associations and non-governmental organizations to develop projects that offer unemployed and underemployed youth work-experiences in their local labour market and in growth sectors of the economy. The majority of YIC employers and sponsors interviewed during this evaluation felt that YIC assisted interns in making the transition to work. Both YIC employers and sponsors agreed that the internships allowed interns to gain new skills, to get a better understanding of what is expected in a work situation, and to offer interns work experience related to their career aspirations. Worthwhile to notice, on all questions related to this subject, sponsors consistently rated the internships higher than did employers. This may be due to sponsored projects tending to be more comprehensive, with various modules and often a training component, whereas employers in many cases provided youth-only work experience.

Youth Internship Programs Design

The formative evaluation addressed issues associated with the adequacy of the client tracking and monitoring system used to collect program-related information on participants and interventions for each of HRDC's youth internship programs. The key finding, relative to program design, pertains to the tracking and monitoring systems (Contact IV and NESS). Specifically, the evaluation found that participant information for many youth internship participants was not available at the time of the evaluation. Information pertaining to their participation could only be obtained by contacting employers and sponsors who then provided lists of participants. While this presents problems for the subsequent evaluation of HRDC's youth internship programs, the more serious problems relate to the lack of capacity to properly monitor the youth internship programs, to ensure that applicant criteria are being respected, and to the lack of information linked to client results to guide policy development and management planning with respect to targeting of clients.

Youth Internship Programs Outcomes and Incrementality

The time lapsed (about 6 months) between the completion of the youth internship projects by participants and the data gathering for the evaluation was too short to fully assess program impacts and effects. As a consequence, the formative evaluation focused on issues associated with satisfaction, current employment status and future prospects.

Youth Internship Canada Program

According to the survey of YIC participants, the program helped participants gain experience and, at the time of the evaluation, 50% of those who had completed their projects were still working for the same employer. An additional 17% were working full-time for another employer and 19% were working part-time also for another employer.

Overall, participants, sponsors and employers were very satisfied with the YIC. The majority of interns interviewed felt the program had allowed them to gain experience that will help them to obtain the kind of job they want (89%). Another large fraction of respondents reported that their participation in the program has helped them in making choices about education (67%) and careers (75%). However, only 43% of participants were satisfied with the salary they received.

On the YIC employers' and sponsors' side, the large majority felt the work done by the intern was useful (70% for employers, 90% for sponsors). Almost all respondents felt the experience would improve the intern's chance of finding a full time job, the most frequent reasons given being the on-the-job experience, formal training and career-related experience. However, 39% of employers reported that they would have hired someone for the intern's position without YIC support. Furthermore, a quarter of all employers indicated that they would have hired the same individual as they did under the program without the financial assistance from YIC.

Science and Technology Internship Program

The 43 participants interviewed from the Science and Technology Internships program reported a high level of satisfaction with the program, stating that the internship provided them with experience and an opportunity to learn. A little more than one half of interns interviewed predicted that they would continue to work for the same employer after the project was completed.

Although many sector councils reported that positions are being developed specifically for youth interns, two out of nine sector councils reported that many of their interns would have been hired without the program. On the interns' side, a slight majority of participants interviewed (24 of 43 interns) indicated that if they were not currently in the internship program, they would most likely be employed (not necessarily with the same employer).

Youth International Program

Interviews with interns in the four case studies of Youth International projects indicated that, overall, the youth participants were satisfied with their internships. Most interns interviewed reported that they were gaining work experience, and would recommend the program to other youth. The employers reported that in many instances, if the interns continued to perform in the same satisfactory manner throughout the remainder of their internships, they would most likely offer the interns positions with their businesses upon completion of their internships.

Interdepartmental Issues

In addition to being responsible for its youth initiatives under the YES, HRDC is also the interdepartmental coordinator for the strategy. Key findings are presented along the following broad themes: implementation of the YES; roles and responsibilities; interdepartmental coordination; value-added of the strategy; and, sustainability of the YES.

The delays in launching YES and the subsequent program announcement, coupled with the short time frame in which to access YES funds, meant that in many cases departments other than HRDC had little time to implement their YES programs. However, despite the problems with implementation, the departments were able to proceed with the strategy.

In order to facilitate the strategy's implementation, HRDC received the communication and coordination mandate for YES, essentially continuing its short-term mandate to chair the initial interdepartmental meeting. Largely due to this HRDC initiative and the cooperation of all departments involved, a common focus on youth has prevailed and provided a sense of direction for YES programming in other departments. As such, it appears that HRDC has functioned as effectively as it possibly could under these circumstances.

From the perspective of those interviewed from HRDC and the other departments, there is value-added in having a national interdepartmental strategy for youth employment. This is due in part to the fact that, as youth employment is a horizontal issue, YES has provided departments with an increased focus on youth. As such, departmental objectives have been tailored to encompass youth issues. The impact of this focus on youth is illustrated by the fact that departments were able to implement the strategy despite numerous initial problematic issues.

Although improvements have been made since the initial implementation of YES, questions remain regarding departmental accountability. For instance, one issue raised related to the international component of YES. Several departments deliver Youth International (YI) and it appears that multiple departments are approaching the same employers and sponsors, and in some instances the same participants. As well, the multiplicity of international programs may be creating the perception that duplication exists within YES in terms of YI programs.

Finally, at the time of the formative evaluation, most departments reported that without YES funding, they did not have monies available to continue the programs. In many cases, departments also indicated that YES funds did not adequately cover the cost of administering the programs within their departments. Several departmental representatives expressed the concern that should a gap in funding occur, the delivery of YES programs would be very difficult to restart. Since the completion of this formative evaluation, the funding for the YES set to end on March 31, 1999 has been made permanent.