|
|
     |
||
 |
     |
 |
|
Employment-Related Programs for Youth:
Lessons Learned from Past Experience
Evaluation and Data Development
June 1997
SP-AH027E-06-97
This study summarises lessons learned from the Department of Human Resources Development Canada’s (HRDC) youth programs, as well as from youth programs in other countries. John Greenwood of the Social Research and Demonstration Corporation, in cooperation with Saul Schwartz of the School of Public Administration at Carleton University, was largely responsible for the development of the methodological approach and the analysis of the existing information on youth programs. The project team conducted the study for the Evaluation and Data Development Branch of HRDC, in support of the Ministerial Task Force on Youth. The results of the study served as an input to the development of the New Youth Employment Strategy launched by HRDC in February 1997. A number of papers were prepared for HRDC as part of this study by the following authors: Gordon Betcherman and Norm Leckie, Human Resource Group, Ekos Research Associates, for the paper Profile of the Youth Labour Market W.Craig Riddell, Department of Economics, University of British Columbia, for the paper Evidence on the Effectiveness of Youth Labour Market Programs in Canada David A.Long, Manpower Demonstration Research Corporation, for What Works? Evidence from Evaluation Research on Programs for Disadvantaged Youth in the United States John A. Burns and Vanessa Thomas, Centre for Public Management, Evidence on the Effectiveness of Youth Labour Market Programs in Australia and the United Kingdom The original papers also benefited from comments provided by Audra Bowlus, Department of Economics, University of Western Ontario; Harvey Krahn, Department of Sociology, University of Alberta; Lars Osberg, Department of Economics, Dalhousie University; and Marc Van Audenrode, Département d’Économique, Université Laval. We are grateful to the many people who assisted the evaluation team.
Human Resources Development Canada (HRDC) and the organisations that came together to form it have long and proud traditions of continuous learning and improvement. Over the years, the Evaluation Services Directorate (ESD) of the Evaluation and Data Development Branch at HRDC has produced numerous evaluations on a wide range of departmental programs. In 1996-97, ESD initiated a new series of evaluation studies with the object of taking the findings of earlier reports and updating them with new literature reviews and expert opinion. The primary goal of the new exercise is to identify the lessons that can be learned from past experience — to focus on what has worked, what has not, and for which client groups. Another important goal is to develop evaluation measurement tools in areas where such tools are non-existent. Not surprisingly, this new initiative has come to be known as the “lessons learned” series. The intended audience includes senior managers, program managers and policy analysts both within and outside the federal government, members of the academic research community and all stake-holders who will benefit from having a clearer idea of the lessons learned based on the evaluation studies of past and present programs. HRDC is pleased to present the first study in the new series. It focuses on the lessons learned from employment-related programs for youth over the period starting in and ending in 1977 to 1995. The information produced for this report was used in the design of the new Youth Employment Strategy launched by the federal government in February 1997. In time, the results of the Youth Employment Strategy will add to the pool of knowledge on youth programming. As a learning organisation, HRDC will continue to experiment with new approaches and evaluate their effectiveness. HRDC recognises the vital importance of the evaluation process and is committed to continuing its work in this area.
For those who watch from afar – policy-makers, politicians, parents – the calculus of risk and reward attending the efforts of young workers has shifted significantly in the past twenty years. The rewards for success – stable, highly-paid and intellectually-fulfilling work – seem greater than before but far less certain. The alternatives to such success for the unlucky, and for those unable or unwilling to compete for positions in the labour market elite, have become less certain. No longer does a strong back and a will to work guarantee steady employment. "Uncertainty" is now a constant.
What must a young person do to succeed? To be sure, education is necessary for labour market success but it clearly is not sufficient. Where once a university degree virtually guaranteed a steady, middle-class income, we can now point to a large number of unemployed or underemployed graduates. Where once high school drop-outs might have found steady unskilled work in factories, in mines or on the ocean, they now must work – if they can find work at all – in low-paid, high-turnover service sector jobs. Young people have always faced the formidable challenge of discovering their capabilities and skills. Now they must also guess at which skills might remain in demand by future employers.
Symptomatic of the increased uncertainty facing young workers is the misnamed "school-to-work transition." In some imaginary or long-since vanished labour market, Canadians went to school until they stopped and, upon stopping, took up full-time work. In today's labour market, many students work and many workers study, so that the line between school and work has become quite blurred.
Is there cause for serious concern about changes in the operation of the youth labour market? To be sure, the job market for young people has long been "worse" than it is for adults: labour force participation rates are lower and unemployment rates are higher. At any point in time, a smaller proportion of young people are working or looking for work because some are in school or raising young children. And those who are in the labour market are more likely to be unemployed, since some will be moving from job to job, hoping to find one that is both steady and satisfying.
Using data from Statistics Canada, Betcherman and Leckie1 have documented that the youth labour force participation rate, which fell dramatically during the last recession, has been continuing to decline during the current recovery. And, while the gap between the unemployment rates for youth and adults widened during the recession (as would be expected), that gap has failed to close in the post-recession period. While their conclusion that the economic recovery of the 1990s seems to have bypassed youth may, perhaps, be too strong,2 there is reason to be concerned by these trends. In particular, the industrial pattern of the jobs held by young people has also changed, so that an increased concentration of youth is evident in industries such as retail trade and personal services that are often characterised by relatively low wages, poor benefit coverage, a high incidence of contingent work, and many part-time jobs.
If we look beyond the more standard labour market indicators, we get some glimpses of two relatively new features of youth labour markets – the increased volatility of employment and increased underemployment.
Part of the uncertainty of employment might be captured by looking at the volatility of earnings for young people. If uncertainty is increasing, we should see young people moving from job to job and from higher-paying to lower-paying jobs, and back. We might also see individuals moving into and out of the labour force, or initially working part time and then full time. These so-called "labour force dynamics" are difficult to analyse in Canada because of the lack of time series labour force data. It is difficult to know if young people now hold more jobs of shorter duration, or if workers are cycling between "good jobs" and "bad jobs", or if periods of unemployment are more or less common unless we are able to systematically "watch" the careers of a group of Canadians unfold. Nonetheless, both longitudinal tax data and the National Graduate Survey can be used to shed some light on the extent to which the working lives of young people have become more uncertain.3
Anecdotes about underemployment are quite common. While we have all heard stories of Ph.D.'s driving taxis and English majors tending bar, actually measuring the extent of underemployment, and judging whether it has increased or decreased in recent years, is quite difficult. For every welder driving a truck, there is an accountant working as a middle-level manager; for every Arts major waiting tables, there is a Social Science major turned construction contractor. "Underemployment" is clearly quite subjective. Much more work remains to be done in this area.
An examination of the youth labour market helps us to understand the intellectual motivation for the youth initiatives mounted by governments. The "school-to-work transition" has been the focus of a number of HRDC programs over the past two decades. However, the increase in involuntary part-time work among youth and the fall in their labour force participation suggests that the problem remains. Similarly, the greater importance of high school graduation might justify efforts such as the Stay-in-School Initiative.
In view of the increasing policy interest in how to design more effective ways of assisting young people prepare for, find and retain employment, this paper tries to summarise "lessons learned" based on the evaluation studies of former and current programs. Knowing what has been tried and what has worked before, and for whom, may help in designing future interventions. In examining program effectiveness, our principal interest is employment-related impacts. In some cases, programs have been successful in producing intermediate outcomes, such as higher educational attainment or a reduction in anti-social behaviour (e.g., involvement in criminal activity). However, the main focus here is on whether programs produce positive results in terms of labour market outcomes – mainly increases in employment and earnings.
The education and training programs discussed here are aimed at young workers. But the employment success of such workers is also affected by other important factors. First, these programs operate on the "supply side" of the market, while the employment of young people is also clearly affected by what is happening on the demand side and by other government policies with other objectives (including deficit reduction and price stability) that may dampen the overall level of economic activity. Perhaps by the middle of the next decade, as today's young people settle into jobs once occupied by baby-boomers, they will enjoy the fruits of such policies. But, like so much of the lives of young people today, these longer-run benefits look uncertain.
Second, an increasing body of evidence suggests that early intervention – as early as the pre-school period – can prevent the occurrence of later problems. In Canada, that evidence is most closely associated with the work of the McMaster University psychiatrist Dan Offord. The problems that we try to solve in the "repair shop" of education and training programs may actually originate at very early ages and may be more fruitfully addressed at that time.
As a final caveat, the original papers prepared for HRDC included information extracted from evaluation literature from other countries. It needs to be kept in mind that, although international experience may provide some useful pointers for Canadian programs, economic, social, cultural and other differences between countries may mean that the lessons are not readily transferable.
What lessons can be extracted from the review of material on Canadian and international experience with youth programs? As a starting point, a number of key messages deserve to be highlighted.
In addition to these general comments a number of important lessons can be drawn concerning specific types of interventions.
School-based programs are an attempt to focus on prevention rather than remediation. The principal source of labour market preparation for young people remains the school system. Young people with more education do better than those with less. Those who drop out without a high school diploma have seen their relative position in the labour market worsen considerably over the past couple of decades. Therefore, the most effective of the strategies reviewed here are built around keeping young people in school. In addition, using the mainstream education system as much as possible is more efficient than building alternative program delivery infrastructures to deal with youth after they drop out.
School-based programs aim at keeping students in school, and they try to build bridges to the work world while young people are still in school, particularly for those who are not destined for post-secondary education.
In general, two broad approaches have been used:
1. Help with Academic Performance
Poor academic performance has frequently been found to be a predictor of those who are at risk of dropping-out (although it is usually symptomatic of other underlying problems). Programs that help poor-performing at-risk students have been shown to be effective in raising the graduation rate for this group.
Several approaches, with a variety of features, have been used to help students raise their level of academic performance, and many of them have had positive effects on high school graduation rates. However, results have varied considerably among sites. In general, programs that are designed to provide on-going help during the school year – including adult and peer mentoring, help with homework, and offering small stipends for successful participation in school-based programs, sometimes provided in the form of credits that can be used later for post-secondary tuition – have been particularly successful in increasing the graduation rate among disadvantaged youth.
On the other hand, programs that provide remedial academic assistance during the summer have produced short-term gains in terms of helping disadvantaged students keep up and avoid summer learning loss, but they have not had much impact on their graduation rates and on their subsequent employment.
2. Alternative Work-Oriented Curricula
Implementing alternative, labour-market-focused curricula has produced mixed results. These programs have tended to be targeted on disadvantaged and other at-risk populations. They aim to increase student retention by demonstrating the relevance of high school education, particularly to those who may not be destined for post-secondary education or who may wish to continue with a trade or vocational program after high school.
There are no rigorous Canadian studies of the impacts of this approach. Much of the discussion is simply a debate between the proponents of "contextualised" learning and those who ascribe little value to "alternative" curriculum elements based on work experience. In the US, there is some evidence that alternative curricula can lead to improved attendance and better grades, but success in raising high school graduation rates has varied considerably from program to program and, within a program, by geographic location. The inclusion of work elements can either reinforce the value of schooling or it can reinforce the notion of work as a substitute for school. The American evidence also suggests that these approaches are more successful with young women; while among young men, particularly blacks, the involvement in work-related activities may actually increase their likelihood of dropping-out of high school to work full-time if a job becomes available.
3. School-to-work Transition Programs
In Canada, experience is based on specific transition programs, usually small in scale compared to the mainstream school programs alongside of which they operate, and usually targeted on a specific student population (typically at-risk or under-achieving students).
In Europe, there are many more examples of how basic education and vocational preparation can be combined in the education system. Two broad approaches are exemplified by the dual system in Germany, in which young people are streamed at an early age into enterprise-based vocational training operated in conjunction with and integral to their basic schooling, and by the National Vocational Qualifications approach in the UK, whereby the private sector establishes, within a framework set by government, a series of competency standards for each occupation, but where it is left up to the individual to decide how they want to go about acquiring the skills necessary for occupational entry (e.g., in school, through on-the-job training, from private training providers).
It is important to recognise that national education and training systems reflect the broader institutional environment within which they operate. Consequently, approaches used in one country may not be readily transferable to another. In addition, elements of alternative systems, such as the development and maintenance of a comprehensive set of national competency standards, the accreditation of training institutions, and the testing of competencies, can be quite costly.
Canadian transition programs aim at developing stronger connections between schools and the workplace. The most common examples are cooperative education programs and in-school apprenticeship programs. The evidence suggests that the effectiveness of such programs, at both the secondary and post-secondary levels, in significantly affecting the employment and earnings of graduates depends on the quality of the job experiences that are provided. The key is to provide young people with experiences which will be valued in the full-time labour market. In particular, there is no substitute for actual paid work experience in the private sector.
Programs that link school and work experience face significant barriers to expansion. Many educators in North America downplay the role of the education system in providing vocational preparation and, consequently, they are reluctant to develop opportunities for and to give academic credit for work experience. Also, the number of work experience places offered by employers is quite limited.
In addition, the results of these programs have been mixed. There is some evidence that cooperative education leads to improved employment outcomes for those in post-secondary education. But the evaluation results on the (albeit more limited) experience with cooperative education in high schools suggests that co-op programs may have impacts on employment. Longer term impacts are presently unknown for high school students. Additional research needs to be undertaken to explore long term impacts, particularly for high school students who pursue vocational or apprenticeship training.
An alternative approach to providing students with work experience is to improve their access to summer jobs. Evaluation results suggest that summer jobs can increase the number of disadvantaged young people who obtain employment. Summer employment, however, also often attract youth who would have found jobs anyway. However, while there is general evidence that work experience is an important determinant of later employment success, it is unknown at present whether summer jobs have any impact on the academic achievement or subsequent employment experiences of participants.
Finally, Evaluation shows that placement services that are set up to cater specifically to the needs of students (e.g., Human Resources Centres (HRCC) for Students or HRCCs on Campus) are not any more effective (nor any less effective) than regular HRCC services in helping students find jobs, or in reducing the time they take to find jobs, or improving the kinds of jobs and earnings they obtain.
Helping disadvantaged young people is not easy, and it is even more difficult once they have dropped out of school. In the US, in particular, youth programs have been predominately focused on young people with serious social problems – young people involved with the criminal justice system, youths drawn from racial minority groups in inner-city areas, unmarried teen mothers. The success rate of these programs has not been high, and in cases where programs have been offered to both in-school and out-of-school youth (such as the LEAP and TPD programs discussed by Long), the results have been much poorer with those who had already left school.
Some interventions have had modest success – but they have often been intensive and sometimes expensive. The best-known American example is Job Corps which provides a relatively lengthy period of training, combined with opportunities to acquire on-the-job work experience, and the provision of job search help. Other cheaper and less intensive programs have had little or no impact in the US. It is also clear that the earlier an intervention occurs, the better. The dynamics of low education and skills, chronic unemployment, poverty, welfare dependency, and declining self-esteem are self-reinforcing. After a while, the problems become almost insurmountable.
Canadian programs for out-of-school youth generally have not focused on a severely-disadvantaged clientele. One Canadian example of a program that was targeted on a more disadvantaged group of clients was the Severely Employment Disadvantaged Option of the Job Development Program (although SED was not specifically a youth program). SED was found to have a small, but positive, short-run impact on participants' employability, but the impact dissipated over the longer run; and SED produced no gains in participants' earnings.
The Entry Option was the component of the HRDC's former Job Entry program that was predominately focused on youth and it did produce significant short-run gains in employment and earnings, although these gains were found to erode over time. Overall, however, Entry participants could best be described as mildly disadvantaged. Only a small proportion of participants came from minority groups, relatively few had an attachment to the welfare system, and half of all participants studied were in Ontario and British Columbia – the most buoyant and diversified of provincial labour markets.
It is in large part because of these disappointing results that there has been a renewed interest in prevention, using school-based programs, rather than relying on remedial approaches to helping disadvantaged youth.
Once young people have left school, there are six general approaches (as opposed to specific programs) that have been used to help them overcome their employment-related problems:
1. help in finding a job 2. projects to provide periods of temporary work experience 3. wage subsidies to encourage hiring by employers 4. supports to self-employment 5. vocational training 6. encouragement to return to school.
1. Job Search Assistance
Job finding help is offered in a variety of forms – vocational counselling, training in job search skills, resumé writing, and job finding clubs. These programs do seem to be able to accelerate the process of finding a job. However, they do not seem to be able to produce lasting gains in terms of employment and earnings. Most commonly, increases in earnings erode quite quickly; after one to two years, participants' earnings are no higher than those for non-participants. This is not surprising; job finding help alone does not do anything to better equip people to participate and advance in the labour market.
However, these programs are relatively inexpensive to operate, and they are usually found to be cost-effective since even modest benefits offset their costs.
2. Work Experience (Job Creation) Projects
These types of programs typically provide temporary periods of work experience, usually in the public or non-profit sectors, and usually providing a top-up payment to a participant's benefit entitlement (welfare or UI) in lieu of a wage.
These programs have produced generally disappointing long term results. For example, the General Projects option of HRDC's former Job Development Program had no impact on participants' earnings and a slight negative effect on their employability. Evaluations of American programs have observed short-term effects but without any longer-term improvement in employment and earnings. Participants have higher earnings while taking part in the program; however, they seem to have no greater success than non-participants in finding and keeping work later on. An Australian program was found to have a slight positive effect on participants' employment, mainly among the long-term unemployed.
Previously, most European countries provided some sort of direct job creation program, usually relying on jobs in the public sector. However, most of these programs have now been abandoned as they are viewed to be costly and ineffective.
What has become clear over time is that just any job does not work. Meaningful employment in growth sectors is needed.
3. Wage Subsidy Programs
Targeted wage subsidy programs pay employers to hire members of a particular target group. The characteristics of programs can vary widely (lump-sum payments, flat-rate and graduated rates of wage reimbursement, front-end and back-end loaded subsidies, bonuses linked to the provision of on-the-job training).
Australia and several European countries provide temporary wage subsidies to employers. These are not strictly youth programs; for the most part, they are targeted on the long-term unemployed. Evaluations of these programs suggest that wage subsidies can increase the probability that disadvantaged people will be hired to fill available jobs. And, in Canada, the recent evaluation of the Job Opportunities option of the Employability Improvement Program estimated relatively large positive impacts on participants' employment and earnings. Similarly, evaluations of former training programs, such as the Critical Trade Skills Training Program and the Workplace-Based Training option of the Skill Shortages Program, found that wage subsidies offered as an incentive to provide training can lead to gains in the productivity and earnings of participants.
On the other hand, there is little evidence that wage subsidy programs increase the number of jobs available. The OECD has estimated that only about one job in five is created as a result of the subsidy. In the other cases, employers would have hired someone anyway. Therefore, those who are hired will, for the most part, simply displace others who would have been hired without the program, and the positive effects of the program will be offset by the losses experienced by those who are displaced by program participants. However, these programs do give an advantage to those who are eligible to have their wages subsidised compared to other job seekers, and such a redistribution of job opportunities may be justified on equity grounds. Disadvantaged people can be provided a chance to gain work experience, keep a connection to the labour market and share in the benefits associated with paid employment. Of course, this is only true to the extent that the people who are displaced are not themselves members of disadvantaged groups.
4. Self-Employment Assistance
The Self-Employment Assistance (SEA) program in Canada, Australia's New Enterprise Incentive Scheme, as well as programs in several European countries, are designed to help unemployed people start their own business. These programs are typically run as extensions of the Unemployment Insurance systems.
Qualitative studies have pointed out the importance of combining financial assistance with other types of support, such as management training and business planning help. There has not been much rigorous quantitative analysis of impacts. However, businesses started through these programs have generally had survival rates that are in line with other new business start-ups, and a UK study estimated that about one in four successful starts under the Enterprise Allowance Scheme would not have started without support from their program.
A recently-completed evaluation of SEA in Canada indicates that, at least in the short term (program participants were followed up, on average, eight months after completing SEA), participants were more likely to be self-employed and SEA produced positive impacts in terms of higher incomes and reduced receipt of UI and SA benefits. Although a portion of the SEA budget is set aside to assist young people, separate program impacts on youth participants are not available. The limited Canadian experience with Student Business Loans suggests that a selectively applied program, operated in cooperation with the private sector, can stimulate young people's interest in entrepreneurial activities, and can encourage self-employment on the part of students (particularly high school students) as an alternative to more traditional summer or part-time jobs.
5. Training Programs
The evidence suggests that well-designed training programs can make a difference. However, the increases in the employment and earnings of trainees will likely be modest, particularly for young men and particularly for the disadvantaged. Also, training is much more effective when it is combined with other services, such as job search help and work experience opportunities.
It should be noted that the paucity of evidence for substantial long-term effects from most training programs is partly a reflection of the relatively short time periods over which the programs have been evaluated. The benefits from training take time to appear and may accrue slowly. In particular, unless there are large earnings differences between those who take training and those who do not, it may take a considerable period of time for earnings gains to offset the opportunity costs associated with staying out of the labour market to complete a training program. Without a sufficiently long follow-up period, such programs are unlikely to be found cost-effective.
Training programs that focus on a disadvantaged clientele, particularly on young men who leave high school prior to graduation, have also been characterised by high drop-out rates. Therefore, the rather disappointing results from these programs may, to some extent, be a reflection of incomplete exposure to the program, rather than the ineffectiveness of the training itself.
The most successful training programs for disadvantaged young people have tended to be those that are relatively intensive and provide support services in conjunction with the training. Many programs complement training with counselling, job search assistance and similar services. In such cases, it is often difficult to isolate the impact of the training itself.
An example of one of the most successful intensive mixed-services programs for disadvantaged young people is the previously-mentioned Job Corps in the US – a relatively lengthy residential program providing basic education, vocational skills and a wide range of support services. Job Corps has been found to lead not only to an increased rate of high school graduation and a reduction in involvement in crime, but has also produced modest increases in employment and earnings and decreases in transfer payments (UI and welfare) to participants. Despite its high cost per participants, Job Corps was also found to be cost-effective in terms of producing benefits to society in excess of costs, particularly when the savings associated with the reduction in serious crimes is taken into account.
However, because these successful models are expensive and complicated to operate, they will always be limited to helping relatively small numbers of people.
In the few cases where shorter-term, less-intensive programs have had some success, they have usually been based on a "work first" approach. The "work first" approach can be characterised as one which maintains a strong focus on jobs. Such programs, for example, emphasise on-the-job training, job search skills, and strong links to local employers. One of the most successful, and best-known, examples is the Center for Employment and Training (CET) in San Jose, California, which operated under the Jobstart program and provided one of the shortest and cheapest training interventions of any site within Jobstart. However, the key factor seems to be a knowledge of, and a connection to, the local labour market. CET has a very strong – almost single-minded – focus on getting participants into jobs. The program maintained close connections to the labour market, including the use of industry advisory boards and recruiting program staff with good labour market knowledge and employer connections. Academic upgrading was combined with vocational skills training and both were provided using contextualised approaches to learning. Curricula were individually tailored, and the training programs were designed to permit open entry and exit to get participants into a job at the first opportunity. Alberta recently began testing the CET model at four pilot Integrated Training Centres.
Not surprisingly, training programs are most likely to be successful when they are focused on skills in demand. The positive evaluation results from the Skill Shortages Program demonstrate that a tight focus on training in occupations in demand can substantially raise the impact of a training program.
Admittedly, this is not easy to do. Skill needs can change rapidly over time, so it may be difficult to anticipate requirements with any degree of precision where lengthy training periods are involved. Occupations in demand also vary from place to place. This may be particularly important in the initial job experiences of young people, who may not be very mobile due to family ties or financial constraints. It may be that the key strength of the CET model is the strong link to the local labour market that allows program administrators to identify and train for skills in demand locally. This would be especially important if local employers use these programs as a method to screen for new workers. Finally, the nature of skills in demand may be changing. Employers are describing their needs less in terms of specific occupational skills and more in terms of generic skills – communications, teamwork, problem-solving – that help ensure a flexible and adaptable workforce.
6. Return to School
Given the importance of education to employability and lifetime earnings, an obvious strategy for helping high school drop-outs is to help them go back to school. Employers are describing their needs less in terms of specific occupational skills and more in terms of generic skills – communications, teamwork, problem-solving. Most provincial education departments and school boards offer some form of adult high school or alternative school for those who left high school without graduating. However, evaluation studies of these approaches are not available. Evidence from the U.S. indicates that academic upgrading on its own produces little in the way of employment and earnings gains. The key here, however, is that a high school diploma can provide additional options. For example, if academic upgrading is a prerequisite for, and is offered in conjunction with, skill training then positive results can result. And, of course, it permits entry to post-secondary education which does produce significant impacts on subsequent employment and earnings.
A more recent approach is the use of vouchers to provide access to educational opportunities, and, in particular, to encourage young people to pursue post-secondary studies. For example, a recently-completed evaluation of a project in Newfoundland (the Student Work and Service Program) suggests that tuition vouchers that were given in return for participation in work experience projects had a significant effect on increasing the number of social assistance recipients (SARs) who decided to go on to post-secondary education after taking part in the program. On the other hand, vouchers had little impact on the decisions of non-SAR participants (the vast majority who pursued post-secondary studies would have done so without the voucher). This would seem to argue for careful targeting of such programs.
A similar program of somewhat longer standing is the Youth Credits program in the United Kingdom. This program is currently moving to full implementation following a pilot phase. Unfortunately, no quantitative impact evaluation has been conducted. Case study evidence suggests that vouchers may have provided greater access to further education to some disadvantaged youth. However, there is no evidence that it has had any effect on the training choices that young people make or on the types of training offered by training providers.
The last section summarised "lessons learned" from empirical evaluation studies. The confidence with which we assert that this or that program "works" ought to be related, however, to our confidence in the methodology that underlies the empirical evaluation results. This section discusses some of the methodological challenges that face researchers in this area.
The evidence from evaluation studies suggests that putting effective programs in place is not easy. The variation in impacts across similar programs that have been implemented in different places and at different times indicates that programs are not easy to set up, operate and maintain over time. A good program design is not, on its own, any guarantee of success. It is important that sufficient time and resources be allowed to plan program implementation, to develop operational procedures, to train staff, and to work out initial start-up problems prior to full implementation. Well-trained and motivated staff, effective program management, and early and continuous monitoring of program implementation and operation are all crucial to program success.
The variability of results may also reflect differences in economic environments.4 A program may be a success in one labour market context but a failure in another. Therefore, an evaluation of a program, such as a training program in specialised skills whose impact in terms of participants' post-program earnings and employment depends on whether vacancies exist for those particular skills, may generate a fundamentally different answer if it is conducted during a period of high unemployment, rather than in a tight labour market. Both the benefits and costs of a program intervention can vary with labour market conditions. Participants' opportunity costs, the extent to which participants drop out of a program, the wages and hours of work obtained after participation, and displacement effects may vary systematically with the tightness of the labour market, and all these factors will affect the social benefit-cost analysis of a program.
Conducting research to determine what works is also not an easy matter. Well-designed program evaluations require the collection of a great deal of information over a considerable period of time. Constraints in terms of time, money and data availability can result in program impact evaluations that are weaker than one would like to see. Ideally, we would want to pay particular attention to those evaluation studies that have used rigorous methodologies and, in particular, those which have estimated program impacts, rather than those which simply provide formative assessments or descriptive analyses. Common sense lies at the heart of all evaluation research. Before a program can be evaluated, the desired outcomes and the time period over which the occurrence of those outcomes will be measured must be specified. Then, we must "compare comparables," which leads to the use of comparison groups and a variety of complicated methods for adjusting for selection bias.
In light of the evolution of evaluation techniques over the past couple of decades, it is perhaps not too surprising that the program evaluation studies reviewed for this report employ a wide range of methodologies. However, the variation in the reliability of the findings from such diverse studies increases the difficulty of generalising from the reported results. Some studies do not even attempt to generate comparison groups and estimate impacts. Other studies provide impact estimates that focus on the short-run effects at the expense of the long. Why should this be so?
First, when policy-makers feel certain that a program will accomplish its objective, a comparison group may be considered unnecessary. For example, summer jobs provided for young people in rural Newfoundland are almost certainly "incremental." All observers seem to agree that, absent the government-funded jobs, there would be no other jobs available.
Second, even if a goal of a program is the long-run improvement of employability and earnings, the study of such impacts may be difficult to organise. A large number of employment programs have come and gone over the past fifteen years. It may be difficult to generate the resources and enthusiasm for studying the post-program success of trainees, ten years after their training, in an environment where both the program and its proponents may have long-since vanished.
We need to read evaluation reports not only to discover the "bottom-line" evidence on program impact but also to ascertain their credibility. Furthermore, we need more long-term studies (studies, for example, of the type illustrated by HRDC's Longitudinal Study of Training). In this regard, there is a vast array of outcome information residing in the longitudinal data that could be exploited in conjunction with past program records.
Much of the American literature emphasises the use of experimental evaluations. There is little doubt that, in the US, experimental valuation is now viewed as "best practice." The decision by the US federal government to mount an enormous and expensive experimental evaluation of its primary training program – the Job Training and Partnership Act – is a symptom of that methodological ascendancy.
An important, although often neglected, advantage of experimental evaluations is their simplicity. Even though they can often be lengthy and expensive, the basic procedures are simple and standardised and one can reasonably expect that different teams of evaluators will use essentially the same techniques. By contrast, in considering non-experimental evaluations, we must constantly worry about the size and composition of the comparison groups (or even their existence) and about the methods (if any) used to account for selection bias.
However, there remains substantial intellectual support for non-experimental methods, largely because experiments are atheoretic. They tell us if the program participants "did better" than non-participants but, by themselves, they cannot tell us why. If, as is often the case, the "program" is a range of services rather than a single intervention, no one experiment can say which of the various services was most important in producing the positive impact. For that reason, we believe that a non-experimental evaluation of any single program may well be equal to, or better than, an experimental evaluation. But for this to be true, the non-experimental evaluation must be in the hands of skilled analysts who have access to a rich set of pre- and post-program information and who operate with the same kind of time and money that are typically devoted to experiments.
HRDC's evaluations, while inevitably imperfect, have at least benefited from the central direction and accumulated experience of HRDC's Program Evaluation Branch in Ottawa. Canada ranks very high in any international comparison of evaluation efforts.5 In the area of employment-related programs, HRDC's evaluations are second only to those in the United States in terms of the rigour of their methodologies and the reliability of their findings.
However, the proposed devolution of programs is also an opportunity. The provinces may design programs that build on the lessons learned over the past decade. If these programs are then evaluated experimentally, we can amass a body of credible evidence that can inform the evaluation of those programs as we move into the twenty-first century.
1. Employability Improvement Program, 1995. 2. Evaluation of the 1977 Young Canada Works Program, 1978. 3. Final Evaluation 1978 Summer Youth Employment Program,1979. 4. Evaluation Report 1979-1980 Youth Job Corps, 1981. 5. A Cost-Effectiveness Evaluation of Choices, 1981. 6. Evaluation Report 1980 Summer Youth Employment Program, 1982. 7. Taking Stock: An Assessment of the National Stay-in-School Initiative 8. National Institutional Training Program (NITP) Final Report, 1985. 9. Evaluation of Work Orientation Workshops, 1985. 10. Evaluation of Challenge '85 Summer Employment/ Experience Development (SEED) Program Element, 1985. 11. Evaluation of the Co-operative Education Option Final Report,1994. 12. Evaluation of the Job Entry Program Final Report, 1989. 13. Evaluation of Compass, 1986. 14. Evaluation of Employability Initiatives for Social Assistance Recipients (SARS) in CJS, 1994. 15. Summer Employment/Experience Development (SEED) – Program Challenge '86, 1987. 16. Youth Training Option Program Evaluation, 1987. 17. Program Evaluation Student Business Loans, 1987. 18. Evaluation of the Job Development Program Final Report, 1989. 19. An Evaluation of Canada Employment Centres For Students (Overview Report), 1990. 20. Evaluation of the Outreach Program Overview Report, 1990. 21. Final Report Program Evaluation Study of the Canada Employment Centres on Campus, 1990. 22. Evaluation of the Canada/New Brunswick Youth Strategy, 1994. 23. Evaluation of the Severely Employment Disadvantaged (SED) Option of the Job Entry Program, 1993. 24. Longitudinal Study of Training Impacts for the Job Entry and Job Development Programs, 1994. 25. An Evaluation of the Private Sector Internship component of Summer Canada 1983, 1983. 26. An Evaluation of the Canada/Newfoundland Youth Strategy, 1994. 27. Strategic Initiatives Evaluation – Student Work and Service Program (SWASP) Newfoundland and Labrador, 1995.
28. Youth Internship Program – An Assessment of the Demonstration Phase
29. Evaluation of the Skill Shortages Program, 1991.
In addition, over one hundred (100) other sources of information were consulted in the making of this paper.
| ||||||||||||||||||||||||||||||||||
![[Table of Contents]](/web/20060127015944im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[To Top]](/web/20060127015944im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}