Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

4.1 Program Assumptions and Highlights of Findings

In designing Graduate Employment/Self-Employment Program (GESEP), partner departments made assumptions about the needs of participants and the interventions that would best address these needs. This section of the report sets out our understanding of these key assumptions, and highlights of the findings related to each of these. This section is followed by more detailed findings on each of the questions posed in the evaluation framework.

Anticipated Outcomes

• Assumption: The Program would facilitate access to employment for recent graduates, particularly employment related to their post-secondary education.

Participants in the Graduate Employment component appear to have found work more quickly after graduation than the comparison group, but this difference could not be attributed to program participation. Graduates who participated in GESEP are more likely to be working in a career-related job, less likely to have had a period of unemployment since completing the program, and for those who have been unemployed, have had shorter periods of unemployment than the comparison group.

• Assumption: The Program would improve the longer-term employment and earnings of graduates and reduce their dependency on income support programs.

Participants earned more in 1997 than non-participants did, but the econometric results indicate that this cannot be attributed to program participation. Social assistance recipients (SAR)s who completed the program are less dependent on social assistance than comparison group SARs and this can be attributed to the program. Overall, participants are no more likely than non-participants to receive Employment Insurance (EI) benefits following the program.

• Assumption: Incremental job opportunities would be stimulated through both program streams.

About 78 percent of employers surveyed said the Graduate Employment (GE) placements filled new positions. About 30 percent are considered incremental in the longer-term in that the position is still being filled by either the graduate or a replacement employee in cases where the graduate has left the company or organization. This outcome is comparable to that of other wage subsidy programs.

While many employers indicated that they would have hired someone to do the GESEP job if they had not received GESEP funding, most would not have hired a recent graduate. Just under 50 percent of employers surveyed would have hired without the subsidy and 12 percent said they would have hired the same graduate without the subsidy. This dead-weight loss impact is again comparable to that of other wage subsidy programs.

Seventy-five percent of Self-Employment (SE) entrepreneurs who are still in business said they would not have started their businesses without the subsidy. Businesses supported through SE are small with few or no other staff employed; however most expect to continue operating and have potential for growth.

Target Group

• Assumption: Recent graduates having difficulty in finding work, and social assistance recipients (SARs) in particular, need support to make a successful transition to the labour market in the province.

The program was not as well targeted as it could have been on graduates who had experienced difficulty in making the transition to work. There are opportunities to improve on this aspect of the program with some fine-tuning of design and delivery. SARs did not participate to the extent intended in the GE component, primarily as a result of difficulties in linking the SAR service delivery network to the program. On the other hand, SARs were well represented in the SE component, a factor of flexibility in the approval process.

SARs have benefited more from the program than other graduates. Proportionately more SARs who completed their placement are still working with their GESEP employer compared to the GE participants in total. As a group they have less dependency on social assistance. They are slightly more dependent on EI than other participants, but are however still finding periods of employment.

Overall, graduates who best matched the criteria for the program were more likely to finish their placements and to find the experience valuable in making the transition to work.⁶

• Assumption: Businesses in the strategic sectors represent the best opportunity for career-related work experience.

Just under 50 percent of employers were in strategic sectors, but there is no difference in the likelihood that those who have finished strategic sector placements are now working in career-related employment. Those who finished strategic sector placements are earning more than other participants.

Interventions

• Assumption: Career-related work experience is important to a successful school to work transition.

The degree of career-relatedness of the placement had no impact on the economic outcomes of income and use of income support for the individual participant or for society in general. However the degree to which placements matched career goals was strongly related to satisfaction levels of participants with their placements. Well-structured work experience with a supportive employer is considered by participants to be very important to their job satisfaction and preparation for future work.

• Assumption: Quality matches between employers and graduates (in terms of relationship to area of study and career goals) would improve the chances for program success.

While there is no relationship between the quality of the match in terms of career-relatedness and successful program outcomes, the level of support and mentoring provided by employers was important to participants.

• Assumption: Work experience would provide access to training and influence decisions on further education.

About 90 percent of placements included either on-the-job or formal training. Participants have returned to school less than the comparison group (30 percent compared to 50 percent). However, the participants’ reasons for returning appear more focused on advancement within a chosen career, and the decision was influenced by the placement for the majority of returnees. The comparison group members were more likely to go back to school because they could not find work.

• Assumption: Young entrepreneurs need support as well as subsidies to make it through the early stages of setting up their businesses.

Participants valued in particular the practical advice provided on immediate issues they faced in setting up businesses. The networking among participants, ongoing support of the facilitator and linking with experts through distance technology were useful features introduced through the program. The subsidy helped see the businesses through the critical first year by providing a steady cash flow and funds to invest in capital expenditures.

4.2 Detailed Findings

4.2.1 Relevance

To what extent did the project reach the intended target groups? Were participants representative of the target groups? If not, why not?

The following table compares the Profile of GESEP participants with the post-secondary graduate population in the province.

Graduate Employment

1. While the majority of GE participants met eligibility criteria, about 22 percent of participants had not experienced the required period of unemployment at the time of approval. The program could have been better targeted with some fine-tuning of design and delivery.

The program was to target those having difficulty in finding work, using a criterion of at least three months of unemployment or under-employment at the time of application. Seventy-eight percent of GE participants were either unemployed or underemployed at the time they were placed, 10 percent were in school immediately prior to starting GESEP and 12 percent were employed and not under-employed.⁷

A number of factors influenced the selection of participants. Program administrators attempted to be client-centred versus rules-oriented in considering the merits of each case. Where exceptions to guidelines were felt warranted, they were made. In some cases, administrators experienced pressures to approve applications as presented because of the poor overall prospects in the labour market, as opposed to requiring the three-month job search.

Program starts were concentrated in the early part of each fiscal year. The majority of graduations occur in that period and HRE made efforts to start contracts early in the fiscal year to avoid carry over of funding commitments. This pattern of approvals appears to have led to the funding of some graduates who were not experiencing difficulties, and meant less money was reserved to assist graduates later in the year who needed the program after a period of unsuccessful job search.

Program administrators marketed the program to colleges and graduates before school completion, and this created expectations of program availability. It was then more difficult to require a job search period for graduates who had taken the initiative to find a placement. If the applicant was on a work term where GESEP would lead to continuation of the job (common for some college programs), exceptions to the three-month criterion were made.

As discussed in detail later, participants who began GESEP immediately after finishing school have benefited from the program to the same extent as those who met the eligibility requirements. Cases where participants were employed and not underemployed do not appear to have benefited to the same extent. Many of these participants left their GESEP placements early because they found better jobs on their own.

While the approach resulted in a responsive program that helped individual graduates find career-related employment, it weakened somewhat the value of the program as a pilot to test an intervention for youth experiencing difficulty in finding work. There were opportunities to fine-tune the design and delivery that were identified in the formative evaluation report.

2. Social Assistance recipients did not participate to the extent intended in the pilot design.

Forty-three SARs participated in GE compared to the 50 targeted when the program was launched. The target was not adjusted as the program funding increased. Presumably there was opportunity to do so, given that there was progress made in reaching the original target. If the same target proportion had been used, SAR participants would have increased to 95.

Data are not available on the total SAR caseload who would have been eligible for GESEP. Currently there are approximately 100 heads of social assistance cases who are under age 29 and who have completed a post-secondary program, from a total monthly caseload of 34,000.⁸ In comparison, the average number of SARs placed in the GE component each year was only 14. This indicates there was some potential for more participation by SARs.

SAR participation was primarily affected by difficulties in linking the social assistance service delivery network to the program. There were also a high number of placements initiated by employers. SARs likely faced a disadvantage compared to other graduates in linking up with employers in these cases, particularly in that they were not provided with job search assistance through GESEP.

Over 95 percent of SAR participants were college graduates. They were just as likely to have studied business, and twice as likely to have taken trades training compared to the total GESEP group. Of note, about 7 percent studied engineering (degree or technology program) compared to 11 percent of the total GESEP group.

3. The mix of university and college graduates does not match the profile of graduates, a consequence of differences in labour market opportunities for graduates and certain features of college program design that could have skewed participation in their favour. Business graduates from both universities and colleges were more likely to participate in the program.

The lower participation level of university graduates in GE has been consistent since the earlier Graduate Employment Program (GEP) was initiated. A number of factors likely influence this:

• University graduates are more successful in finding work and therefore have less need for the program;⁹
• The focus of the program on the private sector limited the opportunity to participate for graduates of some university programs (e.g. teachers, some health professions);
• Private colleges (and more recently the public college network) have placement officers who actively market graduates and the availability of subsidy programs, which may be a factor (18 percent of participants surveyed said they found their GESEP placement through their college);
• A number of college programs have work terms at the end of the study program and graduates were considered eligible for GESEP as soon as their work terms ended. This feature of college program design seems to have given the edge to certain graduates as they were able to continue on with their work term employers through the use of GESEP subsidies;
• Business graduates are much more likely to participate in GESEP than graduates from other areas of study. This is true of graduates from both college and university programs. This may be a consequence of the types of employers participating.

4. A higher proportion of college graduates had a previous attachment to the labour force, as evidenced by previous use of UI. This is consistent with the trend within the student population not to enter college immediately following high school graduation.

Forty-six percent of university graduates and 62 percent of college graduates had received UI benefits at some point between 1993 and their participation in GESEP. A growing number of high school students are not entering college right after high school¹⁰ and this likely leads many to become eligible for UI prior to studies. There has been no similar increase in delayed attendance in the university student population. In this respect, the profile of GESEP participants is consistent with the graduate population.

5. While the program was to target businesses in the strategic sectors, they represented just under half of the participating employers. This result is a consequence of shifting the focus of the program to respond to the opportunities in all areas of the province. Strategic sector businesses more frequently sponsored more than one graduate.

Forty-seven percent of the GE employers were in sectors identified in the Province’s Strategic Economic Plan. Initially, these sectors were to be the primary target for the program. Over time, this was pursued less rigorously. It was not clear in some cases which businesses fit these sectors. (For example, professional services and tourism are two SEP sectors that could include a broad range of businesses.) Applications from these sectors were not sufficient to use the funds allocated to the program, and it was thought that there were fewer employers in these sectors in rural areas. Program administrators opted to expand their focus in order to distribute placements across the province and to fully expend funds.

By completion of the program, 42 percent of the rural employers were in strategic sectors compared to 55 percent of urban area employers being in these sectors. Approximately 16 percent of GE employers were repeat sponsors in that they hired more than one graduate over the life of the program. Strategic sector employers represented 62 percent of repeat employers, which is likely a combination of priority given by the Program and an indication that they are smaller and in need of government support to expand.

6. GESEP employers tend to be small and the majority are growing or are stable in size, which is consistent with the overall focus of the program.

The median number of employees is 5 full-time, 0 part-time. (Employers surveyed range in size from no employees to 250 full-time and 45 part-time employees). Forty percent of those surveyed have grown in the past three years, 44 percent are the same size and 16 percent have reduced their staff. This profile fits with the focus of the program on small but growing businesses.

Self-Employment

7. The profile of SE participants is very similar to that of participants in other self-employment programs and of young entrepreneurs in the Atlantic Provinces.

Characteristics of SE participants:

• Participants were more likely to be men than women (65 percent are men, 35 percent women);
• They were older than participants in the GE component;
• They were less reliant on UI prior to the program than GE participants;
• Almost half were university graduates;
• They had not tended to study a business program;
• Over half had previous work experience in the field in which they started the business — two had over 20 years related experience;
• They were not likely to have young dependant children (none of the women who participated had dependent children).

The profile by gender, age, previous experience, and dependants is consistent with the profiles of entrepreneurs developed in two recent reports: the evaluation of the Human Resources Development Canada (HRDC) Self-Employment Assistance program¹¹ and an Atlantic Canada Opportunities Agency (ACOA) study of young entrepreneurs in the Atlantic Provinces.¹²

8. A high proportion of SE participants were SARs, primarily the result of considerable flexibility in approval of proposals.

While no specific SAR target was set, 10 of the 58 participants (17 percent) were receiving social assistance prior to participation in the program. Key informants expressed the view that a number of the business proposals from SARs were marginal but were approved as the result of outside representation on behalf of SARs to the assessment committee. While the overall success rate for SARs is comparable to that of other SE participants, all 4 of those identified as having a weak business idea are no longer in business.

In summary, the profile of GESEP participants does vary somewhat from the overall graduating population, and is influenced by labour market trends in general, and in some cases, by differences in the design of post-secondary programs and support to graduates. The Graduate Employment component did not target as well as intended on graduates experiencing difficulty in making the transition to work.

Were the services/interventions provided consistent with the needs of the target groups? And of employers?

Graduate Employment

1. The GE component did not provide ongoing support to participants on the assumption that the needs of graduates were limited to obtaining work experience relevant to their career interests. This approach did meet the needs of the majority of participants as employers provided the support required. There was insufficient follow-up and monitoring. As a consequence, there was a lack of needed program support in cases where there was a breakdown in placements.

The majority of participants interviewed were satisfied with the work experience in relation to their studies and career interests (median score 8 on a 0 –10 satisfaction scale, 0 = not at all to 10 = a lot). It should be noted that the majority of the comparison group surveyed were only marginally less satisfied with the relevance of their employment since graduation and the usefulness of the work in preparing them for future job search.

The focus group provided some additional insights into the relevance and impact of the program. The group was mixed — some having had a positive experience, others having experienced a poor match. The former group had positive things to say about what they gained from the placement in terms of job specific skills, work place awareness and confidence, and the support of their employer. The latter group, most now unemployed, did not feel the placements helped them in getting established in the work force. Several focus group participants were obviously experiencing great difficulty because of this.

Focus group participants were also concerned about the lack of monitoring of their progress and satisfaction during the placement by program officials, and difficulties they had in accessing information/assistance when needed. Their sense was that the program was geared more to employer needs than those of the graduates.

Program administrators acknowledged the lack of resources to carry out follow-up, which was exacerbated by the reorganization and downsizing that occurred over the life of the program. They tried to provide at least a modified client-centred approach in the up front assessment of the employer-graduate match, and to a certain extent, to counsel applicants on job search. They cited the need for a case management approach that was not possible to provide because of overall resources and the centralized delivery of the program.

2. Participants were generally not satisfied with the wages paid in GESEP placements. Wages paid do appear to be lower than those of new graduates in general. The majority of participants are earning more since completion of the program and are earning more than the comparison group.

Participants who were surveyed, rated their satisfaction with wages lower than all other aspects of the placement (median rating of 6 on a 0–10 scale). Participants in the focus group reinforced this view. Employers were eligible for a subsidy of 60 percent of the wages up to a maximum of $10,000 per year. The salary at which the benefits to the employer were maximized under this formula was $16,600. Sixty-two percent of GESEP placements paid this amount or less, which is an indication that the program did tend to set a ceiling on the wage rate in the majority of placements.

The recent Post-Secondary Indicators ’98 report of the Department of Education provide some evidence that average GESEP wages are comparatively low. Thirty-four percent of graduates surveyed for that study reported earning $8.00 or less in their current or most recent job, which is equivalent on an annual basis to $16,640.¹³

The majority of participants surveyed said their earnings have increased since GESEP. Their average total income in 1997 was higher than that of the comparison group ($20,630 compared to $17,406).

Participants felt more stringent guidelines should be applied to wages paid in subsidized programs. The development and application of wage rate guidelines present challenges for program administrators in weak labour markets. However, these could be supported by research on current wage rates for graduates and use of this information in negotiating placements with employers as well as more promotion of the program and careful selection of employers to provide quality placements.

3. Employers felt the wage subsidy did give them the opportunity to hire graduates they would not have hired otherwise, and were satisfied with the contribution of the graduates to their businesses. Employers also thought they had made a contribution to developing the graduate.

Graduates brought a number of benefits to employers — specific skills that businesses and organizations did not already have and additional strength and assistance in expanding markets and delivering services.

Employers in the focus group emphasized that graduates had to meet the needs of their businesses before the subsidy was worth their time and other investments in developing the graduate. They thought that their contribution to the program went beyond providing work experience. They saw themselves as mentors and developers of the graduates, an opinion that was borne out by those GE participants in the focus group who had a positive experience in the program.

4. The program goal of promoting graduate-initiated placements was not met. Thirty-four percent of funded GESEP participants used the wage subsidy as a marketing tool; that is they approached the employer and informed him or her about the wage subsidy. Employers initiated the other 66 percent of placements.

The program was designed to encourage graduates to market themselves to employers, using the potential of a wage subsidy as an incentive. This was seen as a method to assist graduates in finding a placement to match their interests. It was also intended to help avoid subsidizing employers who would have hired without the program. Survey results indicate that about a third of the placements were generated by graduates.

A couple of factors likely influenced the outcomes for this program objective. While program administrators made efforts to promote the program to graduates across the province, the centralized delivery seems to have limited awareness of the program. Fifty-two percent of the participants indicate that they first heard of GESEP directly from the employer who hired them using the subsidy. As well, graduates who were informed about GESEP were given no job search assistance or assurance that funds would be available if they found an interested employer. Fine-tuning the program to increase promotion and pre-approve participants would likely have increased the number of placements initiated by graduates.

Self-Employment

5. The support built into the SE component was found to be useful, but was not fully integrated to provide a continuum of support throughout the program.

The subsidy helped most SE participants in the crucial first year of operation¹⁴ — it provided an even cash flow, and in most cases allowed them to make some capital investments to establish the business.¹⁵

Participants found the facilitators, with their small business background, to be useful advisors, but their role was not developed to the extent intended. There were three different facilitators over the life of the program, which affected the continuity and momentum of building this new role. In the absence of a defined process for facilitation, each brought his or her own personal approach to the job, which was not necessarily consistent with the vision of the pilot. For example, a structured procedure for monitoring and follow-up with individual participants was set up by the first facilitator but was not continued in the same way by his replacement.

One facilitator covered the whole province and this meant that individual contact was not frequent, and in-person visits were limited. In the key informant interviews, the email connection provided through the program was cited as an essential tool for carrying out this support role. The support provided to help the participants network with each other appeared to be effective — these included the email link, regular teleconferences with guest speakers, and two workshops.

The mentor program was not developed as intended in the pilot design, apparently a result of turnover in the facilitator position. This would have linked participants with business people in their area. As a result, the facilitator provided what mentoring was done and he was spread too thinly to do this effectively.

The assessment process provided by the P.J. Gardiner Institute was a reasonably priced service that provided some good feedback and guidance to applicants. Some administrators thought that more definitive assessments would have been more helpful to them in making funding decisions. Other key informants felt there was never a shared understanding of the expectations of this screening process.

Overall, the linkages between the three phases of the program — pre-assessment by the P.J. Gardiner Institute, program administration in HRE, and facilitation by the College of the North Atlantic — did not appear to be as fully developed as they should have been to provide a continuum of support to participants intended in the pilot design. The different locations of the centrally delivered phases meant communications among administrators was not as efficient as it could have been. Also, the respective roles and role performance expectations were not fully clarified.

4.2.2 Program Success and Cost-Effectiveness

To what extent has GESEP succeeded in removing barriers to employment and training?

Graduate Employment

1. GESEP participants had shorter periods of unemployment after graduation than the comparison group and also experienced less unemployment in the year after graduation than non-participants.

Twenty-seven percent of GESEP participants surveyed were unemployed for some period in the 12 months prior to the program — five months was the median period. This compares with 62 percent of the comparison group being unemployed at some point in the first 12 months after graduation.¹⁶ Seven months was the median period of unemployment for the comparison group.

The program was intended to help graduates avoid long periods of unemployment and the scarring effect of such unemployment on their long-term employability. The above comparison seems to indicate that program participants found employment more quickly after graduation. However, the numbers also reflect those who were already employed at the start of the program (either fully or under-employed), so one cannot attribute this difference in unemployment periods to the program.

2. The subsidy removes some risk for employers in hiring new graduates and provides funds to offset their investment in training and developing graduates.

Fifty-four percent of employers surveyed said they would not have hired anyone to do the GESEP job without the subsidy. Nineteen percent would have hired someone else to do the job without the subsidy and most indicated that they would have hired someone with more experience. Only 12 percent of employers indicated that they would have hired the same person without the subsidy.¹⁷

More than 90 percent of employers said they provided their graduate with on-the-job training, and approximately one-half provided other types of training. Almost half of these employers indicated that they did not know the value of the training that they provided. The rest ranged the value of training from no direct cost to $30,000,¹⁸ with the median amount being $3,000.

These findings are consistent with the information gathered in the employer and participant focus groups on the value of the subsidy in offsetting the cost of additional staff with limited experience, and the training investment made by employers.

3. More participants are working in career-related employment than the comparison group.

Seventy-one percent of participants surveyed are currently working in their field: 44 percent with the GESEP employer and 27 percent with another employer. Thirty-nine percent of the comparison group are working in their field.

4. The majority of participants thought that the work experience would help them in finding other related work. However, for participants who did not have a positive experience in the program, found that the experience had not helped them make the transition to work.

Over 80 percent of participants interviewed indicated that the GESEP work experience would help in a future job search. Responses were similar for those working and not working now.

Focus group participants had two perspectives. The majority of this group of nine was no longer with their GESEP employer, and several are still unemployed. Those whose placement did not work out felt the program alone was not enough to overcome the employment challenges faced by graduates in the Newfoundland and Labrador labour market. Others shared the positive views of the majority of those surveyed.

There was a noticeable overall difference in the opinions of graduates in this focus group compared to those held during the formative evaluation study on both the quality of the work placements and the outcomes of the program.¹⁹ At the time of the formative study, most participants were optimistic that they would complete the placement and would be retained by the GESEP employer. This recent focus group discussion reflects more fully the outcomes of the program.

5. Support generally considered needed by SARs in finding and retaining work was not provided in the program. However, SARs who participated fared well.

Fewer SARs participated in GE than planned. Nonetheless, those SARs who completed their placements have been more successful than the GESEP group as a whole in most program outcomes (see finding 4 on page 27).

Key informants thought that the program design and implementation was weak in facilitating the participation of SARs. Staff resources to counsel and market SARs, clarity of roles and commitment and accountability for ongoing supports were all areas that could have been strengthened. The program is now administered by HRE, which has merged employment and social services programs. This department offers the potential for significant improvement in program impacts for SARs. This has not happened yet, mainly because the department is still in transition and program redesign efforts are still underway.²⁰

Self-Employment

6. The SE component subsidy has been effective in assisting new business start-ups.²¹

To what extent has the project assisted participants to achieve economic self-sufficiency?

• What activities/interventions were most effective? For what type of participants? For completers? For non-completers?
• Did graduates successfully establish small businesses or get long-term jobs related to their studies?
• For what reasons are some participants unemployed and on income support after the project?
• Did the project motivate participants to go on to further training or education?

Graduate Employment

1. In terms of current employment status, program participants are doing well in the labour market, and have fared better than the comparison group. More are working and moreover, in an occupation related to their studies. The majority are also earning more than they did during the placement. However, the difference in earnings could not be attributed to program participation through the analysis carried out for this study.

Highlights

• Seventy-one percent of participants are now working in a job related to their studies, compared to 39 percent of the comparison group;
• Participants aged 30 or older are more likely to be currently looking for work than those in their 20s;
• All participants are equally likely to be working in their area of study, regardless of the type of institution they attended. Graduates from private colleges are the most likely to be looking for work;
• Graduates in rural areas are slightly more likely to be working for their GESEP employer. They are also more likely to be looking for work. Geographic location is not related to the likelihood that participants are working in a job related to their field of study;
• Male participants and university graduates are more likely to be currently in school.

Seventy-five percent of graduates still with their GE employer are earning more than they did during the placement, and 90 percent who are with other employers are earning more than their GESEP salary. However, these findings must be tempered by the results of the econometric analysis on program impacts. While the participant group has performed better than the comparison group on all measures of economic outcome (total income, earnings, period working and use of income support), none of these outcomes could be attributed to program participation (see finding 1 on page 34).

Econometric analysis was also carried out on the impact of the characteristics of graduates and their work on observed outcomes. Findings from this analysis are as follows:

• Type of post-secondary education: GESEP was open to all graduates of post-secondary programs, from nine-month college certificate programs, two- to three-year college diploma programs, through to four-year or longer university degrees. The type of program was a significant factor only in explaining the proportion of time spent on EI following participation. Graduates of university and college certificate programs were both less likely to have spent post-program time on EI than the diploma program graduates. University graduates had higher earnings (by about $6,000) compared with other graduates.
• Field of study: Field of study has a large effect on average earnings. In particular, graduates who study applied engineering were likely to earn about $6,750 more annually than people in other fields of study. Graduates in trades also showed high post-program incomes; they earned about $5,200 more.
• Employment characteristics: In general, the characteristics of the work had no bearing on success. Graduates who reported that the placement or longest job they held since graduation provided skills that were useful for their future career were likely to earn more (about $600) than people who didn’t, but this effect is significant only at the 10 percent level.²² When we looked at success among the participant group alone compared to employer characteristics, the only important characteristic was that people who were laid off by the employer due to shortage of work, worked less in the post-program period, which is not a surprising finding.

2. The earlier GE participants are just as likely to be working in their field as those who participated more recently.

Seventy-four percent of those who finished in 1995 are currently working in a career-related job (either with the GESEP employer or elsewhere), compared to 68 percent of those who finished in 1996 and 72 percent of those who finished in 1997.

As expected, those who finished in 1997 are the most likely to be currently working for their GESEP employer. While one would expect more attrition among earlier program participants, those who finished in 1995 are almost as likely to be currently working for their GESEP employer as those who finished in 1996.

3. More participants who completed their placement are still working in a career-related job compared with those who terminated early. Overall, however, approximately equal proportions of both groups are working at this point. There are noticeable variations based on the motivating reasons for early termination. In particular, it is evident that a larger number of voluntary early completers, went on to career-related work. This group was likely in less need of the program.

• Seventy-one percent of participants completed their placement;
• Seventy-six percent of completers are in career-related work, compared to 55 percent of those whose placement terminated before the 12-month period was over;
• Eighty-one percent of completers are working now compared with 79 percent of those who did not finish their placement;
• Thirty-three percent of those who finished early because the business faced financial difficulties or work shortages are currently working in career-related employment, compared to 42 percent of those who had problems with the placement, and 90 percent of those who left on their own. This latter group would appear to be more self-sufficient and not as likely in need of the program as others, which relates back to the finding on the targeting of the program.

4. The program had positive results for social assistance recipients who completed their placements. While they are more likely to use EI following the placement, they continue to find periods of work.

• Sixty percent of SARs surveyed are still working for their GESEP employer, compared to 47 percent for the total GE group. Another 15 percent are working elsewhere in employment related to their studies and 5 percent are working in other unrelated employment. Five percent have gone back to school.
• Current income levels of SARs surveyed are not that different from GESEP participants as a whole. For 1997 the median is $2,200 lower, however if only those who completed the program are compared it is just $1,500 lower.
• The majority of SAR participants who terminated their placement early reverted to receiving social assistance. The reasons for early termination are consistent with those for the GE participants as a whole.
• Of those who completed their placement, 58 percent received EI at some point after the placement, compared to 37 percent of all participants. The majority of these SARs have had other work since GESEP.
• The econometric analysis shows that program participation is a significant factor in reducing reliance on social assistance (by about $5,600) for those graduates who had received social assistance before GESEP.

5. Participants who were in placements that were closely related to their studies are doing no differently now than those who indicated that the placement was less related to their studies.

In general, the career-relatedness of the placement had no bearing on success. This is consistent with the literature on work experience for youth: having a well-structured work experience with sustained contact with adults is important to program success.²³

This finding should be considered in light of the fact that there was a generally positive rating of the work experience, and the rather subjective nature of the question. However, it is important since the necessity for career-related work experience for future labour market success is one of the key assumptions of the program design.

6. Fewer GE participants have returned to school than the comparison group but their reasons for returning appear more focused.

Thirty percent of all GE participants have gone back to school either full-time or part-time at some point since they finished GESEP, the majority in studies related to their previous education and in order to advance in their chosen field. Fifty percent of the comparison group have returned, although they are more likely to indicate that they returned to school simply because they could not find a job. The difference in numbers is in part due to the fact that more participants are working than non-participants. Highlights of the characteristics of those who returned to studies include:

• Eighty percent of participants who returned have pursued more training related to the program they took before GESEP. One half said they did so to advance in their chosen field, and another 32 percent did so to improve their chances of getting the type of job they want;
• Comparison group members were more likely to go back to school to improve job chances (60 percent) or because they could not find work (22 percent);
• Twenty five percent of participants who are still working for their GESEP employer went back to school at some point compared with 32 percent of others;
• Those in their 20s are more likely to have gone back to school;
• University graduates are more likely to have gone back to school. College business graduates are the least likely to have gone back to school.

The focus group results support what was learned through the survey. Most focus group participants had returned to school either full-time or part-time, and the placement had influenced this decision. Most recognized that further training was needed in order to advance in their field. Many of those who have not returned to school want to but lack the financial ability to do so.

The program was intended to help with both the school to work transition and decisions on further education, in recognition of the fact that work and learning are increasingly integrated. The program has made a contribution towards helping with these transitions by informing education choices, providing the opportunity to work and learn at the same time, and by providing incomes to finance further education.

7. The outcomes for graduates placed in sectors identified in the Province’s Strategic Economic Plan are similar to those in other sectors, except that their current incomes are marginally higher.

Forty-seven percent of the GE placements were in sectors identified in the Strategic Economic Plan.²⁴ Graduates in these placements are equally likely to be employed now, and to be working for their GESEP employer. Their current incomes are marginally higher — a median difference of $2,000.

8. The program has generally funded new positions and given graduates an edge over experienced workers. The majority of employers interviewed said the GESEP job was a new position that would not have been filled without the subsidy. Few employers would have hired the graduate without the subsidy. The majority of those who would have hired someone else would have looked for someone with more experience.

Seventy-eight percent of employers surveyed indicated that the graduate was hired to fill a newly created position. Based on the employer survey, we estimate that 30 percent of the placements were incremental jobs; that is, a new position that was created for the GESEP placement and it is currently still being filled either by the GESEP participant or a replacement.²⁵ Thirty percent of employers where the GESEP participant left have hired a replacement and only 3 percent received a government subsidy for the replacement. Most of these were cases where the graduate left voluntarily.

The program has resulted in some graduates being chosen over other experienced workers for existing vacancies. Twelve percent of GESEP placements were existing positions. In most cases, the employers surveyed said they would have hired someone with more experience without the subsidy. In essence, this type of displacement is acceptable and justified given the intent of the program to assist graduates to find related experience. There was no indication that existing workers were displaced by GESEP participants.

There is evidence of only a small amount of dead-weight loss in the program, that is, placements where a subsidy was accepted for a worker who the employer would have hired anyway. As reported earlier, only 12 percent indicated that they would have hired the same graduate without the subsidy.

The displacement and dead-weight loss impacts are common concerns with wage subsidy programs. GESEP appears to have done better than other programs in this regard. Evaluations often show that the combined displacement and dead-weight loss can run as high as two-thirds or more of the total increase in hiring. The higher the rates, the less the program is targeted .²⁶

9. About 30 percent of GE participants surveyed said they would have left the province if they had not received a job through GESEP. Five of the 13 SE participants surveyed who are still in business said they would have left the province if they had not received funding from GESEP.

This is consistent with the trend in post-secondary graduates leaving the province, and an indication the program is meeting its objective of providing local opportunities for graduates who would prefer to stay in the province.

10. Participants who previously received EI are less likely to have received EI since participation. SARs are more likely to have received EI benefits than other non-SAR participants following the program.

• Fifty eight percent of all GESEP participants received UI benefits at some point between 1993 and the start of their GESEP job. In the 12 months prior to GESEP participation, 42 percent received UI benefits;²⁷
• Thirty six percent of all GESEP participants have received EI benefits since they completed the program, which is comparable to the extent that comparison group members have used EI since their longest job following graduation;
• University graduates are 14 percent less likely than college graduates to have received UI benefits prior to participating in the program. They are also 13 percent less likely to have received EI benefits since completing the program;
• Fifty three percent of those who received EI in the 12 months since completing the program also received UI benefits in the 12 months before participating;
• Those who received UI in the 12 months before beginning GESEP are slightly more likely to be currently working for their GESEP employer, however are less likely to be working for another employer in a career-related job;
• Of those SARs who completed the program, 58 percent received EI at some point since the program began, compared to 37 percent of all participants.

Overall, the results indicate that the program is not creating a dependency on EI. While SARs are using EI more, the employment impacts of the program have at the same time been relatively positive for this group, as they have found periods of work since the placement.

Self-Employment

11. Of those SE participants surveyed or whose current status was known by program administrators, 54 percent are still operating their business. This is consistent with the national survival rate for businesses within the first year or so of start up.

Because of the low response rate to the survey, we used data from the survey completed by the provincial government in December 1997 and information from program administrators to determine the status of businesses for another 18 participants. Of these, 8 are running a business (6 full time and 2 part-time) and 10 are not. In total, we determined that at least 21 out of 41 participants (54 percent) are still operating their business. Statistics on business survival rates are not systematically collected, but agencies providing services to new entrepreneurs advise it is commonly understood that 50 percent of new small businesses established in Canada are still operating after their first year.

12. It is not possible to be definitive about the characteristics of successful SE participants, given the small numbers and their diversity.

The following are some highlights of characteristics of those still operating their business:

• Three of the 10 social assistance recipients are still operating their businesses;
• Successful business operators are slightly older than the group in total;
• A slightly higher proportion of male participants are still in business compared to female participants;
• Of the six participants who had received UI in the past, only one is still in business;
• Participants who are currently running their own businesses are more likely to either own their own home or live with their parents, whereas those no longer running their own business are more likely to be renting.

13. Businesses established are small and created few additional jobs. Businesses that have been closed tended to be in product/service areas where competitors were already operating.

Of the 8 participants interviewed who are still in business, almost all are sole proprietorships, and only 2 of them have employees. Several are assisted by family and friends. Given that these businesses have only been in operation a short time, the potential for them to grow must be kept in mind.

A wide variety of businesses were funded, ranging from internet/web site services, to craft production, running self-improvement seminars, fitness consulting, photo refinishing, and helping adopted children find their birth parents. As broad a mix of businesses are still operating as those that are not. The one common characteristic appears to be that where businesses are not operating, a direct competitor already existed for the product or service. An exception is crafts producers, who for the most part are still in operation. Firms providing computer services are also doing well.

14. The SE subsidy was a contributing factor in the decision to set up and see a business through the start up period.

Only 25 percent (5) of those surveyed who are currently in business thought that they would have started their business without the subsidy. Former facilitators observed that the subsidy provided a steady cash flow in the crucial first year of operation, and the funds were often used for capital investments that helped build the capacity to provide service and products.

The experience of those not funded is also an indicator of the program impact. Only 6 of the 23 applicants we interviewed who had not been funded by SE²⁸ had gone on to set up a business without a subsidy from other sources.²⁹ They have been less successful than the SE group — 4 of the 21 surveyed who did not receive financial assistance are now in business. The incomes of those who did go into business were substantially lower in 1997 than for SE participants still operating their business.

Almost all of those in the non-participant group who did not start a business gave lack of funding as the reason. In the interviews many indicated that they have been having difficulty finding work. As a group they were noticeably, and naturally, critical of the fact they had not received funding from GESEP.

15. Participants who were given conditional approvals by the P.J. Gardiner Institute have been somewhat less successful in establishing and maintaining their businesses, but a significant number are still in business.

Eleven of the 58 participants approved were assessed by the Institute as “conditional” in that they needed more preparation or assistance in certain areas in order to be viable. These were the most at risk participants who could potentially benefit the most from the assistance of the facilitator. While they are less successful in actually opening and maintaining their businesses, 36 percent are still operating (compared to the overall 54 percent success rate), an indication that the supports provided in the program were useful for that group. This also indicates that the assessments of the Institute were effective in identifying those who needed more assistance in getting established.

To what extent has GESEP prepared participants for economic self-sufficiency?

1. increased motivation and self-esteem?
2. assisted in development of business plan?
3. provided relevant work experience?
4. provided self-employment/business skills?
5. facilitated access to training and business support services?
6. provided mentoring/role models?

1. Both components contributed to preparing graduates for economic self-sufficiency in a variety of ways. However, graduates who did not participate were only slightly less positive about the contribution of their employment to their self-sufficiency.

The majority of GE participants thought the work experience would be useful in future career decision-making and job search. In the employer and participant focus groups, it was stated that most employers played the role of mentor to the graduates, which contributed to the graduates’ perceptions of the value of the experience. Another benefit was the network of contacts developed by graduates through their work. Participants in the focus group spoke of developing work place skills beyond their specific job through special assignments, and developing useful contacts for future job search.

In assessing this outcome, one also has to consider that the comparison group was almost as positive as participants about the value of their work to their future career plans and capacity to find work.

The SE participants learned about entrepreneurship through doing, yet in a supportive environment. One participant observed that the skills he developed in business planning are transferable to other settings. The range of support and the peer network of participants created through the program were considered useful in seeing the businesses through the initial stages.

Key informants emphasized the need for participants to move beyond planning to the operational stage in order to learn about entrepreneurship. They expressed some doubts about the effectiveness of the program for those who were funded while still developing their business plan, as this appeared to reduce the urgency of getting on with the business.

The program was administered in a flexible, client-centred approach. One example is that participants were allowed to use their training allowance to purchase computers. One support that was not developed was the network of business mentors. As a result the facilitator had to take on this role, which was not as effective as the original intention in the program design.

What are the benefits to society, participants and the government as a result of GESEP? What effect did the program have on income support payments and personal finance?

Graduate Employment

1. The participant group performed slightly better than the comparison group on all of the success indicators. However, program participation is a statistically insignificant factor³⁰ in explaining the higher average success of program participants.

Econometric analysis was undertaken to measure the impact of the program based on comparing the performance of the GE participants in the post-program period with a comparison group. Details on the approach and results are included in the technical annex to this report and summarized below.

The indicators chosen to measure the effectiveness of the program were:

• the income earned in 1996/1997 excluding the period when participants were still working in GESEP-funded jobs;
• the amount of time working or receiving Employment Insurance payments in this period;
• and the use of social assistance in the same period.

Table 7 shows that the participant group performed slightly better than the comparison group on all success indicators. However, the only two of these measures where there is a statistically significant difference in the means of the two groups is in the use of social assistance and the proportion of the time spent working. The econometric analysis was carried out to determine if the differences observed above were in fact due to program participation, or could better be attributed to personal characteristics of the people in the participant and comparison groups. The results show that program participation is a statistically insignificant factor in explaining the higher average success displayed by the program participants.

Self-Employment

2. SE participants still in business have more variable incomes than those no longer in business, and overall their incomes are lower.

Econometric analysis of impacts of the Self-Employment component was not carried out, for reasons explained in the methodology report and technical annex to this report. Essentially the number of participants was too small to be able to attribute any observed differences in income to the program. The post-program period is also too short to measure long-term impacts from self-employment, given the low incomes of owners during the initial business-growing period.

The variability of incomes is evident from the survey results. The income of those currently operating a business is more variable than for those who are not now operating a business. Their average income (both mean and median) is lower than for those not currently running a business.³¹ The table below compares income for the two groups. Frequencies are reported rather than percentages because the numbers are so small.

SE is a more expensive program than GE ($22,300 per participant, including administration and income support, compared to $10,000 for GE) but is comparable to the cost of other entrepreneur support programs in the province. This level of investment highlights the need for an effective selection process and continued monitoring based on performance standards to ensure commitment of participants.

Participants in SE are less likely to receive EI following the program, as one would expect since the program subsidy is not EI eligible. Five percent of SE participants received EI benefits since their subsidy ran out, compared to 36 percent of GE participants.

Has the Self-Employment component encouraged the establishment of small businesses (and subsequent creation of jobs) in growth sectors and remote regions? How many have succeeded/failed and for what reasons? Are potential employers in growth sectors able to find appropriately skilled employees?

1. The program has assisted graduates who might not have been eligible for other funding to establish a wide range of businesses.

Twenty-eight percent of businesses funded were in rural areas and 72 percent were in urban areas. Urban participants are more likely to be currently running their business full-time, however more rural participants are both working and running businesses or running businesses seasonally. In total there is not much difference between the proportions of rural and urban businesses still operating. Of the SE participants who were interviewed, all of those who are working in jobs that are not related to their studies or who are unemployed are in rural areas.

Fifty-four percent of SE participants are still operating their business. While the numbers involved are small, the companies in the areas of computer-related services, professional/personal services and crafts have a good success rate. Most of those who closed their business did so because they had difficulties with finances, the business was not profitable, or markets were limited.

Did the SE program help the current situation of participants? Are they worse off than previously?

1. It is difficult to make a judgement on this question, given the many variables involved when considering the motivation and expectations of entrepreneurs.

While participants still in business are earning less than those who are not running a business, and are earning less than GE participants, the majority now in business expect to be operating this time next year. New businesses generally make little profit in the first few years and entrepreneurs view these early years as an investment in the future.

In the long term, has GESEP contributed to the EI/make work cycle? Were any controls established to prevent the misuse of the wage subsidies to qualify for EI benefits?

1.The program has not created any long-term dependence on income support. GESEP participants are no more likely to access EI than the comparison group.

See discussion of finding 10 on page 30.

2. Improved monitoring could have reduced the proportion of SARs who left early. Most of these went back on social assistance.

See discussion of finding 4 on page 39.

4.2.3 Project Design and Delivery

Were significant changes made to the operation, delivery, and/or implementation of the project as a result of the formative evaluation?

1. No changes were made to program design as a result of the formative evaluation. Key informants have different views on why this was the case.

The evaluation process was designed to provide an early reading of the program design and implementation to inform the fine-tuning of the pilot. The formative report was completed in January 1996, mid-way through the initial three-year agreement. Recommendations were made to the management committee based on this report. A management response was prepared in late 1997. None of the recommendations for changes were implemented. There were differing views among key informants on why no actions were taken. These include:

• GESEP was one of three components of the Strategic Initiative Program. The other two components took on a higher profile with program managers — one being a more innovative and controversial approach which had gained public attention, the other was not meeting success in achieving its goals. The management committee spent more time on these other components. GESEP, since it was running well, “came up in the middle”. The management response reflected this view that the changes recommended were not needed to fix a program that did not need fixing. Also, some considered timing to be an issue. The program was initially due to finish in March 1997 (meaning the final intakes would be in March 1996). This would leave limited opportunity for design changes after the formative report was completed in January 1996. As it turned out, the program continued with placements starting as late as November 1996 and with the initial funding being doubled.
• The provincial government went through downsizing and two reorganizations over the period of Strategic Initiatives. GESEP, along with other employment programs, went through assignment to two different departments, and was partially decentralized to new staff with no background in the program. Program administrators were focused on coping with change, not on program improvements.
• There did not appear to be significant momentum for innovation. The Graduate Employment component was essentially unchanged from the previous Provincial program and the Self-Employment component was not totally embraced. Since GESEP ended, the Province has continued the GE component, but not SE.

Essentially, the pilot was run as a program, during a period of much distraction among senior managers and program staff on other issues and pressures. All these factors contributed to a situation where the formative evaluation process, a key aspect of the Strategic Initiatives Program, was not successful in fine-tuning this initiative.

Key informants and focus group participants were asked for their views on the recommendations for changes made in the formative report. Both graduates and employers felt many of the recommendations for fine-tuning GESEP were appropriate and should be considered for future programs, as did some key informants.

Have satisfaction ratings changed appreciably from the time of the formative evaluation? To what extent did participants discontinue before their anticipated completion dates? What were the main reasons?

Graduate Employment

1. Satisfaction with the work experience aspect of the program has remained high for the majority of participants. Participants are less satisfied with wages paid by GESEP employers and the follow up by program administrators than they were at the time of the formative study.

See discussion of findings 1 and 2 on page 17.

2. Participants hired by employers who had repeat GESEP contracts are less satisfied than other participants. However, there is not much difference in employment outcomes.

Nineteen of the participants surveyed had been placed with employers who had repeat GESEP contracts. They are slightly more likely to be currently working than other participants. However, they are much less likely to be working for their GESEP employer and less likely to be working in a career-related area. They were also less satisfied with the value of the placement to their career plans. On all satisfaction questions rated on a 0 to 10 scale, their median rating was at least one point lower than that of other participants.

3. Employer satisfaction with the program remains high.

Employers surveyed were pleased with the fit between their needs and the graduates’ career interests (median score of 8 on a 0–10 satisfaction scale). Employers in the focus group appeared to be a committed group who saw the value of the graduates to their businesses, and who took pride in their role in developing the graduate. They saw the program as a win-win proposition. Even in the one case where the participant was not well matched to the firm, the experience contributed to the company in that they adjusted their hiring practices.

4. Thirty-two percent of all GE placements were discontinued before the end of the subsidy. Reasons were equally split between the participant choosing to leave and the financial situation of the business. Poor match was not a common reason.

Forty-six GE placements terminated early. The duration of early terminations was evenly distributed over the 52-week period of the program. As Table 9 shows, 51 percent of early terminations were initiated by the participant, the primary reason being to accept other work or leave the province. Employers terminated the placement in 38 percent of the cases because of the financial situation of the business. In these instances, more follow-up and support from the program might have helped graduates to find other placements or perhaps maintain the placements. This point was raised in the focus group. Almost all other participants who are not currently with their GESEP employer finished employment as soon as the subsidy ended.

5. Fewer SARs terminated their placement early in comparison with the GE group as a whole, an indication that the opportunity for work experience was particularly useful for these participants.

Ten of 43 SARS (23 percent) terminated their placement early, which was lower than the 32 percent overall termination rate. SARs were less likely to finish early because they found another job, and early completers tended to end up back on social assistance. SARs were no more likely to have their placements end because the employer did not find them suitable. In most cases the placement finished early because of work shortages or the financial situation of the business.

Self-Employment

6. Nineteen percent (11) of SE participants terminated the program before the end of the subsidy.³²

Some key informants thought that the program continued to support some participants who were not making an effort to get their business up and running. A number were given funding for a 10-week period to complete their business plan, but in most instances this period was extended — in a couple of cases through to the end of the program. There appeared to be a need for more rigorous performance standards against which to monitor progress and commitment, and to take action in cases of non-performance.

To what extent did GESEP participants also participate in SEOP and/or SWASP? To what extent did GESEP participants also receive funding from other related federal and provincial programs?

1. Controls were in place to prevent participation by the GESEP funded graduates in other Strategic Initiative Programs and federal programs.

The evaluation team checked National Employment Services System (NESS) and other sources and found no GESEP participants who had participated in other wage subsidy programs.

What monitoring mechanisms were put in place to collect information on participants and interventions? Was sufficient baseline information collected to support the evaluation strategy that was developed for the project?

1. Monitoring of GE participants was minimal, due to the centralized delivery structure and the limited resources devoted to program administration.

A final report was completed by the participant and the employer, but this was not used to any extent in following up on issues raised. Key informants thought that the decentralization of the new provincial GE program along with other employment programs would have resulted in increased monitoring. This is not yet the case because of the limited resources in the HRE regions devoted to the employment programs.

2. The SE component included more frequent contact with participants than that carried out in the GE component, and also facilitated networking among participants.

The facilitator’s role was useful in tracking participants’ progress. The various networking and support services provided through the facilitator were considered by key informants and participants interviewed as being useful.

3. The Strategic Initiatives Program resulted in improvements in the data collected on participants to facilitate both delivery and evaluation. With some fine-tuning, the database could serve as the basis of a useful case-management system for Provincial employment programs.

Efforts were made at the outset to collect information on participants and placements to meet both administration and evaluation needs. This resulted in improved application forms and assessment documentation. The database could have been improved to capture more of the information in these paper records. This would have helped with ongoing assessment of progress made in achieving the objectives of the pilot as well as the program evaluation. For example, a rating grid was used to assess each GE application against key criteria. If the grid had been entered in the database, this would have provided ongoing information to enable administrators to monitor placement activity compared to program objectives. The level of effort required for internal monitoring and evaluations would have been reduced, as the database would have provided easier access to information.

The Strategic Economic Plan sectors were a key innovation in the design of both components, but employers were not coded in this way in the database.³³ This necessitated re-coding for evaluation purposes.

For a number of key variables needed for the evaluation (such as institution, type of graduate, and information on the area of study) information was missing in the database. This was a case where the information was not necessarily critical to program administrators.

Some significant organizational changes occurred over the life of GESEP that no doubt impacted on data integrity. These included downsizing of departments, reorganization of departments and transfer of program responsibilities to regional offices.

The improvements that were made in the information collected could form the start of a useful electronic case management system for HRE employment programs that would also meet evaluation needs.

How and to what extent has the project succeeded in developing successful partnerships among the various levels of government, regional economic partners and small enterprises? To what extent has it succeeded in promoting the joint management of similar services?

1. The Strategic Initiatives process was one of several recent steps in the development of partnership arrangements between federal and provincial governments. These are being further developed now at the regional and local levels through the Labour Market Development Agreement and programs such as Transitional Jobs Fund.

The SE component in particular provided the opportunity to experiment with new relationships and bring new partner departments to the table, as it was creating a new program. Key informants thought that this component was a learning experience in many ways — in terms of the concept of entrepreneurship and related program design, and also in terms of the partnership process.

The GE component was not new, and this both negated the need for new partnerships and stifled the opportunity to develop new approaches where alternative approaches might have helped fine-tune the existing provincial program.

Key informants observed that the Strategic Initiatives approach brought provincial departments together that would not have done so under normal operations. This collaborative approach on crosscutting education and labour market issues (instead of a focus on individual programs) is now part of the business process in government.

2. Roles and responsibilities regarding joint management needed to be better defined for more efficient use of time and more effective contribution of each of the partners.

The partnership brought new departments and agencies to the table at the initial design stage, which created a somewhat cumbersome process and the need to sort out roles. One key informant felt the Strategic Initiatives process pointed out the need for a distinction in roles and accountabilities in such partnerships — the management committee being accountable for providing overall direction, and specific lead agencies accountable for implementation. With GESEP, this distinction in roles was not as clear as needed. Another informant thought that the links between the management and working committees should be strengthened to ensure understanding of each other’s perspectives, and that more informed decisions could be made at both levels.

Links with other economic partners and businesses were not fully developed. There is potential to build on the role of the more committed employers as partners in program design and delivery (in return for the subsidy received) that was not tapped to the extent possible. No significant linkages were made with other agencies involved in self-employment support or with industry associations in promoting the program to key sectors — promotion appears to have been focused on graduates, individual employers, and colleges.

P.J. Gardiner Institute and the Eastern Community College played the role of service providers to the program. The linkages between these agencies and the partner departments described earlier might have been strengthened if they were considered as partners in a continuum of service.

Is the pilot project model the best way of achieving project objectives? Are there better alternatives of achieving the same objective? How do results compare with the results of other programs with similar objectives?

1. Both components tended to be run more as programs than pilot initiatives, but there have been some learnings on the design of pilots in general from this initiative.

The GE component was essentially an existing Provincial program expanded with the additional funds allocated through Strategic Initiatives. These additional funds created pressures to spend rather than a rigorous testing of the program design.

The SE component did have a number of innovative features that resulted in the need to bring additional expertise to the partnership in the early design stages. As the program proceeded, it appears the partners became focused on other priorities, staff turnover occurred, and the pilot became essentially another program to be administered.

There are a number of lessons learned from this pilot process for future initiatives. These include the need for:

• appropriate levels of funding;
• clarity of roles;
• empowerment and independence of the working level staff in making decisions on individual approvals;
• performance measures to track progress;
• a mechanism to make changes in approach in a timely way;
• appropriate delivery structure to achieve the innovations being tested;
• continuity in staff resources throughout the pilot.

2. Evaluations of several national programs do not offer much insight into the relative impact of GESEP interventions because of the different target groups and labour markets.

The HRDC Job Opportunities Program, a wage subsidy program for UI eligible workers, was evaluated in 1995.³⁴ The program showed very significant positive outcomes for participants compared to GESEP, but is not that useful as a comparison given the very different target group.

HRDC carried out an assessment of the Youth Internship Canada Program in 1997.³⁵ Youth Internship Canada Program (YIC) had a similar youth target group but was not limited to post-secondary graduates. The provision of paid employment placements was similar to GESEP design. Results for YIC participants were slightly more positive in terms of the current employment status of participants. The different labour markets in which the program was implemented limits direct comparison.

The HRDC Self-Employment Assistance Program, similar to the SE component, was evaluated recently.³⁶ Over 80 percent of participants were still operating their businesses within a year of sponsorship, significantly higher than the SE results. Again, this is for a different target group and wider range of labour markets and economic conditions.

Comparisons of results with other self-employment programs in the province are not possible. There are a number of programs sponsored by several federal and provincial agencies that have not been evaluated. Some are as intensive and costly as SE in terms of the program support and the income subsidy provided. The field of entrepreneurship support is one that is still being developed. The SE approach of providing support in addition to the subsidy was considered useful. Key informants operating similar programs also feel this is an important feature of their programs, particularly for young entrepreneurs.

Several key informants suggested job search training was a missing component of GESEP that would be useful to support graduates in generating placements. The results being achieved by the job finding club in St. John’s might be an alternative service to offer graduates applying for wage subsidies who do not need as intensive a support to make the link with employers. In the most recent one-year follow-up survey carried out by job finding club administrators³⁷ 80 percent of the post-secondary graduates who participated were working in a field related to their career interests. The cost per participant is $600.

Currently a new federal-provincial youth employment strategy is being developed by HRDC and the Province. One of the key needs identified in the consultation process for the strategy is employability skills training for students before graduation. This is felt to have better potential for long term impacts than wage subsidy programs alone.

While this GESEP evaluation did not question employers on the usefulness of this type of preparation, feedback appears to support the conclusions of the youth strategy working group. Employers in the focus group stressed that the graduate had to meet the needs of their firm before they were prepared to invest in them, and even then employers felt they made a significant investment in developing the skills and workplace competencies of the graduate. One might assume that pre-graduation orientation on employability competencies such as teamwork and communications would be a useful foundation to development in the workplace.

What lessons can be learned from this project on interventions to assist the target group? How and to what extent does it contribute to the development of a policy framework for social security reform? Does the project lead to a more efficient delivery of services? To what extent can this project be successfully expanded or replicated in other areas?

Lessons Learned

GESEP was based on a number of assumptions about the challenges facing graduates and the kinds of interventions needed to address these challenges. Each of these assumptions are examined in this report. The formative evaluation report suggested a number of ways in which GESEP could be fine-tuned to improve the likelihood of program success. These included better targeting of graduates and individual employers, supports to graduates in generating placements, a focus on the overall quality of the placement instead of on career-relatedness, and monitoring of placements. Many of the findings in this current study confirm that the formative recommendations were valid and should be considered in designing future wage subsidy and self-employment interventions.

Program Delivery

The GE component did not create any efficiencies in program delivery, as it essentially used the existing delivery structure. The SE component did include innovative approaches, but these would likely be delivered more efficiently as part of existing entrepreneur support programs and services.

Transferability

Both components of GESEP, if fine-tuned as suggested in the formative evaluation report, would be transferable to other regions where there are weak labour markets for new graduates.

Social Security Reform

GESEP does not contribute significantly to social policy reform. Similar interventions already exist, although most of the lessons learned from this process confirmed what has previously been learned about these interventions for youth.

Strategic Initiatives was intended to inform policy reform through an experimental program design and evaluation process. As discussed earlier, the GESEP pilot was not developed in this manner. From discussions with key informants, it appears that the GESEP pilot experience is not that different from Strategic Initiatives elsewhere. A number did operate as pilots and were fine-tuned based on the formative evaluations. However, the program overall did not adopt an experimental design (with the use of participant and control groups) which would have been more useful in accurately assessing the relative impacts of new approaches and thus would have better informed social policy design.

There appear to have been unresolved differences in needs to be met through Strategic Initiatives. Policy makers had strategic information needs best met through a controlled experiment, while program managers had a more immediate need for the new SI funding to serve clients facing a range of labour market problems.

Footnotes

6	Participants who matched the eligibility criteria consistently gave their placement higher satisfaction ratings and contributed successful outcomes to their participation in the program.
7	These percentages are based on survey findings combined with program data.
8	Direct comparison of this as a feeder group for GESEP type programs is difficult as the education is self-reported by the applicant and could include courses under one year in length. The data does not include information on members of households receiving social assistance.
9	Post Secondary Indicators ‘98, Department of Education
10	In its Post Secondary Indicators ‘95 report, the Department of Education reported that the number of high school graduates enrolling directly in public colleges from 1989-90 to 1993-94 declined on average by 15.8 percent, while the number enrolling at university increased by 2 percent over that period.
11	Evaluation of the Self-Employment Assistance Program, HRDC, undated, page 35.
12	Characteristics and Needs of Young Entrepreneurs and Young Persons Wanting to Become Entrepreneurs in Atlantic Canada, ACOA, 1995, page 6.
13	This survey of all those who graduated from post-secondary institutions in 1995 was carried out 16 months following graduation.
14	Note that 3 participants were already operating their business when their GESEP funding began.
15	In addition to the financial support, SE participants were provided with an $800 training allowance. Many used this to go towards the purchase of a computer.
16	Most GESEP participants were active in the program within a year of graduating.
17	The remaining employers did not know or did not answer the question.
18	Four respondents gave the value of training to be over $10,000.
19	The majority of those who came to the focus group have been less successful in the labour market than GE participants as a whole. A combination of the financial incentive to go and having the time to go accounts for this difference in the mix of participants.
20	Presently, there are limited HRE staff assigned to employment programs. As a consequence there is no case management system in place and minimal monitoring of employment program placements. Some HRE regions do not place SARs in the Provincial Graduate Employment Program which followed GESEP because of the limited budget; rather they market post-secondary SAR graduates through the Support to Employment Program (STEP) program, which is exclusively for SARs.
21	See discussion of finding 11 on page 31.
22	A result that is significant at the 10 percent level is a result that has a 90 percent probability of being “real”, i.e. that the characteristic (in this case, reported value of the job skills) does have an impact on earnings. There is only a 10 percent chance that the result is due to random chance. Results significant at the 5 percent level are true 19 times out of 20, and at the 1 percent level are true 99 times out of 100.
23	Lessons Learned — Effectiveness of Employment-Related Programs for Youth, HRDC, June 1997
24	The evaluation team categorized the placements by Special Employment Program (SEP) sectors based on administrative data. Judgement had to be used in many cases, since there is no coding system for sectors according to the SEP categories. These results should be considered with this in mind.
25	This estimate should be treated with caution, due to the subjective nature of the questions. In order to meet the criteria here, the employer had to indicate that the job was a new position, that it otherwise would not have been created, and the position still exists and is either being held by the GESEP participant or someone else.
26	Wage Subsidies to Encourage the Hiring of Unemployment Insurance Claimants, Heather Robertson, Global Economics, November 1994, page 6.
27	A number of these were likely receiving UI support while in training, but this could not be identified from the administrative data.
28	We interviewed applicants who did not receive funding because of lack of funds, fair distribution of funds, or reasons that had nothing to do with the applicant’s eligibility.
29	Two other non-participants started businesses with financial assistance from another program.
30	The critical values of t for this sample size are 1.96 and 1.645 at the 5 percent and 10 percent levels of significance, respectively (for a two-tailed test). None of the t values in Table 6 exceed these values.
31	In part this may be due to respondents with low income levels not wanting to report income.
32	Source — GESEP Administrative data
33	The SEP identification was used in the database for another SI component.
34	Evaluation of the Employability Improvement Program, HRDC, December 1995.
35	An Assessment of the Youth Initiatives Program, Compas Inc., December 1997.
36	Evaluation of the Self-Employment Assistance Program, HRDC, undated.
37	Margaret Madden, Job Finding Club, St. John’s, NF.