Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

This chapter presents evidence on short-term impacts of TJF, based on job creation results from the survey of project sponsors. Among the issues addressed are the number of jobs created, the type of jobs created, the sustainability of the jobs, the issue of incrementality, and finally, the issue of cost. As indicated earlier, sponsors of 301 projects were interviewed in the survey. The fact that only one-third of surveyed projects are completed means that the results presented in this chapter underestimate final job creation impacts of these projects. Moreover, larger projects are more likely to be in progress, implying that projects with larger job creation potential are under-represented in the sample of completed projects. Where appropriate, outcome information is provided for projects overall and for completed projects only.

4.1 Job Creation

Exhibit 4.1 presents figures on the mean number of new and saved jobs by the TJF projects in the sample, according to type of job based on the survey responses. Three observations may be made about the results. First, in TJF projects, more new jobs are created than saved by averting layoffs. One reason is the fact that about three-quarters of respondents indicated that their projects created only new jobs, while the remainder indicated that their projects both created new jobs and averted layoffs (saved jobs). Job creation varied little by industry of the project and by the share of total funds contributed by the TJF. However, there is a good deal of regional variation in job creation. In Atlantic Canada, for example, the size of projects was relatively small, though the number of projects approved was greater.

Second, Exhibit 4.1 indicates that the majority of jobs created under the TJF are permanent, full-time jobs. Dividing the mean total number of new jobs created (about 20) by the mean number of new permanent full-time jobs (16) yields an estimate of 80 percent that are permanent full-time positions.² Another measure of the permanent, full-time job rate is the (unweighted) mean of the rates for each individual project, which yields a rate of 79 percent across projects. (Including part-time jobs, this rate rises to 95 percent.) This percentage tends to rise with organization size and to be lower in smaller communities (72 percent), public sector organizations (54 percent) and the tourism/hospitality sector (56 percent). The latter finding is not surprising given the seasonal nature of jobs in this sector.

Data were also collected in the survey on seasonal and short-term jobs. These data indicate that 13 percent of all permanent full-time jobs were seasonal. The proportion that was seasonal was particularly high in tourism and primary industry projects, in smaller communities, and in Atlantic Canada. As for short-term jobs, the survey results indicate that the mean number of short-term jobs (which tended to be insignificant, as Exhibit 4.1 indicates) tended to be relatively high in public sector projects and projects in the tourism/hospitality and primary sectors.

Finally, it should be noted that the completed projects (bottom bar of each set) have, to date, generated a larger number of jobs than projects
overall³. This is not unexpected, given that for a large number of uncompleted projects there is still job creation to take place. The results also indicate that, with 25 total positions and 20 permanent, full-time positions created by completed projects, the permanent full-time job creation rate (about 80 percent) is similar to the rate for all projects. This is confirmed by the (simple) mean of the individual projects rates, computed to be at 78 percent (not shown).

The above figures include part-time and seasonal jobs. The conversion from the mean number of jobs created to their full-time equivalent (FTE) is based on the mean number of permanent full-time jobs created for completed projects (20), adding half the number of part-time jobs and half the number of seasonal jobs, to arrive at about 21 FTE jobs per completed project. This yields a FTE job rate of 84 percent.

The percentage of jobs created by occupational group is presented in Exhibit 4.2. Note that, as respondents were permitted multiple responses to this question, percentages add up to more than 100 percent. The results indicate that semi-skilled jobs are the predominant jobs created by TJF projects, with professional jobs being the least common job created. Not shown is the fact that semi-skilled jobs are most likely to be found in large organizations (80 percent) and in the manufacturing sector (85 percent) and least likely to be found in projects in Ontario (45 percent). On the other hand, higher skilled occupations are also most likely to be found in projects in larger organizations (20 percent) and in British Columbia (20 percent). Sales, service and clerical jobs are, as expected, generally found in hospitality and tourism sector projects (54 percent). The occupational mix of jobs is reflected in the mean wages of jobs created (not shown), as only 10 percent of TJF jobs earn more than $16 an hour. The overall mean is $13 an hour.

How do jobs created in TJF projects compare to targets set out in the original signed agreements? Survey results indicate that about one-half (47 percent) of projects matched their targets, with the rest fairly evenly split between those which exceeded targets and those which failed to reach targets. Other results indicate that it was the smaller organizations that tended to match targets (59 percent), while larger organizations tended to exceed them (31 percent).

As for the reasons why divergence occurred, Exhibit 4.3 indicates that the most common reasons, by far, for exceeding targets were higher revenues than expected and initial underestimation of job creation. As for shortfalls, the second panel of Exhibit 4.3 indicates that non-completion is the main reason why targets were not attained, which is not surprising given that about two-thirds of projects have yet to be completed. Other common reasons for shortfalls were higher costs and lower sales than expected.

4.2 Sustainability

One of the goals of the TJF was to create not just jobs, but jobs that would last. Obviously, given that the TJF has been implemented for under two years, it is too soon to tell if the jobs created are in fact durable, but, as a partial solution, project sponsors were asked in the survey if they expected jobs to be in place 18 months after completion of the project. Focusing on completed projects, whose sponsors are assumed to a have a relatively clear idea of the sustainability of the jobs created, responses to the preceding question indicate that most sponsors believed that the majority of the jobs created would be sustained. On average, sponsors of completed projects expected that 84 percent of jobs created would be sustained, a rate which rises with organization size and is particularly low for tourism projects (68 percent) and for projects in the Prairies (70 percent).

The above figures are based on the simple mean, calculated by dividing the number of permanent jobs created (new and saved) by the number of jobs expected to be sustained for each completed project (n=103), and then computing the mean across the projects. Another approach is to divide the total number of jobs created by the total number of permanent jobs expected to exist 18 months from now, for completed projects that have non-missing data for both variables (i.e., for the same group of projects, n=110). Under this approach, which in effect weights for the number of jobs created, we obtain a similar sustainability rate of 86 percent.

Note that the sustainability results must be treated with caution. While some measurement error has been eliminated by focusing on completed projects, the estimates are still based on mere expectations, not real experiences. Moreover, the length of time since project completion was typically short when the survey took place. It remains to be seen, in subsequent evaluations, if in fact these jobs can be sustained. It should also be noted that the other side of the coin, the possibility that firms could create more jobs than expected, was an issue that was not addressed in the survey.

Survey respondents were also asked if other funding would be necessary to sustain the jobs created. Returning to the full sample (n=301), about one-quarter of project sponsors said that other funding would be necessary to sustain jobs, with another 17 percent saying they were unsure. Affirmative responses to this question appeared to be inversely related to organization size and to the share of total project funding contributed by TJF.

Among respondents saying additional funding would be necessary, the predominant reported reasons were higher than expected costs (40 percent) and the need to finance expansion (33 percent). The majority of these respondents expected that additional financing would come from private sector sources or from the sponsors themselves (59 percent).

4.3 Incrementality

The goal of the TJF, like any job creation program, is to create incremental jobs, i.e., jobs that would not have been created in the absence of the program. Survey results provided three measures of TJF incrementality. First, almost 60 percent of project sponsors reported that the TJF influenced their decision to go ahead with the project to a great extent, while another 28 percent said TJF had some influence on their decision.

Another measure of incrementality is capacity to leverage additional funds. Survey results indicate that less than one-half said the TJF enabled them to leverage additional funds and one-third said it did not. A third measure is provided by a more direct question of whether or not the project would have gone ahead without TJF funds. Responses to this question indicate that 53 percent said their project would have not gone ahead, meaning that over one-half of TJF projects were incremental.

Incrementality was relatively higher in smaller organizations (63 percent), smaller communities (64 percent), public sector sponsors (79 percent), and tourism projects (78 percent). Incrementality was particularly low (38 percent) in projects with a low share of TJF funding (less than 25 percent).

Beyond the issue of full incrementality is the extent to which the TJF altered projects. Exhibit 4.4 contains the results from a question on how the TJF affected projects, addressed to the 42 percent of sponsors whose projects would have gone ahead without the TJF. The results indicate that a large proportion reported that TJF funds accelerated the timing of the project (66 percent) or enabled the creation of more jobs than would otherwise have been created (51 percent). In other words, these projects would have gone ahead without the TJF but would have been implemented later if TJF funding had not been obtained or would not have created as many jobs as they did with TJF funding. Only 7 percent of the sponsors of projects that would have gone ahead without TJF indicated that none of the listed effects applied, implying complete non-incrementality for these projects.

How many jobs created by TJF projects could be considered to be incremental? Responses to the incrementality question indicated that about one-half of the projects were truly incremental, but that about one-half of non-incremental projects were in fact incremental because TJF funding enabled more job creation than would otherwise have been produced. In precise terms, about 74 percent of TJF jobs were incremental. This figure is obtained by adding the 53 percent of all projects which would not have gone ahead without TJF funding to 51 percent of the 42 percent of projects which would have gone ahead regardless, but which were expanded due to TJF funding.

4.4 Cost of Job Creation

Turning to the issue of costs, we first estimate the TJF cost per-job based on completed projects (where the estimate of jobs created is more accurate). Survey results for completed projects indicate that TJF's cost per-job created amounted to about $7,500. This result is based on the mean reported TJF contribution of all completed projects surveyed ($187,600 per project) and the mean number of jobs created across completed projects in the sample (24.8 jobs per project). This figure, in effect, considers the cost of each job created under TJF, calculating the return on investment for program expenditures.

It is important to point out, however, that these estimates are biased downward. The reason is that the number of jobs (the denominator of the calculation) includes short-term, part-time, unsustainable jobs, and non-incremental jobs, thus inflating the estimated number of jobs created and under-estimating the cost per-job. To correct for this bias, first, we divide the mean TJF cost per project by the mean FTE jobs created per project based on the FTE job rate (84 percent) as described above. This yields a cost per FTE job of about $8,900 ($187,600 / (24.8 x 0.84)). Second, we adjust for incrementality by applying the mean incrementality rate (74 percent, described above) to the mean number of FTE jobs created per project. This computation gives us an estimated TJF cost per-job of $12,100.

Exhibit 4.5 summarizes the TJF cost per-job estimates. Adjustment for the fact that not all jobs will likely be sustainable could not be made because, as indicated, project sponsors were asked only about how many jobs were additional and not about how many sustainable jobs might be created.

Mean Jobs Created Per Project1	TJF Cost per-job1 ($)
Total jobs	25	7,500
Full-time equivalent (FTE) jobs	21	8,900
Incremental FTE jobs	15.54	12,100
1 Based on survey results for completed projects, (n=98). Source: TJF Survey and HRDC administrative data.

In order to examine how the TJF cost per-job varies across different types of projects, we calculated an average TJF per-job cost for each project included in our sample. Averaging the cost per-job across all completed projects yields a figure of $13,400. (The estimated total cost per-job is $64,300.) This is essentially an unweighted figure that considers the average cost per-job of each project equally, regardless of the level of expenditures or number of jobs created. The distribution of this figure, however, is useful to provide a picture of trends across different types of projects.

Exhibit 4.6 breaks down TJF projects according to the average amount contributed by HRDC. The figures therein indicate that the majority of projects (62 percent) received less than $10,000 in funding per-job from TJF. Other results indicate that the TJF cost per-job was much higher in completed projects sponsored by larger organizations ($17,335) and public organizations ($22,638), and in projects that were new operations ($18,317) as opposed to those that were expanding operations ($9,204). Also, the mean cost per-job appears to increase with the total cost of the project, so that for projects costing less than $250,000 the per-job cost is $7,478, rising to $20,659 for those costing in excess of $1 million.

4.5 Estimation of TJF Total Creation by March 1999

To estimate the total number of jobs created by TJF, one strategy would be to multiply the mean number of jobs created per completed project by the total number of projects. Since the completion of the initial phase of the evaluation, however, there has been a substantial increase in the number of approved projects and, potentially, differences in the types of projects approved. To generate a more rigorous estimate of total job creation, a predictive model the jobs created as reported in the survey was developed for the completed projects. This model was then applied to the full administrative database to extrapolate the number of jobs created.

The first step in this analysis was to test a simple multivariate model using the survey cases for which complete data were available. Administrative data on the estimated job creation, the amount of TJF funding, total project cost and region were included in the model to predict the number of jobs created reported by the survey respondents. Based on the modelling results, the most significant predictors of the survey-reported job creation were the estimated number of jobs created (from the administrative data) and the total project cost.

This model, using the administrative data for the estimated jobs created and the total project funding, was then applied to 826 approved projects recorded on the program administrative database (the number of projects where there was complete data for all variables used in the model). These projects represent about $230 million in TJF program expenditures. Note that the model was based on a small number of projects and that this model was then applied to projects that were outside the range of cases for which the model was originally tested.

Assuming that the remaining project funds ($30 million) will be expended on projects by March 1999 similar in nature to those already approved, predicted actual job creation for the TJF program overall ($260 million) would be 29,500 jobs. Adjusting for incrementality, the number of jobs would be about 22,000. A further adjustment for full-time equivalency would yield 18,500 full-time incremental jobs.