Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

6.1 Program participants and comparison group members

The incremental effects of the Project cannot be measured by comparing the labour market performances of program participants and non-participants in the 31 Small Weeks regions. This is because some participants would have done some small weeks of work even in the absence of the Project. By definition, non-participants from the 31 Small Weeks regions could not have any small weeks of work. Therefore, comparing the performances of program participants with non-participants in the 31 Small Weeks regions would inevitably lead to biased estimates of the Project's incremental effects.

To circumvent this methodological difficulty, we have drawn the comparison group from claims filed outside of the 31 Small Weeks regions. From the administrative records of November 1998 to August 2000, we have randomly selected 260,131 claims. Among them, the individuals who filed the claims might or might not have worked any weeks for less than $150 per week. If these individuals were “clones” of program participants, then evaluating the program's incremental effects would be a simple task. All we would have to do is to calculate the differences in the labour market outcomes between the two groups of individuals. For example, if an average program participant worked three more small weeks in the Rate Calculation Period (RCP) than a comparison group member, then we would take this difference as the incremental impact of the Project on program participants. This would necessarily be the case because by definition a “clone” is identical to a program participant except that the clone did not participate in the Project. In the real world, since the assumption of “clones” is untenable, estimates from descriptive statistics can only be taken as very crude estimates. In the “first approximation” section, in addition to descriptive statistics, we have also used selected econometrically estimated equations to approximate the Project's impacts. They are technically superior to the information from descriptive statistics, but are still not our final estimates. In the last part of this report, we will use the econometric evaluation model to double-check the accuracy of these estimates. If necessary, in addition to generating more detailed results, we will use the model to re-estimate the impacts of the Project on claimants' labour market outcomes.

In the data file of the 31 Small Weeks regions, there were 236,163 claims identified with Small Weeks participation in November 1998 to August 2000. Unfortunately, some of these claims inadvertently miss certain essential information. Of the total 236,163 claims, only 162,830 (i.e., (162,830/236,163) = 68.9 percent)) have complete records.

While we may still use all 236,163 participating claims to profile participants, we may not use all of them to estimate program effects. In this report, our strategy is as follows:

• Use 236,163 participating claims in 31 Small Weeks regions, along with the about 1.6 million claims of non-participants, to profile Small Weeks participants and non-participants, and to estimate the participation rates in the 31 Small Weeks regions.
• For approximating program impacts, we restrict the sample to 162,830 participating claims¹⁸ from the 31 Small Weeks regions and 260,131 claims (comparison group members) randomly selected from the rest of the country. This latter group of claimants might or might not have some weeks of work with earnings less than $150 per week in the RCP.¹⁹

6.2 A first approximation

Table 3 presents selected key statistics on program participants and comparison group members. From the data, we know that very few claimants outside of the 31 Small Weeks regions had small weeks of work in the RCP. This may suggest that the observed small weeks worked in the 31 Small Weeks regions could be partly attributed to the existence of the Project in the regions. Just exactly what were the program effects in quantitative terms remains ambiguous, because in addition to the presence of the Project, program participants and comparison group members had different socio-economic backgrounds, and were confronted with different employment opportunities. The observed differences in small weeks worked could be partly due to the impacts of the Small Weeks Project and partly due to differences in personal characteristics. For this reason, we have to rely on the estimated multiple regression equations to put the two groups on comparable footings. These estimated equations are by-products of our econometric evaluation model, which we will use to finalize the incremental impacts of the Project. These estimated equations are documented in Appendix A of this report.

The present approach is tantamount to approximating the influences of the Project and socio-economic factors simultaneously by two multiple regression equations. The first equation hypothesizes that small weeks of work in the RCP depend upon the claimant's personal attributes (gender, age, etc.), the labour market condition of the region where he or she resides (approximated by the unemployment rate of the region), the province of residence, industrial affiliation, and whether or not the individual is a program participant. The second equation estimates the relationship of the impact of small weeks worked on the total weeks of work in the RCP. This equation is based on the rationale that a small week of work may also lead to additional weeks of work.

The estimated equations (see Appendix A) confirm that, after controlling for all other factors, the Small Weeks Pilot Project had the effect of increasing the small weeks of work of a typical program participant by about two weeks in the RCP. Moreover, for the program participant, a small week of work tended to bring in an additional 0.2 week of work. More specifically, males worked an extra 2.1 weeks in total, while females worked for about 2.4 more weeks because of the Small Weeks Pilot Project.

Figure 2 illustrates the estimated impacts of the Small Weeks Pilot Project pictorially. The estimates show that, after controlling for all other factors, the Project was largely responsible for the observed small weeks of work in the RCP for both male and females claimants (77 percent and 72 percent, respectively).²⁰ In monetary terms, the Project increased the earnings of a male and female participant by more than $300 in the RCP period.²¹ The reader should also note that there are other benefits of the Project not reported here:

• The outcome variable refers to the number of “small weeks excluded” from the EI benefits calculation formula. For about 83 percent of the claimants, this would also be the number of “small weeks worked” in the RCP. However, for 17 percent of the claimants, their large weeks (i.e., weekly earnings greater than or equal to $150) were insufficient to meet the minimum divisor requirements of their Employment Insurance (EI) regions. Some of their small weeks of work would have to be used to meet the minimum divisor requirements, and would therefore not show up in the key outcome variable of this study. Fortunately, we have the same information for program participants and comparison group members. Therefore, methodologically we are still “comparing apples with apples” and the estimated “small weeks worked” actually refer to “small weeks of work excluded” from the rate calculation formula, which is necessarily smaller than the “number of small weeks worked”.
• In addition to the benefits accrued during the RCP period, program participants received higher benefit rates and were entitled to more weeks of benefits when they became unemployed. Total benefits (earnings from additional weeks of work plus additional EI benefits) to program participants are discussed in detail in Section 6.3.4 of this report.

6.3 Incremental effects

6.3.1 The evaluation model: A non-technical summary

The results presented in the “first approximation” section are based on the assumption that what is commonly known as selection bias in the evaluation literature is an irrelevant issue for the evaluation of the Small Weeks Pilot Project. Whether or not this is indeed the case is an empirical question.

Regardless of the evaluator's belief on the existence or non-existence of selection bias in any data set, this question must be settled empirically. Without any empirical tests, critics would find the results presented too tentative because they maintain that the influence of selection bias may change the results and conclusion completely. For example, they may think that self-selection is prevalent among program participants. This could be true; if program participants were personally more motivated to accept small weeks of work than were comparison group members. Under such circumstances, selection bias exists, and the results presented in the “first approximation” section could be misleading because they had not been corrected for the influence of selection bias.

The standard econometric method that deals with the selection bias issues explicitly is the Heckman selection-bias model. In a nutshell, the model estimates the influences of the intangibles (e.g., motivation) and tangibles (personal attributes, socio-economic factors, regional economic climates, etc.) through a system of participation and outcome equations. In the case of the Small Weeks Pilot Project, the model may consist of one participation equation and two outcomes (small weeks of work and total weeks of work in the RCP) equations. The evaluator usually has to consider two possible sources of selection biases, namely administrative bias and self-selection bias. Administrative bias refers to the case in which program officers tend to grant program participation to individuals who are most likely to succeed. Since the Small Weeks Pilot Project has been available to all labour force members in the designated 31 Small Weeks regions, administrative bias is by definition a non-issue in this investigation. However, self-selection remains an outstanding issue. The model first deals with it explicitly in the participation equation and then incorporates the results from the participation equation into the outcome equations. The estimated outcomes by this method are technically and conceptually free of the confounding effects of selection bias.²²

To test the selection bias hypothesis, we have used the econometric techniques proposed by Heckman²³ and the data from 31 Small Weeks Pilot Project regions and the rest of the economy²⁴ to estimate the evaluation model of a three-equation system, namely equations for the probability of program participation, small weeks worked, and total weeks of work.²⁵

The estimated equations for the evaluation model (Equations 1 to 3) and the two O.L.S. equations (Equations 4 and 5) used in the “first approximation” section are presented in Appendix A. Comparing the estimated coefficients of the evaluation model equations (Equations 2 and 3) with those from the equations (Equations 3 and 4) used in the “first approximation” section, we can easily see that they are extremely close to each other. This immediately leads to two obvious conclusions. First, selection bias is not an issue in the evaluation of the Small Weeks Pilot Project. Second, for all intents and purposes, the use of which set of equations would not make any material difference, and would lead to virtually the same conclusion. The results presented in the “first approximation” section, therefore, remain valid. For this reason, in the remainder of this paper, we use the evaluation model to perform the remaining calculations, but will not use it to re-calculate the statistics presented earlier.

6.3.2 Small Weeks by province: The impacts of regional economic conditions

As expected, an individual's probability in program participation and his or her subsequent labour market activities (e.g., small weeks of work and total weeks of work in the RCP) depend on many factors (see the estimated equations of the evaluation model, Appendix A.) To illustrate the impacts of two of the most important factors (the Project and regional economic conditions) on provincial small weeks of work, we have used the evaluation model, along with the actual data, to perform a series of simple simulations. Specifically, we attempt to answer two questions here:

1. To what extent regional economic conditions have contributed to the province's average small weeks of work?
2. In the absence of the Small Weeks Pilot Project, would the participants in 31 Small Weeks regions still work some small weeks?

To answer Question (1), we have to create a hypothetical scenario in which the unemployment rates of all 31 Small Weeks regions were the same, and were equal to the average of the 31 Small Weeks region (i.e., 12.6 percent), while leaving all other factors (actual data) unchanged. Conceptually this would be what would have happened to small weeks of work and total weeks of work by province, if all 31 Small Weeks regions experienced the same 12.6 percent unemployment. The simulated results from this scenario, along with the actual data, allow us to see the effect of provincial economic climate on provincial small weeks of work.

To answer Question (2), in addition to the unemployment rate assumption above, we impose the assumption of “no Small Weeks Pilot Project” in the 31 Small Weeks regions to create a second hypothetical scenario. The results from this simulation show us what the provincial small weeks of work would have been, if all 31 regions faced the same economic condition of 12.6 percent of unemployment and no Small Weeks Pilot Project in place.

Figure 3 highlights the simulation results graphically.²⁶ The first column shows the actual average small weeks of work in that province. The second shows what the province's small weeks situation could have been, if the unemployment rates of the province's Small Weeks regions were equal to the average unemployment rates of the 31 Small Weeks regions in November 1998-August 2000 (12.6 percent.) In this scenario, the small weeks of work in Newfoundland and Manitoba would have been much lower than their actual numbers. This was because the unemployment rates in these two provinces' Small Weeks regions were in reality much higher than 12.6 percent. To a lesser extent, the small weeks of work of Prince Edward Island, Nova Scotia, New Brunswick, and Saskatchewan would also have been lower than the historical records for similar reasons. On the other hand, Quebec, Ontario, Alberta, and British Columbia would have had more small weeks of work than their historical records. This was due to the relative low rates of unemployment (i.e., less than 12.6 percent) in these provinces.

The second (darker) columns on Figure 3 show that by standardizing the economic climate, the variation across provinces in small weeks of work would have been less dramatic than the pattern illustrated by actual data. In particular, Manitoba, which had the second highest figure for actual small weeks worked, would have had its figure below the national average. Conceptually, the number of small weeks of work is determined by the demand for and supply of small weeks. Empirical results suggest that, in the small weeks market, the demand condition may have more or less dictated the outcome. Specifically, workers would have worked fewer small weeks, if the economy of where they lived was buoyant and large weeks were plentiful.

The difference between the second and third columns of each provincial group shows the contribution of the Small Weeks Pilot Project to a province's small weeks of work. Without any exception, the Project remains the most important factor in determining the number of small weeks of work for all 31 Small Weeks regions. The third (last) column shows what would have happened, if all the Small Weeks regions in these provinces experienced an unemployment rate of 12.6 percent and the Small Weeks Pilot Project was not implemented. Under such circumstances, claimants in Alberta would not have accepted any small weeks of work, but claimants in the remaining provinces would have worked from 0.4 to 1 week of a small week.

6.3.3 Female participants of similar personal attributes from different provinces

Historical data tells us that an average program participant from Newfoundland worked about 4.2 small weeks in the RCP, but on average a participant from Alberta accepted only 2.1 small weeks in the comparable period. Removing the effects of different economic conditions would have narrowed the gap considerably but nevertheless a gap still existed. The question is, “Can the provincial differences be attributed to tangible factors rather than to provincial cultures?” To answer this question, we have to know what comparable program participants from different provinces would have done, if their personal attributes were identical and the only difference among them was their provincial residences. Once again, we may use the evaluation model to perform this “what if” calculation. Newfoundland and Alberta have been chosen here for illustrative purpose, but the same calculation could have been carried out for any two provinces.

The “what if” calculation has been performed for one participant from Newfoundland and one from Alberta. Both of them had similar personal characteristics: female, 38 years old, regular EI benefits recipients, main labour market activities in agriculture, not a member of new or re-entrant group, not a repeat user ²⁷ of the UI/EI system, and not a recipient of Family Supplement. In other words, these two participants had a lot in common, except that they lived in two different provinces. The last qualifier is important, because the two different provinces had different unemployment rates and therefore small weeks and large weeks employment opportunities to the participants would not be the same.

Figure 4 presents the results graphically. Given the existence of the Project and the difference in economic conditions in the two provinces, a female participant of these attributes would have worked 4.8 small weeks in Newfoundland and 2.5 small weeks in Alberta. The difference remained a noticeable 2.3 weeks. In November 1998 to August 2000, the average unemployment rate in Alberta's Small Weeks regions was 10.9 percent and the rate for Newfoundland's Small Weeks regions was 19.6 percent. Different economic conditions could have accounted for about 1.3 week of the 2.3 weeks difference. The remaining difference (1 week), in the absence of other empirical evidence, may be attributed to “provincial cultural difference”.²⁸

6.3.4 Total benefits to program participants

The timeframe for the analysis up to this point refers to the 26 weeks prior to the individual's job separation, and the unit of the analysis is implicitly the “additional weeks of work” in the RCP period. Since a program participant received payments for the “additional weeks of work”, the additional employment induced by the Project is of course part of the total benefit to the participant. However, as noted earlier, the total benefit must also include the additional EI benefits received during unemployment. To avoid “mixing apples and oranges”, total benefits to participants must be expressed in monetary terms (dollars). We have only touched upon this topic earlier, because we wish to discuss it in more detail in the present section.

Conceptually the Project's total benefit to a participant may go beyond the benefits mentioned above. For example, the Project might have enhanced an individual's attachment to the labour market, kept the person up-to-date with the skill of his or her occupation, minimized the risk of obsolescence, and maintained one's work discipline. Although these benefits could be very important, they could not be estimated by the available data. In this report, we simply acknowledge their existence without arbitrarily assigning dollar values to them.

In this context, the total (incremental) benefit to a typical program participant is the sum of the individual's incremental earnings resulting from additional weeks of work and additional EI benefits because of program participation.²⁹ With the information on additional weeks of work presented earlier and data on earnings from small weeks, insured earnings, number of divisor weeks, actual benefit rate, status quo benefit rate, and weeks of benefits received, we are able to estimate all tangible benefits to program participants, except the values of additional benefit entitlements. Although we have the information to calculate the worth of the additional weeks of entitlements for all program participants, it would be erroneous to add these figures to the total benefit figures indiscriminately. This is because only a very small number of program participants exhausted their EI benefits entitlements. The majority of program participants left the EI benefits system before their entitlements terminated. For these individuals, the additional entitlements were unrealized benefits. In this section, only realized benefits (entitlements) are included in the benefit figures. The micro-accounting framework that we use to perform the calculations tracks all benefit components person by person. The final results are then tabulated separately for male and female participants.

Figure 5 graphically summarizes the salient features of the results. In addition to presenting “total benefit by gender” the graphs shows the contributions to total benefit by additional weeks of work and by additional EI benefits received.

The Project increased the income of an average female participant by an estimated $658, and of an average male participant by $820. Although these additional earnings came in as either incremental employment earnings or EI benefits over many weeks (the RCP and EI benefit periods), they were undoubtedly essential to the livelihood of many claimants from the high unemployment regions of Small Weeks. The contribution of additional weeks of work (induced by the Project) to a male participant's total benefit was $320.90, and for a female participant it was $321.40. The closeness of these two figures was not surprising. It was partly because of the $150 small week definition, and partly the result of the existing wage differential between male and female claimants. The average male participant tended to have a higher wage rate than his female counterpart, but the female participant worked slightly more small weeks. In terms of additional benefits received, an average male participant received $499.50 and the female participant's benefit increment was $336.90. Once again, we may trace this finding to the average wage differential between male and female claimants.

These results should, however, not be construed as the Small Weeks Pilot Project benefiting males more than females. As shown earlier, program participation rate for female claimants was significantly higher than the rate for male claimants. While the total benefit to an average male participant was higher than that of an average female participant by $162, there were more female claimants in the Project than male claimants (see Table 1).

Footnotes

18	Our statistical test has confirmed that claims with missing information are a random sample of the 236,163 claims. This result is not surprising: Since these are administrative records, claimants have no choice in “what to report and what not to report”.
19	The unemployment rate is different from the previous table since the sample has now been restricted to the 162,830 complete data records available for participants. The 162,830 participating claims and 260,131 claims from the rest of the country are also the data for the econometric evaluation to be discussed in Section 6.3.
20	This includes the Project's direct effect on an individual's small weeks worked in the RCP as well as the effect of the increased small weeks worked on the total number of weeks worked in the RCP.
21	See Section 6.3.4 of this report. These figures include the indirect benefits discussed above. Since the effect of a small week of work could bring in additional weeks of “small or large” weeks of work, the estimated benefits of the Project for male and female participants are slightly larger than the observed small weeks earnings in the RCP.
22	For a more detailed discussion of the Heckman evaluation model, see Heckman (1979).
23	See Heckman (1979).
24	The sample consists of 162,830 participating claims from 31 Small Weeks regions and 260,131 claims (comparison group) randomly selected from the rest of the country.
25	The term “small weeks” of work refers to the small weeks in the RCP that would be excluded for benefit calculation purposes for program participants. “Total weeks of work” denotes the sum of small weeks and regular weeks of work in the RCP.
26	It should be noted that all statistics presented here refer to the averages of the province's Small Weeks regions' data. Regions not designed as a Small Weeks region in the province are excluded.
27	In this study, a repeat user is defined as an individual who, at the time of filing a new EI claim, has had 5 weeks or more weeks of regular benefits since July 1, 1996. This is a term created for descriptive convenience. It is not HRDC's official definition of a repeat user.
28	Our data does not allow us to quantify the contribution of seasonal work to the demand for small weeks. Of course, seasonal industrial activities are unique to specific provinces. In this context, seasonal effect is indistinguishable from the effect of “provincial cultural difference.”
29	Total (incremental) benefit to a program participant = small weeks worked * wage of small weeks + 0.22 x small weeks worked x (insured earnings in the RCP/divisor weeks) + additional benefits from entitlement. The accounting identities used for calculating “incremental benefits” for female and male participants are, of course, more complex and lengthy than the stylistic formula shown here. However, this formula is sufficient to reveal the essentials of the approach.