HRDC - An Evaluation of the EI Pilot Project on Small Weeks, 1998-2001

Directorates

· Director General's Office

· Data Development

· Strategic Evaluations

· Partnership and Continuous Evaluations

Publications

· Evaluation Reports

· Evaluation Briefs

· Labour Market Development Agreement Series

· Lessons Learned Series

· Evaluation Tool Kit Series

· Evaluation Forum Newsletter

· From Theory To Practice

Appendix A: The Evaluation Model and Estimated Equations

In non-experimental program designs, selection bias is a major concern to the evaluator. Without any formal empirical tests, a critic may refuse to accept the findings from descriptive statistics and multivariate statistical methods that ignore the selection bias issue. For example, if, in the absence of the Project, program participants are more motivated to accept small weeks of work than comparison group members, then selection bias exists. Under such circumstances, estimates from econometric estimates without accounting for the influences of selection bias could be misleading, because the effects of selection bias on program outcomes have not been purged.

The standard econometric method that deals with selection bias explicitly, is the Heckman selection bias model. In a nutshell, the model estimates the influences of the intangibles (e.g., motivation) and tangibles (personal attributes, socio-economic factors, regional economic climates, etc.) on program participation and program outcomes. In a typical selection bias model, the evaluator has to consider the existence of two possible sources of selection biases, namely administrative bias and self-selection bias. Administrative bias refers to the cases in which program administrative officers tend to grant program participation to individuals who are most likely to succeed only. Since the Small Weeks Pilot Project has been available to all labour force members in the designated 31 Small Weeks regions, administrative bias is by definition a non-issue. However, self-selection remains an outstanding concern. For this investigation, the evaluation model consists of one participation equation and two outcome equations (small weeks of work and total weeks of work in the Rate Calculation Period [RCP]). The model deals with the issue directly in the participation equation. If program participants were more motivated to accept small weeks of work than comparison group members, then this intangible factor would be reflected in the estimated coefficients of the participation equation. This estimated participation equation could in turn be used to generate the necessary information for estimating the two outcome equations.³⁰ The estimated outcomes by this method are technically and conceptually free of the confounding effects of selection bias.³¹

To test the selection bias hypothesis and to see whether or not selection bias exists in the small weeks data, we have used the econometric method proposed by Heckman, along with the data from 31 Small Weeks Pilot Project regions and the rest of the economy,³² to estimate the participation equation, the small weeks worked equation, and the total weeks of work equation.³³ These estimated equations serve two purposes: (i) to test the relevance of the selection bias issue in the context of the present investigation, and (ii) to provide the evaluator with the necessary tool to shed more light on the importance of small weeks in 31 Small Weeks regions. For example, a priori, we know that a program participant's total benefits from the Project is equal to the increase in employment earnings from additional weeks of work in the RCP plus the extra EI benefits he or she may receive during unemployment. The estimated coefficients of the three-equation model, along with the actual data, would provide us with the necessary information to carry out this calculation.

The estimated equations for the evaluation model (Equations 1 to 3), along with a list of the definitions of variables used in the model, are presented at the end of this Appendix. Two O.L.S. equations (Equations 4 and 5), which are the basis for the quantitative results reported in Section 6.2 of this report, are also included here for reference. The reader may have noticed that Equations 2 and 3 and Equations 4 and 5 have the same dependent variables and explanatory variables. The only difference between the two sets is how they have been estimated.³⁴ Equations 2 and 3 have been designed to account for the potential influence of selection bias, but Equations 4 and 5 assume the non-existence of selection bias. The “no selection bias” assumption is valid if and only if the estimated coefficients of Equations 4 and 5 are not significantly different from the estimated coefficients of Equations 2 and 3. This has been statistically tested to be true. We may, therefore, conclude that selection bias is not a relevant issue for the evaluation of the Small Weeks Pilot Project.

The following is a brief summary of the salient features of the evaluation model and its estimated equations:

The sample size for estimating the evaluation model is extremely large. It consists of a sample of 422,961 claimants who were either Small Weeks program participants or comparison group members. This sample size exceeds the usual sample size requirements for producing reliable micro-econometric estimates. The estimated equations are robust to minor specification changes or minor observation range changes.³⁵ This has been confirmed many times during the estimation process.³⁶
The obvious shortcoming of our (administrative) data is that it only has a limited number of variables for personal characteristics and the socio-economic factors. For example, HRDC administrative data has no information on the educational attainment of claimants, their spouses' educational attainment and income, their children's labour market activities, etc. In this study, we have to work with what is available and attempt to get the most out of it.
The participation equation is based on the specification of a logistic model. This approach ensures that the estimated probability for a claimant's program participation is within the range of zero to 1. In this equation, the gender and age of the claimant is included as a control for the effects of these personal characteristics on program participation. The regional unemployment rate of the claimant's residence is used to capture the influence of labour market conditions on participation probability. A set of binary variables for provinces is included to control for the effects of “provincial culture”. A set of industrial binary variables is used to control for systematic differences in labour market conditions across industries. All estimated coefficients except two are statistically significant at, at least, the 5 percent level and have the expected signs. For example, a male claimant has lower probability to be a program participant than a female claimant. The older the individual, the less likely he or she would become a participant. On the other hand, a region of relatively high unemployment rate tends to induce more claimants to accept small weeks of work in the RCP. This suggests that the demand for small weeks is largely determined by the buoyancy (or the lack of it) of the economy. When regular jobs are plentiful, most workers would prefer regular-hours jobs to small weeks work. The set of binary (0 and 1) provincial variables shed some light on provincial influence on program participation. British Columbia is the “reference province” in the equation. Thus a positive and statistical coefficient for a province means that a claimant from this particular province has a higher probability for program participation than a claimant from British Columbia. The pattern is quite clear: Program participation concentrates heavily in the Maritimes provinces; Quebec and Ontario are next; claimants from the Prairies and B.C. have the lowest probability for engaging in small weeks activities.

The participation equation also demonstrates that an individual's industrial affiliation has some influence on an individual's decision on accepting small weeks of work. Claimants from fishing, forestry, transportation-storage-and communication, trade, and business and other miscellaneous services have a higher probability to engage in small weeks of work than claimants from public administration sector, agriculture, mining, manufacturing, and finance-insurance-and real estate. Farmers and miners are mostly seasonal workers; one may think that they would probably like some small weeks of work during the off-seasons. Their relatively low probability in program participation is likely determined by the lack of small weeks work available in these sectors. Occupations in manufacturing and finance-insurance-and real estate are mostly regular jobs. Individuals from these sectors are unlikely small weeks participants, as suggested by the estimated coefficients.

The second equation of the model refers to the “small weeks of work” of the claimant.³⁷ The equation specifies that “small weeks of work” depends upon a vector of personal attributes, socio-economic factors, Employment Insurance (EI) usage (i.e., whether or not the person is a member of new entrants or re-entrants, a repeat user,³⁸ and or a recipient of Family Supplement), and finally whether or not the person is a program participant. Variables for EI usage appear in this equation but not in the participation equation. This is based on the rationale that EI rules directly influence a claimant's number of small weeks worked but not necessarily the individual's participation decision. For example, a claimant might be willing to work one small week only, if he or she accepted more than one small week in the RCP, the claimant's total family income would have exceeded the maximum allowed by the Family Supplement (FS) rule and would have lost the extra income from the FS. As noted earlier, to circumvent the potential influence of selection bias, we have used the Heckman estimator to carry out the estimation.³⁹ The estimated coefficient for the participation variable is 1.97. This means that, after controlling for the influences of all other factors, the Project increases a typical program participant's small weeks of work by 1.97 weeks. Similar to the participation equation, with very few exceptions, the estimated coefficients for this equation have the expected signs and are statistically highly significant. Their interpretation is straightforward, and will not be elaborated here.
The last equation models “total weeks of work” equation. As contended earlier, for program participants, a small week of work could lead to additional weeks of work with the same firm in the RCP. This equation is designed to capture this “indirect effect” of the Project. In the specification, this indirect effect is captured by the interaction term of “small weeks worked * program participation”. Once again, to purge any possible influence of selection bias, we have used the Heckman estimator to estimate the coefficients of this equation. The estimated coefficient for this variable is 0.22, which is extremely close to its counterpart of 0.23 estimated by the O.L.S. used in the “first approximation” section. This result, along with the evidence from the second equation of the model, confirms that selection bias is a non-issue for this investigation.

Evaluation model: Estimated equations

Equation #1 (model component): Probability of participation*
Dependent Variable (PARTICIPATION): With weeks of work earning less than $150 per week in the RC=1, otherwise=0 Sample: 422,961 cases Cases included in analysis: 417,017 Method: Maximum likelihood — binary logit
Variable**	Coefficient	Standard Error	Z-Statistic	Significance
CONSTANT	-17.569	0.096	-182.052	0.000
MALE	-0.435	0.015	-27.570	0.000
AGE	-0.007	0.001	-10.583	0.000
URATE	2.037	0.010	202.554	0.000
NFL	2.778	0.140	19.839	0.000
PEI	1.650	0.616	2.676	0.007
NS	1.784	0.034	52.491	0.000
NB	7.710	0.104	74.161	0.000
QUE	1.658	0.020	83.314	0.000
ONT	2.252	0.025	89.746	0.000
MAN	-3.089	0.636	-4.860	0.000
SASK	-3.089	0.241	-12.811	0.000
ALB	-3.431	0.120	-28.679	0.000
TERRITORIES	0.176	1.145	0.154	0.878
AGRI	-0.195	0.057	-3.424	0.001
FISHING	1.881	0.191	9.837	0.000
FORESTRY	0.302	0.095	3.167	0.002
MINING	-1.129	0.131	-8.633	0.000
MANUF	-0.307	0.042	-7.255	0.000
CONS	-0.582	0.047	-12.478	0.000
TRS_ST_COM	0.290	0.048	5.983	0.000
TRADE	0.272	0.043	6.397	0.000
FIN_INS_RE	-0.423	0.056	-7.558	0.000
BUS_SER	0.088	0.046	1.889	0.059
ED_HEALTH	0.010	0.044	0.223	0.823
OTH_SER	0.348	0.043	8.160	0.000
Probability (LR stat) = 0.000; McFadden R-squared = 0.765 * For the definitions of all variables, see the mnemonic list at the end of this Appendix. ** “FEMALE”, “BC”, and “GOVERNMENT” are the binary-reference variables for gender, province, and industrial classification in estimation. They are not included on the mnemonic list.

Equation #2 (model component): Small Weeks worked*
Dependent Variable (EXC_SW): Weeks of work of earnings less than $150 per week Sample: 422,961 cases Cases included in analysis: 417,017 Method: Heckman´s instrumental variable estimator
Variable**	Coefficient	Standard Error	Z-Statistic	Significance
CONSTANT	-1.154	0.021	-55.722	0.000
MALE	-0.288	0.006	-45.782	0.000
AGE	-0.000	0.000	0.063	0.950
URATE	0.145	0.002	96.582	0.000
NEW_REENT	0.165	0.019	8.535	0.000
REPEAT	0.236	0.006	37.977	0.000
FS	-0.153	0.010	-15.688	0.000
NFL	0.420	0.019	21.659	0.000
PEI	0.605	0.027	22.696	0.000
NS	0.469	0.015	31.670	0.000
NB	0.615	0.017	36.416	0.000
QUE	0.285	0.011	25.041	0.000
ONT	0.247	0.009	26.267	0.000
MAN	0.303	0.015	19.659	0.000
SASK	0.194	0.0171	11.353	0.000
ALB	0.214	0.011	18.893	0.000
TERRITORIES	-1.722	0.098	-17.652	0.000
REG_CLAIM	0.107	0.008	13.680	0.000
AGRI	-0.124	0.023	-5.412	0.000
FISHING	-0.985	0.033	-29.655	0.000
FORESTRY	-0.417	0.034	-12.219	0.000
MINING	-0.092	0.026	-3.571	0.000
MANUF	-0.156	0.014	-10.840	0.000
CONS	-0.058	0.016	-3.677	0.000
TRS_ST_COM	-0.001	0.017	-0.034	0.973
TRADE	0.304	0.015	20.552	0.000
FIN_INS_RE	0.103	0.019	5.298	0.000
BUS_SER	0.064	0.017	3.837	0.000
ED_HEALTH	-0.129	0.015	-8.630	0.000
OTH_SER	0.209	0.015	14.105	0.000
PARTICIPATION	1.971	0.015	131.141	0.000
Probability (F-statistic) = 0.000; adjusted R-squared = 0.435; Durbin-Watson = 1.995 * For the definitions of all variables, see the mnemonic list at the end of this Appendix. ** “FEMALE”, “BC”, and “GOVT” are the binary-reference variables for gender, province, and industrial classification in estimation. They are not included on the mnemonic list.

Equation #3 (model component): Total weeks of work*
Dependent Variable (WKS_STAT): Total weeks of work in the RCP Sample: 422,961 cases Cases included in analysis: 422,961 Method: Heckman´s instrumental variable estimator
Variable**	Coefficient	Standard Error	Z-Statistic	Significance
CONSTANT	26.764	0.025	1051.994	0.000
MALE	-0.431	0.008	-53.433	0.000
AGE	-0.001	0.000	2.112	0.034
URATE	-0.216	0.003	-77.440	0.000
NFL	-0.557	0.028	-20.147	0.000
PEI	-1.165	0.039	-29.891	0.000
NS	-0.056	0.022	-2.606	0.009
NB	-1.177	0.025	-46.679	0.000
QUE	-0.136	0.016	-8.515	0.000
ONT	-0.061	0.013	-4.541	0.000
MAN	-0.122	0.022	-5.533	0.000
SASK	-0.171	0.024	-6.985	0.000
ALB	-0.281	0.016	-17.699	0.000
TERRITORIES	1.567	0.141	11.110	0.000
EXC_SW*
PARTICIPATION	0.219	0.008	28.550	0.000
Probability (F-statistic) = 0.000; adjusted R-squared = 0.085; Durbin-Watson = 1.999 * For the definitions of all variables, see the mnemonic list at the end of this Appendix. ** “FEMALE”, “BC”, and “GOVT” are the binary-reference variables for gender, province, and industrial classification in estimation. They are not included on the mnemonic list.

Equation #4 (supplementary equation): Small Weeks worked*
Dependent Variable (EXC_SW): Weeks of work of earnings less than $150 per week in the RCP Sample: 422,961 cases Cases included in analysis: 417,017 Method: Ordinary least squares
Variable**	Coefficient	Standard Error	Z-Statistic	Significance
CONSTANT	-1.152	0.020	-57.865	0.000
MALE	-0.283	0.006	-46.492	0.000
AGE	-0.000	0.000	1.784	0.075
URATE	0.146	0.001	115.746	0.000
NEW_REENT	0.143	0.019	7.601	0.000
REPEAT	0.192	0.006	31.870	0.000
FS	-0.154	0.009	-16.272	0.000
NFL	0.425	0.019	22.567	0.000
PEI	0.615	0.026	24.044	0.000
NS	0.475	0.014	33.426	0.000
NB	0.625	0.016	39.677	0.000
QUE	0.292	0.010	28.169	0.000
ONT	0.246	0.009	27.024	0.000
MAN	0.305	0.015	20.464	0.000
SASK	0.195	0.017	11.818	0.000
ALB	0.214	0.011	19.623	0.000
TERRITORIES	-1.736	0.095	-18.356	0.000
REG_CLAIM	0.095	0.008	12.439	0.000
AGRI	-0.116	0.022	-5.246	0.000
FISHING	-0.980	0.032	-30.417	0.000
FORESTRY	-0.409	0.033	-12.330	0.000
MINING	-0.087	0.025	-3.465	0.000
MANUF	-0.155	0.014	-11.140	0.000
CONS	-0.047	0.015	-3.083	0.002
TRS_ST_COM	0.002	0.017	0.129	0.898
TRADE	0.299	0.014	20.842	0.000
FIN_INS_RE	0.097	0.019	5.111	0.000
BUS_SER	0.063	0.016	3.903	0.000
ED_HEALTH	-0.127	0.015	-8.710	0.000
OTH_SER	0.208	0.014	14.463	0.000
PARTICIPATION	1.975	0.009	210.652	0.000
Probability (F-statistic) = 0.000; adjusted R-squared = 0.468; Durbin-Watson = 1.999 * For the definitions of all variables, see the mnemonic list at the end of this Appendix. ** “FEMALE”, “BC”, and “GOVT” are the binary-reference variables for gender, province, and industrial classification in estimation. They are not included on the mnemonic list.

Equation #5 (supplementary equation) : Total weeks of work*
Dependent Variable (WKS_STAT): Total weeks of work in the RCP Sample: 422,961 cases Cases included in analysis: 422,961 Method: Ordinary least squares
Variable**	Coefficient	Standard Error	Z-Statistic	Significance
CONSTANT	26.737	0.022	1242.911	0.000
MALE	-0.395	0.008	-50.346	0.000
AGE	-0.001	0.000	2.509	0.012
URATE	-0.218	0.002	-126.932	0.000
NFL	-0.557	0.027	-20.471	0.000
PEI	-1.171	0.037	-31.459	0.000
NS	-0.095	0.021	-4.593	0.009
NB	-1.184	0.022	-52.889	0.000
QUE	-0.172	0.014	-11.921	0.000
ONT	-0.055	0.013	-4.118	0.000
MAN	-0.095	0.022	-4.366	0.000
SASK	-0.143	0.024	-5.920	0.000
ALB	-0.254	0.016	-16.278	0.000
TERRITORIES	1.646	0.138	11.903	0.000
EXC_SW*
PARTICIPATION	0.234	0.002	109.417	0.000
Probability (F-statistic) = 0.000; adjusted R-squared = 0.108; Durbin-Watson = 1.998 * For the definitions of all variables, see the mnemonic list at the end of this Appendix. ** “FEMALE”, “BC”, and “GOVT” are the binary-reference variables for gender, province, and industrial classification in estimation. They are not included on the mnemonic list.

Mnemonic list: Definitions of variables

AGE: Age of the claimant at benefit period commencement.

AGRI: Binary variable equal to “1”, if the claimant's industrial affiliation is agriculture, otherwise equal to “0”.

ALB: Binary variable equal to “1”, if the claimant's most recent residence is Alberta, otherwise equal to “0”.

BUS_SER: Binary variable; equal to “1”, if the claimant's industrial affiliation is “business services”, otherwise equal to “0”.

CONS: Binary variable; equal to “1”, if the claimant's industrial affiliation is “construction”, otherwise equal to “0”.

ED_HEALTH: Binary variable; equal to “1”, if the claimant's industrial affiliation is “education, health, and related services”, otherwise equal to “0”.

EXC_SW: Number of small weeks (weekly earnings less than $150 per week) of work in the RCP.

FIN_INS_RE: Binary variable; equal to “1”, if the claimant's industrial affiliation is “finance, insurance and real estate”, otherwise equal to “0”.

FISHING: Binary variable; equal to “1”, if the claimant's industrial affiliation is “fishing and trappings”, otherwise equal to “0”.

FORESTRY: Binary variable; equal to “1”, if the claimant's industrial affiliation is “forestry”, otherwise equal to “0”.

FS: Binary variable; equal to “1”, if the claimant is a recipient of EI Family Supplement, otherwise equal to “0”.

MALE: Binary variable; equal to “1”, if the claimant's gender is male, otherwise equal to “0”.

MAN: Binary variable equal to “1”, if the claimant's most recent residence is Manitoba, otherwise equal to “0”.

MANUF: Binary variable; equal to “1”, if the claimant's industrial affiliation is “manufacturing”, otherwise equal to “0”.

MINING: Binary variable; equal to “1”, if the claimant's industrial affiliation is “mines, quarries, and oil wells”, otherwise equal to “0”.

NB: Binary variable equal to “1”, if the claimant's most recent residence is New Brunswick, otherwise equal to “0”.

NEW_REENT: Binary variable; equal to “1”, if the claimant is a member of EI's “new entrant and re-entrant” group, otherwise equal to “0”.

NFL: Binary variable equal to “1”, if the claimant's most recent residence is Newfoundland, otherwise equal to “0”.

NS: Binary variable equal to “1”, if the claimant's most recent residence is Nova Scotia, otherwise equal to “0”.

ONT: Binary variable equal to “1”, if the claimant's most recent residence is Ontario, otherwise equal to “0”.

OTH_SER: Binary variable; equal to “1”, if the claimant's industrial affiliation is “services other than “public administration and defense” and the service industries listed”, otherwise equal to “0”.

PARTICIPATION: Binary variable; equal to “1”, if the claimant has some small weeks (weekly earnings less than $150 per week) of work in the RCP, otherwise equal to “0”.

PEI: Binary variable equal to “1”, if the claimant's most recent residence is Prince Edward Island, otherwise equal to “0”.

QUE: Binary variable equal to “1”, if the claimant's most recent residence is Quebec, otherwise equal to “0”.

REG_CLAIM: Binary variable; equal to “1”, if the person is a “regular EI benefits” claimant, otherwise equal to “0”.

REPEAT: Binary variable; equal to “1”, if the claimant at the time of filing a new EI claim has had 5 weeks or more weeks of “regular EI benefits” since July 1, 1996, otherwise equal to “0”.

SASK: Binary variable equal to “1”, if the claimant's most recent residence is Saskatchewan, otherwise equal to “0”.

TERRITORIES: Binary variable equal to “1”, if the claimant's most recent residence is Northwest Territories or Yukon, otherwise equal to “0”.

TRADE: Binary variable; equal to “1”, if the claimant's industrial affiliation is “wholesale or retail trade”, otherwise equal to “0”.

TRS_ST_COM: Binary variable; equal to “1”, if the claimant's industrial affiliation is “transportation, storage, communication, and other utilities”, otherwise equal to “0”.

URATE: Regional unemployment rate of where the claimant resides.

Footnotes

³⁰	There are two versions of the Heckman estimator. The first one uses the participation equation to calculate the Inverse Mills Ratio, and then include the Ratio as one of the explanatory variables in the outcome equations. The second version is essentially an instrumental variable method. It uses the participation equation to calculate the probability of program participation, and then use this calculated series as an instrumental variable to replace the participation variable (explanatory variable) in the outcome equations. The instrumental variable approach is used in our estimation.
³¹	For a more detailed discussion of the Heckman evaluation model, see Heckman (1979).
³²	The sample consists of 162,830 participating claims from 31 Small Weeks regions and 260,131 claims (comparison group) randomly selected from the rest of the country.
³³	The term “small weeks” of work refers to the small weeks in the RCP that would be excluded for benefit calculation purposes for program participants. “Total weeks of work” denotes the sum of small weeks and regular weeks of work in the RCP.
³⁴	In particular, Equations 2 and 3 have been estimated with the Heckman estimator (instrumental variable), whereas Equations 4 and 5 have been estimated with the O.L.S. method.
³⁵	For example, if we randomly drop 10 percent of the claimants from the sample and re-estimate the model, the estimated equation would have remained more or less the same as the equations from the full sample.
³⁶	The sample size and the actual number of cases included in estimating each equation vary slightly. This is because a few cases miss information for all explanatory variables. The computer program therefore automatically excludes these few cases from the sample during estimation. Since the three estimated equations of the model do not use the same list of explanatory variables, the actual number of cases included in estimating the equations vary slightly.
³⁷	A small week is defined as a week with earnings less than $150. For a more detailed explanation of this dependent variable, see the footnotes of Table 3.
³⁸	See footnote #27.
³⁹	See footnote #30.

Last Modified: 2001-09-21	Important Notices