Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

This section of the report presents the methodology for the Formative Evaluation of the Provincial Benefits and Provincial Measures under the Canada/Manitoba LMDA. For a more detailed description of the methodology, see the Technical Report under separate cover.

The final design of the evaluation consisted of 28 key informant interviews, 12 focus groups, a survey of 1,400 program participants (1,100 PBPM participants and 150 participants in each of the PBT and POT programs)¹⁶ and a survey of 500 non-participants.

The following is a description of the methodology used for each component of the data collection process.

2.1 Key Informant Interviews

A series of 28 key informant interviews was conducted for the formative evaluation. The list of potential key informants was prepared by the Joint Evaluation and Accountability Committee (JEAC). The interview respondents included a variety of program managers and policy makers who collectively were in a position to comment on the design, implementation and delivery of the PBPMs that were assessed in this evaluation.

The key informants included the following:

• four (4) federal officials with Human Resources Development Canada (HRDC), including a representative of the Canada Employment and Immigration Union (CEIU);
• six (6) provincial officials with Manitoba Education and Training, including one official from the Apprenticeship Branch;
• eleven (11) key community partners; and
• seven (7) third-party delivery agents.

2.2 Focus Groups

A total of 12 focus group discussions were conducted. Focus groups had an average of eight to 10 participants each and were conducted in each of the three service regions of the province: Winnipeg; Southern Manitoba (Brandon); and Northern Manitoba (Thompson). There were five types of participants: clients; staff; third-party delivery agents; employers; and community partners (including employee groups, employer groups, sectoral associations and others). The distribution of the 12 focus groups is presented in Table 2.1.

For recruiting clients, program participants were selected from the sample for the survey of participants (representing a variety of PBPMs). For employers, community partners and delivery agents, the focus group recruitment relied on lists provided by the JEAC.

2.3 Survey of Participants and Non-Participants

Survey of Participants: Population

The final participant data set included only those participants who started a PBPM intervention on or after November 27, 1997 and completed/terminated before August 15, 1999. The length of time since completing or terminating the intervention was a key variable for stratifying the sample in the final analyses as the length of time since completion is related to employability outcomes.

Survey of Participants: Data Sampling

The data files were originally developed to include participants who participated in LMDA employment programs and services at any time between November 27, 1997 and August 15, 1999. As mentioned, these files were aggregated, yielding a single data file containing information for 19,252 participant cases, with the individual client as the unit of analysis. This was not equal to the sum of all the cases from the administration and data files because clients that had taken part in more than one intervention could appear in more than one file. Following the removal of all cases without valid phone numbers, start and end dates for EI benefits, and start and end dates for most recent interventions, the final data file consisted of 14,784 individuals.

Three primary sampling variables were taken into account when developing the survey sample frame. These variables were program type (EAS¹⁷, Wage Subsidies, Self-Employment, Employment Partnerships, SLG/Enhanced Fee Payer, Purchase of Training, Apprenticeship and Project-Based Training), region (Winnipeg, Southern Manitoba and Northern Manitoba), and claimant status (active EI claimant, near reachback, mid reachback and far reachback).¹⁸ Table 2.2 presents the sampling plan for the participant survey (in the column labelled "quota"), as well as the number of interviews that were actually completed (in bold) with participants who differed according to these three variables. The targets for individual cells were adjusted based on the distribution of participants by geographical area, program and claim status, thus attempting to ensure that a minimal sample size (e.g., 40 cases) was available to permit segmented analysis based on any two of the variables simultaneously.

Given the size of the final sample (i.e., intended to be 1,400 interviews), segmenting the sample on the basis of all three variables simultaneously for the purpose of analysis would require that the sample be divided into 96 cells, reflecting the product of the number of categories in each of the sampling variables (i.e., eight programs multiplied by three regions and four claim status categories). For some cells, however, the available sample was not sufficient to support a quota of 40 interviews for the analysis of the data by two of the sampling variables (i.e., Employment Partnerships in Northern Manitoba). Thus for some programs, additional interviews were completed in other regions that had sufficient sample (i.e., Winnipeg) and, if no additional sample was available within the same region, attempts were made to complete additional interviews within the same program and claimant status category. A total of 1,393 interviews were completed. The distribution of sample drawn to obtain the desired number of survey completions is presented with response rate information in Table 2.2.

Survey of Participants: Response Rate

The response rates and refusal rates for participants in each type of program are presented in Table 2.3. The response rate is the proportion of cases from the functional sample who responded to the survey, while the refusal rate represents the proportion of cases from the functional sample¹⁹ who declined to participate in the survey.

Table 2.3 Response Rate for the Participant Survey

	EAS	WS	SE	EP	SLG/ Enhanced Fee Payer	App	POT	PBT	Total
Initial sample	3,565	581	864	487	1,677	728	381	1,158	9,441
(less) Unused sample	2,513	115	245	37	675	107	0	569	4,261
(less) Attrition
Number not in service	288	118	113	114	226	120	148	148	1,275
No knowledge of topic/ineligible	90	42	78	13	32	6	33	12	306
Language barrier (not English/French)	13	2	8	3	3	3	8	3	43
Functional sample	661	304	420	320	741	492	192	426	3,556
Other number retired (not due to attrition)
No answer/busy (<10 calls before end of fieldwork)	253	44	132	91	353	226	9	173	1,281
Unavailable for duration of survey	4	6	5	5	7	5	4	4	40
Retired/called 10+ times	20	15	19	7	17	14	28	14	134
Quota filled (Number called but no longer needed)	17	1	18	7	24	6	0	19	92
Other/illness	31	17	11	15	26	15	12	17	144
Total numbers retired	325	83	185	125	427	266	53	227	1,691
Non-response
Refusal	89	60	71	38	77	44	31	38	448
Incomplete refusal1	11	3	3	1	2	1	2	1	24
Total non-response	100	63	74	39	79	45	33	39	472
Total completed	236	158	161	156	235	181	106	160	1,393
Refusal rate	15.1%	20.7%	17.6%	12.2%	10.7%	9.1%	17.2%	9.2%	13.3%
Response rate	35.7%	52%	38.3%	48.8%	31.7%	36.8%	55.2%	37.6%	39.2%
Margin of error	±6.3%	±6.7%	±7.0%	±6.5%	±6.2%	±7.0%	±8.6%	±7.2%	±2.5%
1 Incomplete refusals refer to interviews during which the respondent refuses to continue part way through.

The response rate for the survey was lower than expected, ranging from 31.7 per cent among SLG/Enhanced Fee Payer participants to 55.2 per cent for Purchase of Training respondents, with an overall response rate of 39.2 per cent. The overall refusal rate was also higher than expected (13.3 per cent) and ranged from 9.1 per cent for Apprentices to 20.7 per cent for Wage Subsidies participants. The sampling error for the survey is ±2.5 per cent. That is, the overall survey results are accurate within ±2.5 percentage points, 19 times out of 20. The margin of error is higher for the sub-group analyses.

Survey of Participants: Weighting of Data

The grouping of weighting variables was dependent on the type of analysis. For the overall analyses, the participant data were weighted according to program type, region, claimant status and age. For analyses by program type, the data were weighted by region, claimant status and age. For analyses by claimant status (i.e., active EI claimant versus reachback clients), the data were weighted by program type, region and age.

Comparison Group Survey: Sample Frame Creation

The comparison group case file was drawn from a file of EI claims that were active at some time between 1994 and 1998. The administrative files included T1, Status Vector, ROE, NESS, and BNOP files. This dataset of comparison group members was then linked to the administrative data files, yielding a single file of 204,531 active EI claimants and reachbacks from which to draw the comparison group sample.

The comparison group active EI claimant and reachback samples were developed separately. To sample comparison group active EI claimants, the comparison group data file was matched to the participant data file based on the time periods for which members of the comparison group were receiving EI. To accomplish this, three time periods were defined according to observed values for program end dates in the population of active EI claimant participants.

The comparison group active EI claimant sample was drawn in the same proportions as were observed for active EI claimants in each of the three time period cohorts in the participant population.

To develop a comparison group sample for reachbacks, the proportion of different reachbacks (i.e., near, mid and far reachbacks) in each of the three time cohorts was determined for the participant population. Comparison group reachbacks (i.e., they were not on claim during any of the three theoretical reference dates) were then randomly assigned to one of the three time cohorts. Reachback status (i.e., near, mid, far) was determined based on the length of time since the end-date of their most recent EI claim period and the theoretical reference date (i.e., mid-point of the time cohort). Near, mid and far reachback comparison group cases were then sampled such that they reflected the proportion of these cases in the three time period cohorts that were observed for the participant population.

To correct for the fact that a comparison group case could possibly fall into any of the three time cohorts, each time period cohort was sampled separately. The final comparison group sample frame consisted of 2,633 cases in three mutually exclusive time period cohorts from a population of 204,531.

Comparison Group Survey: Response Rate

The response rate for the survey is presented in Table 2.4. The response rate is the proportion of cases from the functional sample who responded to the survey. Conversely, the refusal rate represents the proportion of cases from the functional sample who declined to participate in the survey.

The response rate for the survey was 34 per cent²⁰ and the refusal rate was 26.4 per cent. The sampling error is ±4.4 per cent. That is, the overall survey results are accurate within ±4.4 percentage points, 19 times out of 20. Comparison group survey results presented by claimant status (i.e., active EI claimants versus reachback) were weighted by region and gender.

Footnotes

16	It was decided that participants in the Project-Based Training (PBT) and Purchase of Training (POT) programs, both of which were phased out in 1999, should also be surveyed.
17	For EAS, only clients from the EI system were used in the sample and therefore the results/conclusions are only applicable to that client subgroup.
18	As described previously reachback clients are defined as clients that were not active EI claimants (i.e., collecting employment insurance benefits at the time of their participation in a PBPM) but who had had an active claim at some time in the three years leading up to their program(s) participation — or in the case of a maternal/adoption claim, in the five years leading up to their program(s) participation. Near reachbacks are participants whose most recent EI claim ended between one week and six months prior to program participation, mid reachbacks' most recent EI claim ended between 6.1 months and 36 months prior to program participation and far reachbacks' most recent EI claim ended between 36 months and 60 months prior to program participation. The decision to divide the sample according to these three groups was more pragmatic than theoretical in nature. First, far reachbacks were already operationally defined as a distinct group in the LMDA (i.e., far reachbacks comprise only clients who were on maternity or parental leave) thus it made sense to treat them as such. Secondly, the use of the near and mid reachbacks groups represented a relatively even division of the remaining reachback clients based on the distribution of these clients in the population.
19	The functional sample factors out the attrition in the survey, leaving only the sample which resulted in completions, refusals, and those numbers attempted but not reached by the completion of fieldwork (e.g., retired phone numbers, respondents who were unavailable for the duration of the survey, respondents who were unable to participate due to illness or some other factor, etc.). Attrition includes numbers not in service, respondents who do not speak either English or French, respondents who indicated no knowledge of the topic and respondents who were still available to be contacted once a quota had been reached (thus rendering them not needed).
20	Although the response rate for the comparison group survey (34 per cent) is similar to that obtained for the participant survey (39.2 per cent), this response rate is generally considered to be quite adequate for comparison group surveys. Considering that comparison group respondents have little direct connection to the topic of interest (employment programs and services), it is more appropriate to compare this response rate to rates obtained from surveys of the general public, where a response rate of 30 per cent is considered satisfactory.