Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Participant Survey

Data Acquisition

Participant data came from 10 different data source files, sent electronically at various times by Human Resources Development Canada (HRDC). With the exception of the EAS participants list, data lists were originally prepared by the Department of Labour. The EAS participants list was originally prepared by HRD-NB, while a file of matching administrative data for all EI clients over a three to five year period was prepared by HRDC. These data include:

• Participants in data file called CDF.RESU.XLS, including participants in the following intervention types: Partners/Wage Subsidy, Entrepreneur, Job Action/Creation, Skills Loans/Grants, and Employment Assistance. Received May 15, 1998 from HRDC (Strategic Services, Fredericton). No start or end dates for interventions were included; N=4,693.
• Participants not in the above file (i.e., not in CDF.RESU.XLS), including participants in the following intervention types: Job Action Top-up, Teachers Aid, SEB, Rural Experience, 1997-98 Partners, and 1996-97 Partners. Received from HRDC (Strategic Services, Fredericton), May 15, 1998; N=7,293.
• EAS participants. Received December 22, 1998 from HRDC (Fredericton); N=6,734.
• SLG participants. Received December 23, 1998 from HRDC (New Brunswick); N=250.
• Participants intervention start and end dates. Four files of start and end dates for interventions were received December 23, 1998 from HRDC (Strategic Services, Fredericton).
• Date of birth and language preference information from T1 data, received January 20, 1999 from HRDC (Hull).
• Additional EAS cases. Received February 11, 1999 from HRDC (Fredericton); N=50.

Data Issues

The first step in the sample development process was to merge the 10 participant data files and match them to the five administrative files to produce a single overall data file of 17,570 Cda/NB LMDA participants. When the 10 participant case files were merged together and matched to the five administrative information files, however, it was discovered that 250 participant cases did not appear in any of the administrative data files. As well, while addresses and phone numbers were found in all four participant files, as well as the "Name and Address" administrative file, there was a good deal of missing information in each file. Moreover, the data that were available were often inconsistent, which was most likely the product of inconsistent data collection protocols and changes of address owing to the fact that the data were provided by five different sources and were collected at different points in time. Where there was discrepant address and phone number information for a given case, the most recent complete entry was retained in the overall data file. Where data were missing for a given case, the most complete data from any source were retained.

Data Sampling

The data files were originally developed to include participants who participated in LMDA employment programs and services at any time between April 1, 1997 and October 31, 1998. As mentioned, these files were aggregated, yielding a single overall data file containing information for 17,570 participant cases, with the individual client as the unit of analysis. This was not equal to the sum of all the cases from the administrative data files because clients that had taken part in more than one intervention could appear in more than one file. Following the removal of all cases without valid phone numbers, start and end dates for EI benefits, and start and end dates for most recent interventions, the final data file consisted of 14,807 individuals.

The survey sample was randomly drawn from the final data file using a three to one "sample to survey completion" ratio for each different participant group (i.e., three times as many participants were sampled as were expected to complete the survey). For the EAS, Job Action and Entrepreneur participant groups, however, there were not enough cases available to obtain this three to one ratio. Thus, for an expected total of 1,665 survey completions, a total final sample of 4,564 cases was drawn from the data file of 14,807 program participants.

It is important to note that the small number of program participants in EAS proved insufficient to obtain the expected number of survey completions. To partially compensate for this, Ekos received an additional 50 unique names and phone numbers of EAS participants from HRDC in Fredericton. Although this provided a few more survey completions with EAS participants, there were still far fewer completions than originally expected (117 as opposed to 235). Thus, the final total number of completions for the participant survey was 1,600 rather than 1,665 as was originally anticipated.

Response Rate

Fieldwork for the survey began on January 15, 1999 and was completed on February 8, 1999. The response rates and refusal rates for participants in each type of program are presented in Exhibit 1. The response rate is the proportion of cases from the functional sample who responded to the survey, while the refusal rate represents the proportion of cases from the functional sample who declined to participate in the survey. The functional sample factors out the attrition in the survey, leaving only the sample which resulted in completions, refusals, and those numbers attempted but not reached by the completion of fieldwork (e.g., appointments for interviews that were not kept, retired phone numbers, respondents who were unavailable for the duration of the survey). Attrition includes numbers not in service, duplicate phone numbers, respondents in groups for which the quota had been reached and respondents who indicated no knowledge of the topic.

The overall margin of error for the survey is ±2.3 per cent. That is, the overall survey results are accurate within ±2.3 percentage points, 19 times out of 20. It should be noted that the response rate for the survey was very good, ranging from 45.5 per cent among SLG participants to 66 per cent for Job Action respondents, with an overall response rate of 53.5 per cent. The overall refusal rate was also highly satisfactory (7.1 per cent) and ranged from 4.8 per cent for Job Action participants to 10.9 per cent for EAS participants.

Comparison Group Survey

Data Acquisition

The comparison group sample was drawn from a file of EI claims that were active in 1998 (n=97,132) and a file of dormant EI claims (claims that were not being processed) from 1994 to 1998 (n=565,766), both of which were received from HRDC, Fredericton. These files were merged into a single file of EI claims (n=662,898), representing a total of 250,001 individuals. Entries for EI claimants who had participated in LMDA programs and services (n=17,570) or who had not had active EI claims since April 1, 1997 (n=130,204) were removed, leaving a comparison group population of 102,227 claimants from which to draw the comparison group sample.

Data Sampling

It was decided that the comparison group data file would be matched to the participant data file based on the time periods for which they were receiving EI. To accomplish this, three time periods were defined according to observed values for program end dates in the population of program participants. That is, within the population of program participants, the end dates of participants' most recent interventions ranged from April 1, 1997 to October 31, 1998 and this time period was divided into two six month and one seven month time period. For comparison group sampling purposes, the following time periods were derived: April 1, 1997 to September 31; October 1, 1997 to April 30, 1998; and May 1, 1998 to October 31, 1998. Reference data flags were then computed using the mid-point in each of these time periods (June 1 1997, December 1, 1997, and July 1, 1998) so that if an individual in the comparison group was EI eligible at the reference date (at the mid-point of the time period), that individual would fall into the time period cohort. This meant, however, that these were not necessarily mutually exclusive cohorts because an individual could have been EI eligible at more than one of the reference dates.

Based on the participant population characteristics, for each time period cohort, a listing was produced of the time in weeks between the end of the latest intervention and the start date of the most recent EI eligibility period. These time frames were further broken down into five categories based on the amount of time into the EI eligibility period that the participant's intervention came to an end. These categories were 13 weeks or less, 14 to 26 weeks, 27 to 39 weeks, 40 to 52 weeks, and 53 weeks or more. Each comparison group cohort was then similarly broken down into the same five categories based on the time in weeks between the reference date (the mid-point of one of the three time period cohorts) and the start date of the most recent EI eligibility period. (see Exhibit 2).

The comparison sample was drawn in the same proportions as were observed for each of the three time period cohorts in the participant population. The comparison group sample was further stratified by the number of weeks between the end date of the latest intervention and the start date of the most recent EI eligibility period.

To correct for the fact that the comparison time cohorts are not mutually exclusive, each time period cohort was sampled separately and a flag was computed to identify sampled cases. As such, it was possible to track these cases and not include them when sampling from subsequent time periods. Thus the final comparison group sample consisted of 2,400 cases in three mutually exclusive time period cohorts from a population 102,227.

Response Rate

Fieldwork for the survey began on February 9, 1999 and ended February 13, 1999. The response rate for the survey is presented in Exhibit 3 using the same response categories as described earlier. The overall margin of error is ±3.5 per cent. That is, the overall survey results are accurate within ±3.5 percentage points, 19 times out of 20. The response rate for the survey was 48.4 per cent and the refusal rate was 8.8 per cent.

Survey of Employers/Unions

Data Acquisition

A single data file was received from DOL which contained the contact information for 2,074 employers who had participated in the Partners, Job Action and/or Rural Experience programs. This data file was cleaned by removing all duplicate cases, that is, cases where the same combination of employer name and contract type (i.e., the program participated in) appeared more than once in the data file. This resulted in a file with 1,867 cases that were unique in so far as no two cases had the same employer name and contract type, however those employers who had participated in more than one program had more than one entry in the data file.

Data Sampling

The sample was drawn to ensure that three times as many employers were included in the sample as were needed to complete the survey. For the Job Action program, however, only 197 cases were available in the original data file, thus all Job Action cases were included in the sample. The sampling procedure also ensured that employers who participated in more than one program were only included in the sample for one type of program. In order to obtain 275 completions, a total of 772 cases were sampled.

Response Rate

The employer survey went into the field on February 12, 1999 and was completed on February 24, 1999. The response rate for the survey is presented in Exhibit 4 using the same response categories as used for the participant and comparison group surveys.

The overall margin of error is +/— 4.4 per cent. That is, the overall survey results are accurate within +/— 4.4 percentage points, 19 times out of 20. The refusal rate for all employer groups was very low and ranged from 3.9 per cent for Rural Experience employers to 6.8 per cent for Partners program employers, with an overall refusal rate of 5.3 per cent. The response rate for all employer groups was equally satisfactory, ranging from 41.6 per cent among Partners program employers to 54.7 per cent for Rural Experience employers. The overall response rate was 47.9 per cent.