Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The response rate for the participant survey was 40.3%, for the non-participant survey, 28.8%. This also represents the proportion of the population surveyed, since all participants and non-participants were eventually included in the sample.

	Participants	Non-Participants
Sample universe	1,609	1,870
Completed interviews	648	538

Because the response rate was modest, it pays to check that those who responded to the survey are not somehow different from those who did not. To check for such a bias, we compared various traits of participants and non-participants in the final sample to the population in terms of key variables. The next four tables show a lot of statistically significant differences between those who responded to the survey and those who did not. But seldom is difference of practical significance. In many instances, this occurs because the statistical tests available for nominal data are sensitive to the number of cases, and there are almost 2,000 cases for most of the tests. Therefore the tables also include the phi coefficient, a "measure of association," which indicates the strength and nature of relationships between nominal/ordinal variables⁶⁰. This statistic never reaches .15 for any of the variables tested, meaning that most differences between respondents and non-respondents are not large.

Table C.1 shows the distribution of respondents and non-respondents by Compass component, for participants and non-participants. In both groups, TTO was over-represented and WEO under-represented among respondents. This should not have a large impact on data analysis because most analyses are presented by component. Where the effect of the program as a whole is being considered, however, the results may be slightly biased to the extent that TTO and WEO have different effects.

	Participants		Non-Participants
Option	In Sample	Not In sample	In Sample	Not In sample
WEO	40.9%	49.5%	25.8%	36.6%
TTO	52.5	46.5	71.9	61.8
EDO	6.7	4.0	2.3	1.6
Statistical test	X2 = 14.3, df = 2, p<.01		X2 = 19.6, df = 2, p<.001
Measure of association	phi = .094		phi = .106

Tables C.2 and C.3 examine demographic variables. For participants and non-participants, there were significant differences between respondents and non-respondents for region, gender, marital status, education, age and number of children. Yet, a look at the distributions/means suggests that respondents and non-respondents are much alike in most of these areas. For example, the distributions by education level look reasonably close, though the differences reach significance. The low phi coefficients indicate that the differences are not great, however. The story with age is similar: the mean age was significantly different between respondents and non-respondents for both groups. Yet, in absolute terms, the difference was only 1 ½ years, not enough to worry about. An even better illustration of the contrast between statistical and practical significance is the data for average number of children for participants. Those responding to the survey had an average of 0.7 children; those who could not be reached had an average of 0.6 children; still the difference was statistically significant. This is not to imply that all of the differences can be ignored, though. Differences by gender are noteworthy, and could bias results to the extent that men and women fare differently after Compass. And, Halifax cases are under-represented, Cape Breton cases over-represented in the final sample. If results differ by region, that could bias overall findings. Single (never married) cases are also under-represented in the final sample. The analysis will check for differences by these key variables.

Table C.3 Mean Age, Number of Children and Number of Children Needing Child Care

	Participants		Non-Participants
Option	In Sample	Not In sample	In Sample	Not In sample
Age	31.6	30.0	34.0	32.6
Statistical test	t = 3.5, df = 1503, p<.01		t = 3.1, df = 1766, p<.01
Number of Children	0.7	0.6	0.9	0.7
Statistical test	t = 2.1, df = 1529, p<.05		t = 5.1, df = 1760, p<.001
Number of Children Needing Child Care	0.2	0.2	0.3	0.2
Statistical test	t = 1.3, df = 1559, p>.20		t = 2.1, df = 1807, p<.05

The last table divulges a considerable difference between respondents and non-respondents in cost of the intervention. The mean cost for respondents was about 10% higher than that for non-respondents.

Table C.4 Mean Compass Expenditures

Cost Category	In Sample	Not In Sample
Total Compass Cost	$4,920	$4,461
Statistical Test	t = 3.7, df = 1419, p<0.001
Opportunity Fund	$9.98	$8.24
Statistical Test	t = 0.9, df = 3646, p>.30
Hourly Pay on Placement	$6.03	$5.80
Statistical Test	t = 2.8, df = 1127, p<.01

Perhaps not surprisingly, those we were not able to reach were significantly more likely to have quit Compass than were those who were surveyed (X² = 10.3, df = 1, p < .01): 18% of non-respondents had quit versus 12% of respondents. Again, though, the association between the variables was weak (phi = -.080).

We also checked for differences in earnings, income and UI history between those in the sample and those not. There was only one important difference on the participant side: earned income during 1995 was $527 higher for respondents than for non-respondents. For non-participants, weeks on UI during 1993 and 1996, and total income in 1994 were significantly different. But the absolute differences were very small: keep in mind that when running dozens of t-tests, a few are bound to be significant at the 5% level just by chance. (Some differences that may in fact be significant may also have turned out non-significant just by chance.)

Finally we checked for differences between participants in and out of the survey sample in attitudes and we found significant differences for six of the 24 statements. But, again, the same caveat holds for running so many t-tests. Moreover, although six of the t-tests were statistically different, none were of much practical difference. All the significant differences were within 0.25 points on the five-point scale. It is noteworthy, though, that three of the six differences involved respondents' projections of how well they would do in the future. Those in the sample tended to be slightly less optimistic that they would maintain steady employment, get off social assistance, or apply the skills they expected to learn.

Footnotes

60

Tests of statistical significance determine whether or not a relationship exists between variables, but they don’t measure the strength of the relationship. Measures of association were developed for this purpose. Interpretation of phi is as follows (Rea and Parker, 1992): .00 - .09 negligible association; .10 - .19 weak association. No phi coefficient in this section is higher than .15.