Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

A number of procedures are available for dealing with the potential problem of selection bias. In order to describe these methods it is helpful to use some concrete examples of the forms that selection bias frequently takes in evaluation research on education/training programs.

Example 1: Coop versus non-coop education

Cooperative education programs combine periods of formal education with periods of work experience in a systematic manner, while the more common non-coop educational programs provide formal education without organized periods of work experience. Many observers feel that the combination of education and work experience is likely to enhance the employability and earnings of coop graduates. Does empirical evidence support this belief? A simple way of answering this question would be to compare the post-graduation employment and earnings experiences of a sample of coop and non-coop graduates. Would this be convincing evidence of the impact of coop programs on employability and earnings? The answer in general is no.

The reason is that there may be reasons why those completing coop programs may have different levels of employability and earnings than those completing non-coop programs that are independent of the educational programs themselves. For example, if there are a limited number of coop programs (as is generally the case), these programs may be able to admit more qualified students, on average, than comparable non-coop programs. Similarly, the students who apply for coop programs may be more career-oriented, on average, than those applying to non-coop programs. Both these reasons would lead to the coop graduates having higher levels of employability and earnings even in the absence of completing a coop program. That is, if the coop graduates had instead completed non-coop programs, their employability and earnings would have been higher than the non-coop graduates. To some extent, higher levels of employability and earnings of the coop graduates are due to their being better, more qualified students and to their being more career-oriented than their non-coop counterparts.

Of course, it is also possible that the coop programs have a positive impact on the employability and earnings of graduates. If so, the total observed difference between coop and non-coop graduates consists of two components: one because coop programs attract more qualified and more career-oriented students (the selection effect), and one due to the impact of the program (the program impact effect).

This is the problem of potential selection bias. Coop programs are selecting the more qualified and career-oriented students who would have had higher employability even in the absence of the program. In these circumstances, a simple comparison of coop and non-coop graduates would over-estimate the true impact of the program. However, as discussed below, selection bias may be either positive or negative; that is, without accounting for selection bias, the estimated impact may be either above or below the true impact.

Example 2: Private schools versus public schools

Many of the same issues arise if one wishes to compare the outcomes (such as average grades in standardized examinations, high school completion rates, or post-secondary success) of private and public schools. Private schools may be able to select more qualified students, on average, than their public school counterparts. Similarly, the average student attending a private school may be more likely to have other attributes (such as having parents who place a higher value on education) than the average student attending public school.

For these reasons, a simple comparison of student outcomes in private and public schools is unlikely to give unbiased estimates of the impact of private schools on these outcomes. Some of the observed differences arise because private and public schools select students who differ systematically on such attributes as qualifications, parental wealth and family attitudes towards education.

Example 3: Impact of government-sponsored training programs

As will now be clear, similar issues of selection arise in the context of assessing the impacts of training programs. Those who undertake training are likely to differ systematically from those who do not take training. This could be due to differences between the trainees and non-trainees themselves; for example, those who are more educated or more labour market oriented may be more likely to apply for training. Alternatively, these differences could arise because of selection by program administrators, who may be more likely to enroll those who are most likely to benefit from training.

This example also helps make clear why selection bias may be either positive or negative. Suppose a program is designed to focus on the most disadvantaged among a particular population. In these circumstances, a simple comparison of trainees and non-trainees is likely to under-estimate the true impact of the program.

These three examples illustrate the point that virtually any program evaluation is likely to face some potential selection bias. The reason is that participation in virtually all programs or interventions involve choices on the part of participants and non-participants and on the part of those administering the program. Because of such choices, the pools of participants and non-participants are likely to differ in systematic (or non-random) ways. If such differences between participants and non-participants are also related to the program outcomes, simple comparisons of participants and non-participants will give biased estimates of program impact. To deal with the very pervasive problem, a number of statistical methods have been developed.