Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

6.1 Cost-Effectiveness and Alternatives

The evaluation question directly addressed in this sub-section is the following: Is the NLSCY approach the most cost-effective and appropriate mechanism to produce the information needed to develop programs which would improve the socio-economic well-being for all children and youth in Canada? How can the current mechanism be improved?

In general, existing research indicates that the benefits of longitudinal surveys outweigh their costs. Numerous researchers agree that when the cost of information is compared to the benefits of its use, in prevention of suffering and in lowering health care costs, it is clear that money spent on longitudinal surveys of children is an excellent investment. Also, arguing that causal analysis can be more effectively undertaken using longitudinal data, authors state that it makes more sense to spend the money to get the right answers through longitudinal research than to obtain less conclusive answers in cross-sectional research.

Key informants and Delphi Panelists included in the evaluation tended to agree with the literature. The general consensus of the Delphi panel was that longitudinal studies are the most cost-effective research method to provide the information that is needed to help researchers find ways to reduce risks to children and improve their lives. Similarly, the majority of Delphi respondents (58.3 percent) felt that cross-sectional studies did not provide the best data for use in developing programs that reduce risks to children and improve their lives.

Even those panelists who did say that cross-sectional studies could provide the necessary data to reduce risks to children were generally also in favour of longitudinal studies. For example, one participant stated that cross-sectional studies could provide key data, but that they were "definitely second best" to longitudinal studies.

Alternatives: Several of the Delphi Methodologists suggested alternative methods of conducting the NLSCY in order to make it more useful, without significantly escalating its costs. For example one panelist suggested that it may be preferable to follow only two cohorts longitudinally, rather than the current seven. The panelist believed that if only 2 cohorts were retained for the longitudinal portion of the NLSCY, extra resources would be available to augment the sampling.⁵¹ Likewise, it was suggested that cost-savings could be achieved by surveying the NLSCY children every three, rather than every four years.

Other panelists argued that the number of cohorts should be limited if budget constraints require it. The panelists who were in favour of limiting the number of cohorts generally tended to agree that the cohorts to be retained in such a situation should be those of children aged 0-11 months and 10-11 years. However, other Delphi participants disagreed with this suggestion for a variety of reasons. One panelist stated "without question the sample is the strength of the study — this should be preserved", while another panelist argued "there are risk factors associated with each age cohort. I don't agree that some of the age cohorts are more important than others".

The Length of the Survey: The number of instruments and their length were considered to be a weakness of the NLSCY by several Delphi Methodologists, one of whom suggested eliminating all instruments/questions that did not obtain good response rates in Cycle I. A majority of the Delphi panelists agreed with this suggestion. 42.9 percent of the Delphi participants said that all instruments without good response rates should be eliminated, while 21.4 percent stated that only specific instruments (i.e. principal's questionnaire or those instruments with less than 5 percent response) without good response rates should be eliminated.

Table 6.1
Allocation of Hypothetical Extra Monetary Resources
by Delphi Panelists*

Allocating extra funds: Delphi Methodologists were asked how they would allocate additional monetary resources if they became available. Panelists tended to favour spending money on oversampling target groups, increasing the sample size in each province and many suggested including indicators of mental health (see Table 6.1).

Table 6.2
Suggested Improvements to the NLSCY Methodology
by the Delphi Panelists*

Importance of Suggested Methodological Changes: In general, the Delphi Methodologists felt that the most important changes to improve the NLSCY would be to follow the children into their late twenties, to "top up" the sample to maintain its representativeness and to oversample key populations and cohorts (see Table 6.2).

6.2 Some Content Suggestions for Future Cycles (Adolescents)

A variety of suggestions on content for future surveys of the NLSCY youth were put forward by participants in the evaluation, the final round of which came from the discussion group (these suggestions are complementary to other content suggestions that were noted in Section 5.2).

The evaluation participants suggested topics which they felt were important to cover in future rounds of the NLSCY (i.e. for children 11 years and older) including: career planning; educational expectations; dating behaviour; sex education; sexual activities; attitudes towards drugs and smoking; and delinquency.

The discussion group participants felt that such additions were important to cover for a variety of reasons including the fact that: educational expectations are one of the best predictions of drop-out; 29 percent of grade 9 students are sexually active; and 50 percent of students are sexually active by grade 11.

Further, it was suggested that it was important to be able to answer questions such as: Are these adolescents using precautions against pregnancy and sexually transmitted diseases? Is sexual education in the schools effective in preventing pregnancy and STD's? Does sex education at an early age contribute to increased sexual activity?

Footnotes

51	Please refer to the Working Report on the Delphi Panel Study of Methodologists for more details.