Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Program Description

The National Longitudinal Survey of Children and Youth (NLSCY) was developed with the objective of collecting data to identify risk factors for Canadian children, thereby improving society's understanding of the important process of child development. It is expected that information form the NLSCY will provide strategic insight for the formulation of more effective programs and policies for children at risk. Data was collected beginning in the fall of 1994 from a targeted population of 25,000 Canadian children.

The survey is expected to be repeated every 2 years until at least 2002. The sample was comprised of cohorts of age groups from 0-11 years in 1994, and information was provided by the parents, the children themselves (for children of 10-11 years of age), and school teachers and principals. The sample is drawn from all Canadian children, excluding Aboriginal children living on Reserves and children in institutions.¹ The survey was developed by Human Resources Development Canada and Statistics Canada with input from technical and advisory subject matter experts, other departments and Provincial/Territorial representatives.

While the immediate objective of the NLSCY is to create a database on child development and risk factors to guide future policy and programs, the downstream social implications of carrying out the NLSCY are to reduce the long term costs to society. Information resulting from the NLSCY will create a new base of knowledge. New knowledge will hopefully result in various preventive actions to increase the health and well-being of children and youth. This will in turn result in cost savings and clear societal benefits by reducing, among other things, sickness (and therefore visits to the doctor and hospitalization), and the frequency of delinquent behaviour (and thus the institutionalization of troubled teenagers).

The NLSCY is part of the broader interdepartmental Child Development Initiative (CDI) of the Federal Government. The CDI provides a wide range of programs for Canada's children in such areas as nutrition, child health, and justice and is Canada's response to the United Nations summit for children. The CDI's objective is to address conditions of risk which threaten Canadian children's health and well-being.

Evaluation Description

This report outlines the findings of the Evaluation of the NLSCY. The Evaluation was undertaken in response to the Treasury Board's requirement that all programs conducted as part of the Child Development Initiative be evaluated by the end of the fourth year of the five year initiative (1992-97).

As the NLSCY is a long-term project as and data from the first cycle of the NLSCY survey were not available at the time of this study, this report constitutes a process evaluation of the NLSCY. The purpose of this evaluation was to examine quality of survey design and development; relevance, rationale and need for the NLSCY; anticipated usefulness and potential application of data from the longitudinal survey. In addition to providing an initial formative evaluation, it lays the groundwork for a later, summative Phase II evaluation of the NLSCY which is expected to examine actual uses made of the data and impacts of the NLSCY.

This formative evaluation study involved a number of research components and activities including: a Review of Background Information and comparisons with other Canadian and international children's studies; Key Informant Interviews (n=30); a Delphi Panel Study of Methodologists (n=24); a Survey of Potential Users across Canada (n=124). As well a Focus Group of Delphi Methodologists and Subject Matter Experts was conducted at the end of the study. These participants in the evaluation interviews, surveys and focus groups are referred to within this report as evaluation participants. Results from each of the components were detailed in separate Working Papers and are bound separately from this report (see a list of working papers in Appendix B).

Key Findings

Relevance and importance of the NLSCY: A review of recent Canadian research on children indicated that the NLSCY does not duplicate any other Canadian study currently being undertaken or having been recently completed. Other Canadian research on children tends to be based on smaller samples; to be conducted regionally rather than nationally; to be cross-sectional rather than longitudinal; and to focus on specific topics rather than a broad range of topics.

Thus, the NLSCY appears to be unique in its combined characteristics which provide a national sample, broad topic coverage, and longitudinal methodology. The NLSCY fills an important gap in current Canadian research on children, which suggests that the NLSCY is very relevant.

The relevance of the NLSCY was clearly supported by findings from each of the four research components. All of these component studies indicate that NLSCY data will be a valuable tool for future research which will contribute to an increased knowledge of the characteristics of Canadian children. When these research results are combined with the fact that longitudinal data are necessary for the analysis of causal relationships, the relevance and importance of the NLSCY are well supported. In fact, many evaluation participants believe that the NLSCY will grow in relevance and usefulness over time as more longitudinal data becomes available.

Role of the Federal Government: The evaluation research identified a variety of reasons why the federal government should play a major role in a study such as the NLSCY. First, longitudinal studies require secure funding throughout the study in order to guarantee their longevity. Second, governments can guarantee the organizational survival of a longitudinal study even after the original principal researchers have changed. Finally, it is more practical for a national organization, such as the federal government, to conduct national studies.

The NLSCY development process: The development of the NLSCY was generally a positive process. Key informants, for the most part, were very satisfied with the consultative and collaborative nature of the process, particularly members of the academic community. Although most participants were pleased with the study process, some indicated that the process would have been aided by a clearer prioritization of study goals.

As well, the NLSCY development process reportedly was hindered somewhat in the early phases of the project by a need for clearer operational goals and roles for HRDC and Statistics Canada. These issues may have stemmed from the unique mandates of the two organizations: that of HRDC as principal funder and the source of scientific and intellectual leadership for the study, and that of Statistics Canada, with its specific role in the conduct of the field surveys and generation of statistical results (and its unique related legal responsibilities under the Statistics Act). Recent reports suggest that efforts to clarify the HRDC/ Statistics Canada goals and roles have been successful and that communications and definitions of roles and responsibilities have improved considerably over the past year. Tight time frames were also a concern during the NLSCY development process. Additionally, many Provincial/Territorial representatives indicated that the development process could have included stronger consultation.

NLSCY Survey Design: The NLSCY survey design was highly rated by participants in the methodologists Delphi Panel and the Survey of Potential Users. Overall soundness of the survey design, and almost all aspects of sampling, content and survey instruments, were rated highly by the Delphi Panel Methodologists. Despite positive evaluations of the NLSCY survey design, most evaluation participants had suggestions for improvements.² Delphi Panelists and others suggested that small improvements to existing measures in the NLSCY could greatly enhance the usefulness of the NLSCY and should be a priority for the future iterations of data collection. As well, one area in which the NLSCY was not highly rated by Delphi Panel members was with regards to the study's breadth as opposed to depth of information gathered. Many panelists suggested that some information should have been obtained in greater depth. Even so, for the most part it was acknowledged by methodologists that a trade-off has to made between breadth and depth in the NLSCY content, and that the breadth of the NLSCY was reflective of the study's diverse goals.

Data Collection Methods: The evaluation found that methodologists and potential users generally approved of the NLSCY data collection methods. Listed strengths of the data collection methods included the use of personal home interviews; the variety of informants consulted for each child; and the use of both personal interviews and self-report questionnaires.

Some Issues: The Computer Assisted Personal Interview (CAPI) computer systems were reported as being too slow, during Round I of the NLSCY data collection, making interviewer rapport with the respondents more difficult. As well, it was indicated that some of the parental consent forms needed to implement the in-school component of the NLSCY were not completed at the time of the in-home interview, as a result of the CAPI set-up. This is thought to have been a factor affecting the subsequent cumulative response rates for the in-school component.

These problems have been addressed for Cycle II of the NLSCY. However, it has been suggested that response rates for the in-school component may be difficult to improve significantly, even with the changes in the CAPI system which have been undertaken since Cycle One since other factors such as school board authorizations and teacher responses compound response rate problems. Also, evaluation focus group participants were sceptical that mail surveys or personal interviews alone could easily resolve these problems, since they believe a good portion of non-response comes from teacher's apprehensiveness to give up their time to fill out surveys or take part in an interview. It was thus suggested that important information could also or alternatively be obtained from the child's school records to supplement teacher data.

New Knowledge: The evaluators found widespread expectations that the NLSCY will generate new knowledge about all aspects of children's lives. It was pointed out by many evaluation participants that the NLSCY is providing new knowledge simply by the fact that it is the first national and longitudinal study of children in Canada. It was noted that the most new knowledge collected by the NLSCY will come from determining the various factors which place children at risk, and how those risk factors interrelate. Until now, research has linked certain risk factors to development, but a multitude of risk factors have not been incorporated into a single study in order to provide researchers with an overall perspective concerning the relative importance of a multitude of factors. The NLSCY is the first study in Canada to do this.

Content Gaps: A variety of content gaps were noted by evaluation participants. The content gaps mentioned most frequently included information on sexual, verbal and physical abuse; poverty; positive influences in the child's life; drug and alcohol use; and social influences in the child's life. Evaluation participants also felt that collecting information on paternal parenting styles, literacy and expanding tests like the PPVT to all age levels were important issues to address. Some of these gaps are in fact covered by the NLSCY but, some evaluation participants suggested, not covered in sufficient detail to provide the data these evaluation participants wanted.

It must be concluded that certain content gaps are inevitable given NLSCY's need to focus the survey, control the length of the survey questionnaire, and meet tight scheduling and budgeting requirements. Furthermore, the NLSCY was designed to provide breadth of issue coverage rather than depth of issue coverage. Therefore, the evaluators concluded, many of the gaps in NLSCY coverage which were identified, occur because of the chosen breadth of the NLSCY — within which not all topic areas can be covered in full detail.

Potential Uses of the NLSCY: All respondents to the Potential Users Survey felt that they would use the NLSCY data from Cycle I, while approximately 70 percent of the respondents also felt that they would make use of the longitudinal data which would be available in the future. Largely because of their only becoming aware of the NLSCY at the time of this study, a substantial minority of potential users reported that they did not yet know what their plans would be (if any) for the use of the longitudinal data.

Evaluation participants rated highly the potential use of the NLSCY data to design and improve delivery of programs for children. The other potential uses that received high ratings included: to develop policies; to conduct analyses of key research issues; and to identify targets for new programs. Most potential users felt that the longitudinal data would be more useful than any cross-sectional data: to identify targets for new programs; to design new programs; to improve delivery of current programs; to develop policies; and to conduct analyses of key issues covered by the NLSCY. According to reports from potential users the most useful potential outputs of the NLSCY are expected to be the highlights report by Statistics Canada; distillations of results designed to aid children's programs; and specific reports that will follow the release of the data. The methods of disseminating the data which were most often suggested include: through government publications; news releases; conferences; and workshops. A large number of the evaluation participants also suggested using the Internet to provide information on the NLSCY and on how to access NLSCY data.

Obstacles to Using the Data: According to potential users, the main obstacle to using the NLSCY data is the likely cost of purchasing the data .³ The limitations of Provincial and Territorial sample breakdowns, the lack of human resources (within their organizations), and the lack of knowledge on how to use longitudinal data were also frequently identified by potential users as being likely obstacles to the use of the NLSCY data. Lack of technical expertise and resources were also stressed as a clear obstacle for community groups to using the longitudinal data. Providing materials and reports specifically for use by community groups, written for lay persons, was suggested as a way to reduce this concern. Lack of awareness of the NLSCY was also noted as a substantial obstacle to use of the survey, as many potential users were aware of the survey only because of being contacted by the evaluators. Extensive use of the media in advertising the NLSCY was suggested by many of the NLSCY evaluation participants.

Support Systems: A variety of support systems to assist users of the NLSCY were suggested. These include: the creation of an NLSCY user group; provision of a resource person/team for consultation on how to use the data; the provision of training to assist all researchers, especially those who are used to working only with cross-sectional data; providing clear and accessible documentation to facilitate the use of the NLSCY; using the Internet; providing a media contact or a spokesperson to facilitate the use of the data by the media; facilitating research which is conducted with the NLSCY data; and the hiring of an NLSCY data librarian. These needs point to a variety of important support roles which Statistics Canada might consider following, beyond its usual role of facilitating the understanding of the data files.

Cost-effectiveness: In general, existing research examined in the evaluation literature review indicates that the benefits of longitudinal surveys outweigh their costs. When the cost of information is compared to the benefits of the use of the data, it was suggested that the resources dedicated to the NLSCY are appropriate. Furthermore, most researchers argue that causal analysis can be more effectively undertaken using longitudinal data — cost-effectiveness goals are met when complete and verifiable answers are obtained (longitudinal data) rather than inconclusive answers (cross-sectional data).

Alternatives: Several alternative methods of conducting the NLSCY were suggested in order to make it more useful or cost-efficient, without significantly increasing costs. For example, it was suggested by some members of the Delphi Panel that it may be preferable to follow only two cohorts longitudinally, rather than the current seven. It was suggested that if only two cohorts were retained for the longitudinal portion of the NLSCY, extra resources would be available to increase the sample size. Not all Delphi participants agreed with this suggestion however, many believing that the number of cohorts is a strength of the NLSCY.

It was also suggested that instruments/questions that did not obtain good response rates in Cycle One be eliminated in order to reduce the length of the questionnaire. Many Delphi participants agreed with this suggestion. Further suggestions included collecting data from the children and their parents at 3 year rather than 2 year intervals, reducing the sample size in larger provinces and increasing the sample size in smaller provinces, and eliminating the cross-sectional samples. It was suggested that the money saved from these measures could allow for increased sample sizes, replacing those survey respondents who have attrited from the survey; oversampling immigrants, Aboriginals, or other risk groups.

Conclusions

The NLSCY was found to be highly relevant and an appropriate initiative of the Federal Government. As well, it appears to be a cost-effective and significant undertaking for the development of new information on Canadian children which does not duplicate any other Canadian study. The NLSCY will provide a wealth of new knowledge on the development and characteristics of children in Canada, according to a wide range of subject matter specialists and methodologists who participated in the evaluation. Further, the NLSCY appears to have provided a variety of important advances in measurement and survey techniques. Thus the NLSCY was seen to be a significant accomplishment.

The NLSCY survey design was highly rated throughout the evaluation. Overall soundness of the survey design and most aspects of sampling, content and survey instruments were also highly rated by the evaluation's methodological Delphi panelists. Some areas for improvement were suggested, along with the suggestion that goals of the survey be clarified/prioritized. Generally, however, the NLSCY and its methodology were strongly supported by methodologists and potential users from across Canada.

The development of the NLSCY was reportedly a positive process. Key informants, for the most part, were satisfied with the development process, although the Provinces/ Territories would have preferred a stronger voice in the consultations. Need for better role definitions in joint operations of the Statistics Canada/HRDC Project Team , stemming from the unique mandates of the two organizations, were found to have been substantially remedied over the past year.

A wide range of potential impacts were noted, with the understanding that additional planning was required by the NLSCY managers to maximize awareness of the data among potential users.

Footnotes

1	Note that extensive effort, involving several Departments, was given to the topic of developing a study component for Aboriginal children living on reserves, but obstacles to such a component were not resolved. Note also that future cycles will study Cycle One children who later became institutionalized.
2	For a more detailed look at the suggested improvements, content gaps, etc. please refer to the detailed Working Reports on the Survey of Potential Users and the Delphi Panel Study of Methodologists. This working report lists suggested changes to the NLSCY's methodology as a whole, as well as suggested improvements to individual measures found in the NLSCY.
3	Few respondents were aware however, that costs to academic users will be relieved under the Statistics Canada Data Liberation Initiative.