Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

1.1 NLSCY Background: Context in Child Development Initiative, Objectives and Cost

The National Longitudinal Survey of Children and Youth (NLSCY) was implemented as part of the Interdepartmental Child Development Initiative (CDI), whose goal is to address conditions of risk which threaten children's health and well-being. CDI is Canada's response to the United Nations World Summit for Children, "Brighter Futures", and involves numerous departments, namely Health, Justice, Solicitor General, Heritage, Human Resources Development and the RCMP. There are over 30 programs under CDI. HRDC receives approximately 2 percent of all CDI funding.

The NLSCY was developed jointly by Human Resources Development Canada (HRDC) and Statistics Canada, and is implemented by Statistics Canada under contract to HRDC. The NLSCY survey design was developed with input from an expert advisory group, an interdepartmental consultation committee and Federal/Provincial/Territorial consultations.

The primary objective of the NLSCY is to create a national database on the characteristics and life experience of Canadian children as they grow from infancy to adulthood, and thus to aid development of national social policies. The database will measure and monitor the prevalence of various characteristics and risk factors, and the impact of these conditions on the development of children. It is intended that the application of the findings by government agencies, academics, community groups, schools, and others, will help in the development of more effective long-term policies and programs for children at risk, and aid Canada in providing policy leadership in these and related areas.

A total of 25,000 children from across Canada (with an additional 2,300 children from the Territories) from newborn to 11 years old in 1994, were targeted for taking part in the NLSCY. This group, excluding children of parents who decided not to participate, or who cannot be located, will be surveyed every two years as they grow to adulthood. Participating households in the ten provinces were selected from a list of households provided by Statistics Canada's Labour Force Survey (LFS). The NLSCY sample is generally representative of the Canadian population but excludes certain populations which did not take part in the LFS, specifically children in institutions and children living on Indian reserves. Although the LFS excludes the Yukon and the Northwest Territories, the NLSCY drew a sample of households for these areas.

The NLSCY sample for the first round of the survey included all children who were members of the same economic family,⁴ who resided in the selected household and who were under the age of 11. A maximum of four children were surveyed in each family, with children aged 1 and under oversampled. In families with more than four children under the age of 12, four children were selected randomly. For purposes of analyzing the information that was collected, the children who took part in the NLSCY were divided into seven age groupings: 0-11 months, 1, 2-3, 4-5, 6-7, 8-9, and 10-11 years.

Sample size was chosen so that there would be a sufficient number of respondents in each age group after five survey cycles to reliably study, at the national level, characteristics with a prevalence of 4 percent across Canada. Furthermore, a sufficient sample in each province and territory was required to produce reliable estimates for all children ages newborn to 11 years, with no age breakdown.⁵

The first round of the NLSCY data collection was completed during the winter and spring of 1994-95. The person most knowledgeable about the child (usually the mother) answered questions relating to the family environment and the child in a personal interview which lasted, on average, 2 hours. Interviews were conducted using Computer Assisted Personal Interview (CAPI). Children aged 10-11 years completed an additional written questionnaire. The Peabody Picture Vocabulary Test (PPVT) and a mathematical test⁶ were administered to children in specific age groups. Also, teachers and principals of "NLSCY children" of school-age were asked to complete an additional written questionnaire. Data from the first wave of cycle one of the NLSCY are now expected to be released in the Fall, 1996.

The cost of the first cycle of the NLSCY, including survey development, field operations and data preparation is estimated at $6 million (spread over a three-year period), which is comparable to the costs of other Statistics Canada national longitudinal surveys.

1.2 Evaluation Description: Purpose, Methods, Constraints

An evaluation of the NLSCY was undertaken to respond to a Treasury Board requirement that CDI programs be evaluated by the end of the fourth year of the five year initiative, to provide input to the overall interdepartmental evaluation of the Child Development Initiative (CDI). The purpose of the NLSCY evaluation is to identify and clarify key issues relating to relevance, rationale, need, development and alternatives of the NLSCY.

The evaluation focused on the anticipated usefulness of the NLSCY, the quality of its survey design, the potential application of survey results, the processes followed in developing the instruments, the potential applications of the NLSCY data, and strategies for dissemination and marketing of the data. Suggestions for improvement, and alternatives for future cycles were also examined.

The evaluation of the NLSCY involved a number of study components and activities. These activities and outputs included:

• A Review of Materials, including program documents, current literature and literature from similar Canadian and international studies (Output: Literature Review: A Working Paper for an Evaluation of the National Longitudinal Survey of Children);

• Key Informant Interviews with 30 persons, including interviews with members of the HRDC/Statistics Canada Project Team, the NLSCY Advisory Group, and the Interdepartmental Consultation Committee, and participants of the Federal/Provincial/Territorial Consultations (Output: Key Informant Interviews: A Working Paper for the Evaluation of the National Longitudinal Survey of Children);

• A Delphi Panel Study of Methodologists, in which the panel was made up of 24 academics knowledgeable in methodological issues, as well as a specific subject area of expertise related to children — psychology, health, education, sociology, and others. The Delphi Panel Study consisted of three rounds of data collection to assess the survey methodology (Output: Delphi Panel Study of Methodologists: A Working Paper for the Evaluation of the National Longitudinal Survey of Children); and

• A Survey of Potential Users of the NLSCY data, from all fields, including government, universities, media, education, advocacy groups, children's service providers, health care providers, and others (n=124 potential users) (Output: Potential Users Survey: A Working Paper for the Evaluation of the National Longitudinal Survey of Children).

As well, a discussion group was held in Toronto on June 8, 1996. Nine experts participated in the discussion, selected from participants in either the Delphi Panel of Methodologists or the Potential User Survey, and representing expertise in developmental psychology, sociology, economics, epidemiology, justice, education, and social work.

Constraints: Because the NLSCY is a long-term project and data from the first cycle of the survey were not available for this evaluation, this study constitutes Phase I of the NLSCY evaluation, examining process, quality, relevance, need and anticipatory usefulness.

Thus, in addition to providing an initial formative evaluation, Phase I will lay the groundwork for a more summative definitive Phase II evaluation of the NLSCY, which is expected to examine the actual use of NLSCY data.

1.3 Key Evaluation Issues

The specific evaluation questions addressed in this study are:

RELEVANCE

1. To what extent are the mandate and objectives of the NLSCY still relevant?

2. Was the NLSCY properly targeted (all children/youth)? Any gaps in coverage re: topic areas?

3. Is there a continuing role for the Federal government to play in collecting data of this nature?

RESULTS

4. To what extent can the creation of the Canada-wide longitudinal and cross-sectional database on children contribute to increased knowledge of: (I) the characteristics of children in Canada? (ii) the processes which shaped their development? and (iii) the development of effective policies and programs for children at risk?

5. Will objectives be met for the cross sectional analysis?

6. To what extent will the NLSCY and associated activities provide new knowledge and increased public awareness regarding conditions of risk faced by children? (Assessment of quality of survey design.)

7. What effects have the activities under NLSCY had on the development of partnerships and cooperation among and with other levels of government, and with the voluntary and private sectors (effectiveness of survey development process)? How is the partnership working out between HRDC and Statistics Canada?

8. To what extent are the various players, stakeholders, and beneficiaries satisfied with the first cycle: (I) processes; and (ii) expected "products" of the NLSCY? What is the overall measure of support for the NLSCY project?

9. What uses (will be? can be?) made of the products from the NLSCY? What will be the end products and data to be released? Feasibility of intended research activity? Has a dissemination strategy been identified?

   ALTERNATIVES/COST-EFFECTIVENESS

10. To what extent does the NLSCY and its associated activities complement/supplement/duplicate that of other Federal/Provincial governments or institutions? Were other existing sources examined prior to NLSCY design?

11. Have appropriate systems been established to coordinate and monitor the activities under NLSCY?

12. Is the NLSCY approach the most cost-effective and appropriate mechanism to produce the information needed to develop programs which would improve the socio-economic well-being for all children and youth in Canada? How can the current mechanism be improved?

1.4 Structure of the Report

This report is comprised of eight sections. Section 1 gives background information on the NLSCY and this evaluation, including the key evaluation questions addressed by this evaluation. Section 2 outlines the evaluation's findings on the relevance of the NLSCY, Section 3 provides the findings relating to the effectiveness of the NLSCY development process and Section 4 presents the findings on the quality and soundness of the NLSCY design. Section 5 describes the results anticipated from the NLSCY while Section 6 discusses the cost-effectiveness of, and alternatives to, the NLSCY and Section 7 examines dissemination strategies. Section 8 provides conclusions for the NLSCY evaluation. All findings throughout the report are related to the specific evaluation questions.

Footnotes

4	For the NLSCY, an economic family is defined as all family members related by blood, marriage, common-law relationship or adoption; foster children are considered to be part of a family.
5	For more detailed information regarding sampling, please refer to: National Longitudinal Survey of Children: Overview of Survey Instruments for 1994-95, Data Collection, Cycle I February 1995: Statistics Canada and Human Resources Development Canada, Catalogue No. 95-02.
6	Statistics Canada has sponsored the development of the Mathematics test for Cycle One and Cycle Two, as well as the collection of data from school principals.