Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Applied Research Branch would like to thank all those who participated in the evaluation of the National Longitudinal Survey of Children and Youth (NLSCY), the main activity of the "What Works for Children — Information Development Program" component of the Child Development Initiative. We are pleased to see evidence of widespread support for the survey. In general, the evaluation report indicates that the evaluation participants overwhelmingly acknowledge the importance of the NLSCY and the value of the data which it collects.

Positive suggestions were gathered from the user community related to dissemination, user support and facilitating research. These will be used by the NLSCY Project Team to ensure that the survey's full potential is achieved and user requirements are met to the greatest degree possible. The evaluation report documents strong support for the NLSCY and for the activities accomplished to date for the first data collection cycle. However, as with any undertaking the size of the NLSCY, evaluation participants proposed suggestions for improvements to the survey. We would like to respond to a few of these.

Objectives

When the project was conceived, the Treasury Board Directive stated that "social, health, educational and other indicators of well-being will be collected and analysed in order to better identify effective interventions for children at risk." Furthermore, "the research findings will be applicable to the modification of existing and development of new policies for children at risk." To respond to this, the Project Team developed a series of broad objectives and designed the survey as a strategic policy tool encompassing a wide variety of factors known to influence child development. The methodology and technology are in many ways innovative and have been developed in response to the unique characteristics and complexities of this survey. The strategic policy needs which it was designed to serve were not mandated to any particular department or jurisdiction and thus the objectives are purposely non-specific. However, the changing social policy environment and the fact that the NLSCY like other government programs is subject to close budgetary restraint necessitate a continuing close scrutiny of the survey objectives, its collection instruments and its costs in the future.

Design

Participants in the evaluation were highly satisfied with the NLSCY design. However, as with content, the report contains a series of suggestions from these participants about various aspects of design. Suggestions were made that only certain cohorts should be followed longitudinally, that data should be collected at 3-year instead of 2-year intervals, that the sample size in larger provinces should be reduced to pay for increased sample in smaller provinces, that the cross-sectional sample should be eliminated and that certain high risk groups such as Aboriginal peoples and recent immigrants should be oversampled. Later in the report, other ideas are proposed such as topping up the original sample, standardizing measures over the course of the study, ensuring that the same respondent answers on behalf of the children in subsequent cycles and suggesting that the PPVT be extended to all age groups.

These suggestions, although appreciated, are ideas which have been discussed by the NLSCY Project Team during the development of both the first and second collection cycles. It is important to remember that the complex nature of the survey and the competing demands which it strives to fulfill result in a delicate compromise constrained by a fixed budget and a sensitivity to respondent burden. Unfortunately, this compromise cannot respond exactly to the preferences of all users. However, in preparation for future cycles, the survey design will be continually reviewed to ensure that the objectives of the survey can be satisfied given the available budget.

In response to the concern that Aboriginal children living on reserves were not included in the sample, we would note that while the survey sample frame did not include such children, considerable consultation was carried out with the Aboriginal community regarding the feasibility of a parallel survey of Aboriginal children both on and off reserves. Because reaction from the Aboriginal community was mixed and support was not strong for such a survey and also because of the need to reduce expenditures during the period of program review in 1995-96, a decision was made to not proceed with this survey. However, Health Canada is supporting a series of regional health surveys with Aboriginal communities which should provide some information on children.

Survey Content and Measures

The content of the NLSCY was rated very highly by the evaluation participants, although some would have preferred to collect more detailed information on certain topics. The evaluation correctly notes that the NLSCY was designed to provide breadth of issue coverage rather than depth of issue coverage.

NLSCY Project Team members conducted a wide consultation in order to assure the inclusion of appropriate content. However, some evaluation participants offered suggestions to resolve what they considered content gaps. Many of these suggestions were considered during the course of the content development process but they were excluded for a variety of reasons. Appendix A provides a list of the topics proposed by the Evaluation participants and a description of their current status. However, the survey content (as well as design) will be reviewed periodically for future cycles, keeping in mind the need to maintain consistency of content for longitudinal purposes.

There is a suggestion in the report that some of the survey measures have been shortened arbitrarily and will therefore produce questionable results. Evaluation participants proposed that complete measures be re-inserted. It is true that some survey measures have been reduced from their original length. However, the reductions were done by members of the NLSCY Expert Advisory Group using their expertise and the results of other surveys in which the longer versions were applied. Results to date indicate that all of the reduced measures are working quite well.

Development Process

The report states that Provincial/territorial representatives indicated that the development process could have included stronger consultation. In the process of content development, the Project Team met with several hundred people. Among these, valuable comments were received from provincial representatives who had the difficult job of collaborating with colleagues in other provincial departments to produce a summary of recommendations to the Team. Because of the volume and variety of input received, the team relied heavily on the members of the Expert Advisory Group for their knowledge of key issues pertinent to child development that would provide a framework into which comments received from others could be integrated. Although all comments received were taken into consideration, the challenge of the exercise was to reach a consensus which would respond to the majority of the needs of the majority of interested parties in a limited time period.

The survey content development and selection process was a challenging exercise requiring a balancing of the vast data needs of the research communities both internal and external to the federal government. The process was dependent on the availability of existing, reliable and tested measures and was subject to the logistical constraint of conducting a household interview of reasonable length. The requirement to include a broad range of survey content with a tolerable level of response burden required a judicious selection and occasional reduction in the length of some measures.

Potential Use of NLSCY Data

In conclusion, it should be noted that some concern is expressed in the evaluation report about potential obstacles to use of NLSCY data, including its cost, the lack of awareness of the NLSCY by potential users and the lack of technical expertise and resources in the user community. HRDC wishes to encourage and facilitate the use of the NLSCY data. It supports the creation of the Data Liberation Initiative at Statistics Canada which promotes wider and less expensive access to data files such as the NLSCY for academia. At the present time, almost 50 universities are participating in this Initiative.

Information from the first NLSCY collection cycle was released in November 1996 and has been highly publicized in the media. This release included a publication entitled Growing Up in Canada, a collection of articles of original research based on the first NLSCY data and written by members of the NLSCY Expert Advisory Group. Also in the fall of 1996, the Canadian Council on Social Development released a report, based in part on NLSCY data, entitled The Progress of Canada's Children 1996 which drew considerable media attention. An article on the CCSD report in the December 1996 issue of the magazine Canadian Living also highlighted the NLSCY. This comprehensive reporting of the state of well-being of children was only possible because of the NLSCY, and this was acknowledged.

The survey has also been publicized through presentations at workshops and conferences, and more are planned. A major Canadian conference on children in November 1996 entitled Canada's Children, Canada's Future included a workshop presentation on the NLSCY results as well as several references to the value of the survey during different key speeches; this conference was attended by over 1,000 participants.

These reports and activities will be supplemented over time by results emerging from the research program being established for the NLSCY data. It is expected that by providing a range of information options, the individual needs of all in the user community will be largely met.