Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

1. To what extent are the mandate and objectives of the NLSCY relevant?

The relevance of the NLSCY was clearly indicated throughout the evaluation: data from all research components of the evaluation point to the fact that the NLSCY will be a valuable tool for future research and that it will contribute to the increased knowledge of the characteristics of Canadian children on a national scale. Furthermore, the relevance of the NLSCY is expected to grow over time as the longitudinal data accumulates. It was also pointed out that the NLSCY data will always be relevant because it relates to enduring health and social issues, and because it is based on children.

2. Was the NLSCY properly targeted (all children/youth)? Are there any gaps in coverage regarding topic areas?

The evaluators concluded that the NLSCY was properly targeted toward all children and youth, though improvements could have been made by collecting more information for Aboriginal and immigrant children. Participants of the evaluation also emphasized that the utility of the data will be maximized if the NLSCY follows the children into adulthood, perhaps into their late twenties.

A variety of content gaps were noted by the evaluation participants, as well as a number of suggested improvements to the content of specific instruments. The topic areas where gaps were listed most often included: information on sexual, verbal and physical abuse; poverty; literacy-positive influences in the child's life; and social influences in the child's life. The current lack of information on paternal parenting styles was also emphasized as a major gap in information in the NLSCY. Some other gaps were also noted.

However, it may be that this is inevitable if viewed in the light of the NLSCY's need to focus the survey, reduce the length of the survey questionnaire, and meet tight scheduling and budgeting requirements. Furthermore, the NLSCY was designed to address a broad range of issues rather than a few issues in-depth. Therefore, many of the mentioned gaps in NLSCY coverage occur because of the study's focus on providing a broad range of indicators related to risk factors affecting children.

3. Is there a continuing role for the Federal government to play in collecting data of this nature?

The evaluation highlights a variety of reasons why the Federal government should play a major role in collecting data of the nature of the NLSCY's. First, longitudinal studies require secure funding throughout the study in order to guarantee longevity of the research. Such monetary resources are rarely available outside of government. Second, government can guarantee the institutional survival of a longitudinal study even after the original principal researchers have changed employment, retired, etc. Thirdly, it is more practical for a national organization, such as the federal government, to conduct national studies than other smaller or regional organizations. Finally, the Federal government can play a unique role in: (i) coordinating the complex data collection, (ii) aiding the policy interpretation and application of such data for reducing or preventing conditions of risk to children's well-being; and (iii) in identifying the further implications for related issues in societal development.

4. To what extent can the creation of the Canada-wide longitudinal and cross-sectional database on children contribute to increased knowledge of: (i) the characteristics of children in Canada? (ii) the processes which shaped their development? and (iii) the development of effective policies and programs for children at risk?

The NLSCY will provide a wealth of new knowledge on the development and characteristics of children in Canada. It was pointed out by many of the evaluation's respondents that the NLSCY is providing new knowledge simply by the fact that it is the first national longitudinal study of children in Canada. However, the most new knowledge collected by the NLSCY will come from determining the various risk factors in the development of children, and how those risk factors interrelate. Until now, research has linked certain risk factors to development, but a multitude of risk factors have not yet been incorporated into a single study in order to provide researchers with perspective on the relative importance of each risk factor within a multitude of factors. The NLSCY is the first study in Canada to do this.

5. Will objectives be met for the cross-sectional analysis?

The collection of cross-sectional data will allow for the immediate use of the NLSCY data (to develop policies and programs, and to provide current snapshots on the status of Canadian children), while the longitudinal data is being accumulated. The collection of cross-sectional data from children who are not to be followed longitudinally may not provide enough benefits in order to justify the costs, however. The cross-sectional sample will grow as the study progresses, reflecting an additional expense during each round of data collection. Some of the evaluation's participants felt that the money could be better spent to strengthen the NLSCY in other ways.

6. To what extent will the NLSCY and associated activities provide new knowledge and increased public awareness regarding conditions of risk faced by children? (Assessment of quality of survey design.)

The NLSCY survey design was highly rated consistently throughout the evaluation. Overall soundness of survey design, and almost all aspects of sampling, content and survey instruments were highly rated by all of the evaluation's methodological panelists. One area in which the NLSCY was not highly rated by all, however, was the study's breadth rather than depth of information gathered. However, for the most part it was acknowledged that a trade-off had to be made between breadth and depth in the NLSCY content.

It was often suggested throughout the evaluation that the NLSCY would benefit from larger sample sizes in the smaller provinces. Larger sample sizes in the smaller provinces would allow for greater policy and program use at the provincial level, and would also allow for comparisons of the differing social policies and their effects on children.

Almost everyone who praised the NLSCY also had suggestions for improvements. Evaluation participants felt that the most important suggested improvement to the NLSCY was to verify the adequacy of the shortened measures such as the psycho-social measures. Concern was expressed that the psycho-social measures may have been shortened arbitrarily and because of this, the usefulness of the initial data collected by the NLSCY may have been reduced. The inclusion of full measures where possible was also strongly suggested.

Some other suggestions which were considered important by the majority of the evaluation's participants, included filling certain content gaps (100 percent of Delphi Panel Methodologists agreed that it is important to collect data on paternal parenting styles) and including literacy tests in the NLSCY. Furthermore it was suggested that certain tests such as the PPVT should be expanded to all age groups so that researchers are able to compare and contrast test results from data collection cycle to cycle.

Suggestions were put forward by evaluation participants as regards the need for precautions which must be taken to reduce attrition in future rounds of the NLSCY. It was suggested, for example, that greeting cards or some other annual mailing to families in the sample could help reduce attrition rates, and that as much information as possible, including names, addresses and phone numbers of friends, family and neighbours be collected to help locate respondents in the future. The NLSCY is already following many of these suggestions.

Further suggestions include standardizing the measures over the course of the study to ensure comparability of the data between data collections and ensuring that the person answering the questions on behalf of the child generally is the same throughout the study.

7. What effects have the development of partnerships and cooperation among and with other levels of government, and with the voluntary and private sectors (effectiveness of survey development process)?

The development of the NLSCY was generally a positive process. Key informants, for the most part, were very satisfied with the development process, although the Provinces/ Territories would have preferred a stronger voice in the consultations, and the implementation of the NLSCY reportedly had problems due to insufficiently clear goals and roles for HRDC and Statistics Canada, and tight time frames. It was noted that specific efforts have been undertaken to improve the working relations between Statistics Canada and HRDC, and reports suggest that the efforts have been successful and that working relations have become smoother over the past year.

8. To what extent are the various players, stakeholders, and beneficiaries satisfied with the first cycle: (i) processes; and (ii) expected products of the NLSCY? What is the overall measure of support for the NLSCY project?

In the national survey of potential users of the NLSCY, all respondents felt that they would use the NLSCY data from Cycle One, while approximately 70 percent of the respondents also felt that they would make use of the longitudinal data which would be available in the future. Another 20 percent did not yet know whether or not they would use the data. There is strong support for the NLSCY from those who are aware of it. Delphi panelists and others emphasized the importance of Canadian longitudinal data for informing policy and developing effective programs.

9. What uses will be/can be made of the products from the NLSCY? What will be the end products and data to be released? Feasibility of intended research activity? Has a dissemination strategy been identified?

In the survey of potential users, the highest rated potential use of the NLSCY longitudinal and cross-sectional data was to design and improve delivery of programs for children. Other potential uses that received high ratings included: (a) to develop policies; (b) to conduct analyses to research further into issues covered; and (c) to identify targets for new programs. Most potential users felt that the longitudinal data would be more useful than the cross-sectional data: to identify targets for new programs; to design new programs; to improve delivery of current programs; to develop policies; and to conduct further research into issues covered by the NLSCY.

According to potential users, the most useful potential outputs of the NLSCY are expected to be the highlights report by Statistics Canada, distillation of results designed to aid children's programs and the specific reports that will follow the release of the survey. The methods of disseminating the data which were most often suggested includes through government publications, news releases and through conferences and workshops. A large number of the evaluation participants also suggested using the Internet to provide information on the NLSCY and on how to order the NLSCY data.

Throughout the evaluation the importance of using the media to disseminate findings, and to advertise the availability of the data was emphasized. Using the media will not only increase usage, which in turn will make the data set more valuable, but will also help reduce attrition (people will be proud to take part in a survey which is widely talked about). A media strategy for dissemination is considered by many as the most important aspect of the NLSCY dissemination strategy.

Concern was expressed that the cost of the database will inhibit the use of the data by some potential users. While it was noted that the Data Liberation Initiative may result in the data base being more affordable for academics, it was suggested that strategies to provide the same advantage to the non-profit and advocacy sectors be considered so that the data may be used to its maximum potential.

Other suggestions for the NLSCY's dissemination strategy included establishing a User Service/Group to encourage NLSCY data users to team up for collaborative multidisciplinary research and to aid each other (i.e. self help group). The group could also provide training sessions on how to use the data and answer researchers' questions.

It should be noted that as of writing the study data have still not been released for public use. As many potential users will link their analysis plans to other time-tables, it may be useful for release dates for future Cycles to be more certain.

10. To what extent does the NLSCY and its associated activities complement/ supplement/duplicate that of other Federal/Provincial governments or institutions? Were other existing sources examined prior to NLSCY design?

A review of Canadian research on children shows that the NLSCY does not duplicate any other Canadian study currently being undertaken or having been recently completed. Other Canadian research tends to have smaller samples, to be regionally, rather than nationally based, and to be topic specific. The NLSCY is unique in its combined characteristics of population coverage (national sample), broad topic coverage, and longitudinal methodology. No other Canadian study provides such broad longitudinal and cross-sectional data on children.

11. Have appropriate systems been established to coordinate and monitor the activities under the NLSCY?

The HRDC and Statistics Canada managers of the project have established a variety of reporting mechanisms and publications and have provided regular feedback to participants in the NLSCY process. Continued development of these mechanisms, particularly as regards ongoing information for participating federal departments, and the Provinces/Territories should be emphasized.

This coordinating monitoring process could also be extended through the establishment of a mechanism to monitor and report on uses of the NLSCY data, once it is released (for example follow-ups on users). This will be essential to minimize duplications of efforts, and to document uses and impacts of the survey.

12. Is the NLSCY approach the most cost-effective and appropriate mechanism to produce the information needed to develop programs which would improve the socio-economic well-being for all children and youth in Canada? How can the current mechanism be improved?

Existing research indicates that the benefits of longitudinal surveys outweigh their costs, and thus illustrate the cost-effectiveness of NLSCY. When the cost of information is compared to the benefits of their data, in providing input for the development of appropriate policies and programs, it appears that funding for longitudinal surveys of children such as the NLSCY is an excellent investment. Furthermore, causal analysis can be more effectively undertaken using longitudinal data, meaning that it makes more sense to spend the money to get highly useful (longitudinal) data than to spend similar amounts of money for a less conclusive answer. Many alternatives to the current format of the NLSCY were suggested throughout the evaluation, and included, for example, collecting information from the child's school records rather than from teacher questionnaires; lowering the sample size in larger provinces and increase them in smaller provinces; and collecting data at three year intervals rather than two year intervals.

The NLSCY was found to be highly relevant and an appropriate initiative of the Federal Government. As well, it appears to be a cost-effective and significant undertaking for the development of new information on Canadian children which does not duplicate any other Canadian study. The NLSCY will provide a wealth of new knowledge on the development and characteristics of children in Canada, according to a wide range of subject matter specialists and methodologists who participated in the evaluation. The NLSCY survey design was highly rated throughout the evaluation. Overall soundness of the survey design and most aspects of sampling, content and survey instruments were also highly rated by the evaluation's methodological Delphi panelists. Some areas for improvement were suggested, but generally the NLSCY and its methodology were strongly supported by methodologists and potential users from across Canada.