Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

5.1 New Knowledge

The evaluation question specifically addressed in this section is: To what extent can the creation of the Canada-wide longitudinal and cross-sectional database on children contribute to increased knowledge of: (I) the characteristics of children in Canada? (ii) the processes which shaped their development? and (iii) the development of effective policies and programs for children at risk?

Not surprisingly, the evaluators found that it is generally expected that the NLSCY will generate new knowledge about all aspects of children's lives.⁴⁴ For example, when asked if the creation of the longitudinal data base will contribute to the development of effective policies and programs which will reduce the conditions of risk for children, key informants interviewed by the evaluators agreed unanimously that the kind of data that the NLSCY will provide is fundamental and of the utmost relevance to the understanding of children. Delphi panelists reiterated the same view: 13 out of the 19 panelists who answered the survey question, felt that the NLSCY's potential will be good to excellent in achieving these objectives with its present design.⁴⁵ The most new knowledge expected to be developed by the NLSCY would be by:

• expanding knowledge on known risk factors, in part by confirming some factors as contributing to "at risk children" and negating other factors as "risk" factors;

• identifying factors which reduce or increase risks to children;

• providing rich information on the relationship between risk factors, and the cumulative effects of risk factors; and

• providing important new knowledge on positive factors in the development of children.

One Delphi Panelist summed things up in a general way by emphasizing that research to date has focused on finding the effects of one risk factor at a time on child development. The panelist pointed out, however, that the effects of a combination of risk (and other) factors have yet to be studied. With numerous risk factors being studied in the NLSCY, it can be determined which factors, or combination of factors, pose the most risk. Therefore the NLSCY will allow researchers to evaluate more than one cause at a time. : "In short, we will be able to move away from the single cause mentality and inspect a variety of variables in different contexts."

Another important way the NLSCY is expected to expand existing knowledge is simply by providing the first national and longitudinal information of children in Canada — a benchmark indicator on the status of Canada's children.

Also, 68.2 percent of key informants agreed that the data provided by the NLSCY would contribute to the development of programs and policies to support child development.

5.2 Content and Content Gaps

The evaluation question addressed in this sub-section is: Was the NLSCY properly targeted (all children/youth)? Any gaps in coverage re: topic areas?

Table 5.1
Good-to-Excellent Ratings of the Content of the NLSCY
by the Potential Users and the Delphi Panelists*

The content of the NLSCY was generally rated very highly by the participants of the various components of the evaluation. For example, when key informants were asked to assess the content of the NLSCY they agreed that there was a strong need for the type of information provided the NLSCY (96.6 percent of key informants provided ratings between five and seven where 1=very poor and 7=excellent). Ratings were equally positive by the Potential Users and the Delphi Panelists (see Table 5.1)⁴⁶ , an important finding considering that most of these respondents also had suggestions for improvement to the content.

5.2.1 Breadth vs. Depth

The lack of depth of issue coverage, was identified as a weakness of the NLSCY. However, for the most part, it was acknowledged that a trade-off had to be made between breadth and depth in the NLSCY.

Key informants suggested that since the NLSCY was developed with breadth of issue coverage, rather than depth as a priority, the NLSCY will be useful as a foundation for more in-depth research which will in turn have important strategy implications. And while half of the Delphi Methodologists did not rate NLSCY data as above average as regards depth, their comments on the issue suggested that more depth may not always be desirable.

For example one panelist stated that "the larger the sample the more the breadth, and the more expensive to increase the depth. I do not think more depth will increase significantly the usefulness of the NLSCY." Another participant of the Delphi Study agreed: "unless I'm way off base, the data never will be useful for in-depth psychological interpretations — nor is it intended to be. What it does is steer people towards topics warranting in-depth analyses with other data". Finally another evaluation participant argued that "I tend to think that the breadth of the questions is one of the strong points to the survey".

5.2.2 Information Gaps

Not surprisingly, many of the participants in the various components of the evaluation identified some type of information as needed but not collected by the NLSCY despite the recognition of the need to restrict the length of the questionnaire. Yet some methodologists felt that their suggested additions would add considerable depth without seriously increasingly the length of the survey. The most frequently mentioned of these gaps in content included:⁴⁷

• topics relating to violence and abuse. This would include both spousal and child abuse (verbal and physical), and violence within and outside of the home;

• information about paternal parenting styles;

• detailed information on family finances;

• information on social influences outside of the family, in particular the influence of peer groups;

• emphasis regarding positive influences in the child's life; and

• detailed information on literacy.

Most emphasis was placed on the need to collect information on paternal parenting styles (100 percent of the Delphi panelists felt it was important or very important to assess the role of fathers) and on abuse and violence. Without collecting information on abuse and violence, the NLSCY would be overlooking major risks that confront families and children, according to the Delphi panelists.

Also, abuse has serious personal emotional consequences, as well as health care delivery implications. Ignoring this part of children's lives could have serious consequences for the usefulness of the NLSCY in many areas, including restricting the relevance of the NLSCY to the child protection and justice systems.

Discussion group participants acknowledged, however, that while it was very important to collect information on abuse and violence in the NLSCY, there are also serious ethical and confidentiality concerns in collecting these data. Some suggestions were put forward on how this data could be collected,⁴⁸ but the participants recommended nevertheless that a discussion group take place with experts in the field to consider this matter further.

Paternal parenting styles was seen as an important content gap to fill since very little knowledge is currently available on the impact of fathers on their children's development. Furthermore, it would be interesting to find out if and how fathers' perceptions of their children differ from that of their mothers. As one discussion group panelist summarized: "It would be so unique in developmental psychology to have good data from both parents."⁴⁹

5.3 Cross Sectional Analyses

The evaluation question specifically addressed in this sub-section is: Will objectives be met for the cross-sectional analysis?

The main uses of the cross-sectional data, according to the Potential Users, will be to design new programs and/or improve the delivery of current programs; to develop policies; and to use for education purposes. Other possible uses of the NLSCY Cycle I data that were mentioned included conducting analyses to research further into issues covered by the NLSCY, and to identify targets for new programs.

The evaluation found that the cross-sectional data which will be provided by each cycle of the NLSCY be useful to Potential Users, all of whom felt that they could use the cross-sectional results from Cycle I. Furthermore, the cross-sectional data will allow for the immediate use of the NLSCY data (to develop policies and programs, and to provide current snapshots on the status of Canadian children), while the longitudinal data is being accumulated. Since funding for the NLSCY is dependent on favourable attitudes among governments and the general public as to NLSCY results, immediate usefulness can only be beneficial to the NLSCY. On the other hand, it was suggested by some discussion group participants that the NLSCY could reduce costs (if this is deemed necessary) by not collecting the cross-sectional data. The cross-sectional sample will grow as the study progresses, adding expenses during each successive round of data collection. The panelists felt that if resources were tight, the money spent on collecting cross-sectional data may be better spent reducing attrition in the original sample.

5.4 Potential Uses of the NLSCY

The evaluation question directly addressed in this sub-section is: What uses will be/can be made of the products from the NLSCY? What will be the end products and data to be released? Feasibility of intended research activity? Has a dissemination strategy been identified?

All Potential Users felt that they would use the NLSCY data from Cycle I, while approximately 70 percent of the respondents also felt that they would make use of the longitudinal data which would be available in the future (over 20 percent did not know if they would use the data as of yet). When considering that over 40 percent of the respondents had not heard of the NLSCY before contact from the NLSCY evaluation team, it becomes clear that when informed about NLSCY, the potential users are pleased with the information that the NLSCY will provide to policy makers, academics, the media, and others.

When asked which of the potential outputs of the NLSCY they expected to use, almost 80 percent of the respondents said they planned to use the highlights report from Statistics Canada, while the distillation of results designed to aid children's programs⁵⁰ and the specific reports that will follow the release of the survey were also expected to be very useful (see Table 5.2, next page).

Table 5.2
Types of Outputs from NLSCY Cycle-One to be Used
as Indicated by the Potential Users

Other potential outputs which the respondents mentioned would be useful were data files available to government jurisdictions, and a separate database on the NLSCY's data regarding education. The potential users felt that the NLSCY would be able to be used in a variety of ways (see Table 5.3 below).

Table 5.3
Potential Uses of Cycle-One and Longitudinal Information
as Indicated by the Potential Users

It is interesting to note the identified uses for the longitudinal data will differ somewhat from the uses of the Cycle I data. The NLSCY longitudinal data, according to the survey results, will tend to be used more to identify targets for new programs; to design new programs or improve delivery of existing programs; to develop policies; to conduct additional research into issues covered; and to develop or test academic theories. On the other hand, it is surprising to note that the potential users felt that they would use the Cycle I data more often than the longitudinal data for public education purposes. The use of the data for advocacy support reasons and to link with other data bases for further analysis, according to the results, would remain about the same.

Other uses for the NLSCY Cycle I information that were suggested included to educate lawyers, nurses, doctors and health care professionals; to provide support for continued funding of programs and for the creation of new ones; to help produce research proposals; as a baseline for program evaluation; and to conduct research for feature articles on relevant topics (public education).

Related uses for the NLSCY longitudinal information included the development of outcome measures for program monitoring; as a baseline for program evaluations; to identify trends; and to assess legal strategies and reforms.

The expected use of the NLSCY data suggested by the survey of potential users would be primarily focused on issues affecting children at risk; issues affecting children in general; issues affecting families in general; and on child development (see Table 5.4 below). Other types of issues that the NLSCY data use would focus on, according to the potential users included: leisure and recreation; literacy; child custody; and child care.

Table 5.4
Focuses of the Potential Uses of the NLSCY
as Indicated by the Potential Users

Footnotes

44	For more information on specific areas of new knowledge which were mentioned by the Potential Users and the Delphi Panelists, please refer to the Working Reports on the findings from these two evaluation components.
45	One of the objectives of the NLSCY is to develop information for policy analysis and program development on critical factors affecting the development of children in Canada.
46	Please note that Potential Users and Delphi Panelists rated the need for the type of information provided by the NLSCY on a five point scale, and not the 7 point scale which the key informants used.
47	For more information on these gaps, or on the less frequently mentioned gaps, please refer to the Working Reports for the Key Informant Interviews, the Potential Users Survey, and the Delphi Panel Study of Methodologists.
48	Suggestions included that: (1) the NLSCY survey respondents, after they have reached the age of 18, are asked about abusive situations in their home while they were growing up. It was also suggested that this would be a good time to collect information about abuse from family, etc.; (2) the NLSCY ask teachers whether or not they believe the child is being abused or whether or not the parents have been reported for child abuse in the past; and (3) the NLSCY ask questions relating to abuse in a manner that does not seem offensive. In this vein, questions on more positive indicators of parenting, parenting style and child well-being could be practical alternative approaches for the NLSCY.
49	Collecting data from both mothers and fathers may be very expensive, however. Therefore the participants suggested economical ways that this data could be collected: (1) Interview mothers and fathers (separately) in a subsample of the NLSCY children; (2) Collect data from both parents at the same time. This may take the format of the interviewer asking the parents questions, and having each parent answer the questions simultaneously on individual answer sheets; and (3) Ask the mothers (who are normally the person most knowledgeable about the child) about the parenting styles of the fathers.
50	This could perhaps take the format of a National Advocacy Group analyzing the NLSCY data and producing practical programs from the findings.