Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The evaluation questions addressed in this section are the following: (1) What effects have the activities under NLSCY had on the development of partnerships and cooperation among and with other levels of government, and with the voluntary and private sectors (effectiveness of survey development process)? How is the partnership working out between HRDC and Statistics Canada?; (2) Have appropriate systems been established to monitor the activities under NLSCY? and (3) Is there a continuing need for the Federal government to play in collecting data of this nature?

The Statistics Canada/HRDC Project Team developed the NLSCY in consultation with a wide variety of federal, provincial and academic representatives. An Expert Advisory Group of 16 academics was created in order to provide advice on the key design parameters, consultations were held with provincial government representatives, and an interdepartmental consultation committee comprised of representatives of several federal government departments (Health, Secretary of State, Solicitor General, RCMP, Justice, HRDC and others) provided guidance to the Project Team throughout the development process.

3.1 Need for Clearer Goals and Definition of Roles

The Statistics Canada and HRDC Partnership: One key concern in the developmental process was reported to involve the need for more clearly defined goals and roles for the partner agencies. Evidence of problems occurring due to this need for clearer goals/roles were found in the Expert Advisory Committee, the HRDC/Statistics Canada Project Team and the Federal/Territorial consultations.

For example, a number of members of the HRDC/Statistics Canada Project Team were dissatisfied with the initial process of collaboration between the two departments. Key informants from both departments indicated that one of the major problems was the need for better clarification of the roles of the two departments. This concern extended, in a way, to "ownership" of the NLSCY. These concerns were no doubt amplified because of the two unique interests of the two partner agencies: HRDC's role as principal funder providing scientific and intellectual leadership for the study, and Statistics Canada's specific role in the design and conduct of the field research methodology and generation of statistics (and its unique related legal responsibilities under the Statistics Act for the study's statistical appropriateness, for its reliability and for its availability in the public domain).

In this vein, it was suggested, for example, that Statistics Canada staff tended to consider the NLSCY to be "their" survey, and therefore found it difficult to accept the fact that the funding, and therefore the responsibility, for the NLSCY, was mainly provided by HRDC. These differences in perspective seemed to stem primarily from a lack of clarity in the original interdepartmental understandings and objectives for the study.

It is important to note, however, that more recently (since several committees were created to reorganize the structure of the project), the working relationship between HRDC and Statistics Canada has been strengthened and roles have been made clearer.

Some expert advisors¹¹ who were interviewed for the evaluation indicated that the lack of a strong vision of the potential uses of the NLSCY, the policy interests the NLSCY should serve, or the public education purposes the NLSCY should be directed toward, resulted in information gaps in subject areas which were not represented among the expert advisors. Concern was also expressed by one key informant that there was no clear expression of "what the government wanted to know" about children.¹²

Territorial representatives in the Federal-Provincial/Territorial Consultations also found some aspects of the NLSCY process to be frustrating. The Territorial representatives, unlike their Provincial counterparts, were involved in integrating the National Population Health Survey (NPHS) and the NLSCY in order to produce a single survey to be administered in the Territories, meeting the needs of the Northern communities. Territorial representatives reported being frustrated with the lack of communication between those in the federal government working on the Territorial component, the number of Ottawa contacts with which they had to deal, and the lack of a consistent view of the NLSCY among those contacts.

3.2 Time Constraints

Time constraints in the NLSCY developmental process were also noted as a major problem by some key informants. Some HRDC key informants expressed disappointment with the lack of time and money to consult more frequently with the Interdepartmental Consultation Committee and the Provinces/Territories. Some expert advisors also regretted that time constraints meant that older children's issues did not get as much attention in their meetings.

Overall, it was felt by many key informants that tight time lines did not allow those developing the NLSCY to participate in a proper developmental consultation process. Provincial/Territorial informants, for example, felt that the consultations with the provinces were, in reality, only information sessions because of the deadlines faced by the NLSCY staff. Representatives of several provinces reported being frustrated by the fact that most of their suggestions for modifications to the survey design were not implemented due to time constraints.

Tight deadlines may have also contributed to strained project working relations between HRDC and Statistics Canada. Key informants from Statistics Canada, for example, reported that it was difficult to satisfy the communications needs of HRDC when they were already overworked. Tight time schedules also caused problems for the Territorial representatives who faced a key deadline for the integration of the NPHS and the NLSCY just days after they received the final NLSCY questionnaire.

3.3 Satisfaction with the NLSCY Developmental Process

In spite of specific stresses related to interorganizational relations and times pressures, these results — overall satisfaction in the face of many stresses — appear to point not to a contradiction, but rather to a high level of support for the survey and value, which participants rated highly. Key informants reported that they were, for the most part, satisfied with the overall NLSCY development process. For example:

• All key informants who participated in the Interdepartmental Consultation Committee were satisfied (to varying degrees) with this process of consultation and collaboration. In general, it was found that the input provided by the members of the committee was most helpful to the NLSCY staff in the initial development stages for feedback and for prioritizing ideas. Furthermore, members of the Interdepartmental Consultation Committee reported that they appreciated being kept up to date on the NLSCY 'news' by the NLSCY staff.

• HRDC key informants were positive about the input they received from the members of the Expert Advisory Group. Similarly, the experts themselves were satisfied with the consultation process in the development of Cycle I of the NLSCY. However, they also noted suggestions for improvement, which were mentioned above.

• Federal key informants were satisfied with the collaborative process of the Federal/Provincial/Territorial consultations, although Provincial/Territorial representatives were less satisfied.

The HRDC and Statistics Canada managers of the project have established a variety of reporting mechanisms and publications and have provided regular feedback to participants in the NLSCY process. Continued development of these mechanisms, particularly as regards ongoing information for participating federal departments, and the Provinces/Territories should be emphasized.

3.4 Role of the Federal Government

Almost all those consulted in the evaluation agreed that the Canadian government should play a major role in collecting data such as that collected by the NLSCY, and interpreting its policy significance and implications.¹³ A variety of reasons were outlined explaining the importance of the government's role in collecting national longitudinal data on children, including that:

• the NLSCY is national in scale;

• longitudinal studies require secure funding rarely available outside of government; and

• institutions often outlive individuals and government involvement can guarantee the survival of a longitudinal study long after the original principal researchers have changed employment, retired, etc.¹⁴

The Canadian government, therefore, was seen by most evaluation participants as the preferred organization for overseeing a study such as the NLSCY.

Thus, according to Key Informants and Potential Users, the specific role of the Federal government in longitudinal studies such as the NLSCY should generally include funding; leading and coordinating the research; facilitating dissemination of the data; supporting research which is to be done using the data;¹⁵ and implementing relevant policies and processes.

3.5 Summary

The development of the NLSCY was generally regarded as a positive process in spite of many areas for improvement noted. Key Informants were generally satisfied with the development process, even if they reported problems. Notably, a number of Key Informants reported the high level of effort on the part of the NLSCY staff to develop an effective survey. As well, almost all those consulted felt that the federal government's role in collecting data of this nature was appropriate.

Footnotes

11	Some of the same concerns were re-iterated by the discussion group participants. Many participants in the discussion group felt that it was difficult to make important methodological suggestions regarding the NLSCY's content, sampling, and data collection because the articulated goals of the NLSCY are very general.
12	This key informant did not believe that these gaps were the result of NLSCY staff decisions. On the contrary, this informant saw the staff as caught between "a rock and a hard place." They were seen as having to deal with too many agendas that differed between disciplinary objectives, the provinces and the federal government, and between different Federal departments. Frustration was also expressed by some key informants over the fact that responsibilities between HRDC and Statistics Canada were unclear to the Expert Advisory Group, and that because it was not known who would be making the decisions, it was hard (for expert advisors) to know who to present concerns to.
13	Over 99 percent of Potential Users and all of the Key Informants felt that the Federal government should collect such data.
14	The importance of an organization providing stability for a longitudinal survey is clearly outlined through the experience of the National Child Development Study (NCDS) in Britain. The British study had numerous problems with high staff turnover. The study has survived, however, because the same central institution has overseen the study since its inception in 1958.
15	However, some Key Informants suggested that to maximize independence, studies of this nature should be contracted out by the government to non-government or non-profit bodies.