Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

This chapter considers the types of evaluation issues that were addressed and the methods used to answer those issues.

2.1 Evaluation Issues

The Terms of Reference specified 34 evaluation questions. Following the general guidelines of the Office of the Controller General, the issues were of four broad types: relevance, program results, achievement of objectives, and possible alternatives.

Program Relevance

Under this category of issues, the main focus is on the continuing relevance of the program in current circumstances. Do the mandate and objectives of the program make sense under current conditions?

Program Results

Here the emphasis is on what CCIF accomplished. How were funds and projects distributed? Were results widely disseminated? Were the findings applied beyond the original project? What lessons were learned?

Achievement of Objectives

The focus under this heading is on the impact CCIF has had. Has the program achieved what was expected?

Program Alternatives

The primary focus here is to determine whether there are better ways of achieving the results. What would happen in the absence of CCIF?

2.2 Overall Conceptual Approach

This evaluation used several different methods to determine the effectiveness of CCIF in achieving its mandate. Although the number and type of methods used in analysis varied across issues, most were investigated using three or more methods. The advantages of using multiple lines of evidence are that each issue is examined from several perspectives and that one can have greater confidence in the reliability and validity of the findings should multiple lines of evidence converge. When different methods yielded inconsistent or puzzling results (a rare occurrence), alternative interpretations were considered.

2.3 Methods

To address the evaluation issues, the evaluation used the following methodologies: a review of the literature; an analysis of administrative data; interviews; case studies; a survey of project sponsors; and a peer review of selected projects.

This section presents an overview of the methods used in the evaluation.

2.3.1 Literature Review

A succinct review of child care issues and initiatives was undertaken during the early stages of the evaluation to provide a theoretical framework for analysis. The review attempted to summarize the literature in three major areas: child care in Canada (including availability, affordability and quality), models of child care, and rationale for government intervention in child care.

2.3.2 Analysis of Administrative Data

Child Care Initiatives Fund (CCIF) data were analyzed to draw a profile of the program and its projects. Data were drawn from the Contribution Information System (CIS) computer files, and covered the period from the inception of the program to the fall of 1994. The main purposes were to determine which types of projects were funded by CCIF up to November 1994, the primary activities of projects, which groups were targeted, the amount and duration of funding, etc.

2.3.3 Survey of Project Sponsors

The survey of representatives from projects funded by CCIF collected the opinions of respondents about the relevance of the CCIF, its effectiveness, impacts and effects, and alternatives. Data from the survey were used to address most of the evaluation issues.

We began the exercise by selecting the sample of project representatives to be surveyed. This was done by reviewing the CIS data set and removing any duplicates. Also, we wanted to limit the number of surveys any single organization had to complete to five so we randomly selected five projects for the few organizations with more than five projects included in the file. This process yielded 479 cases from 393 organizations.

Using the evaluation issues as a guide, the questionnaire was constructed and pre-tested it with seven respondents. Some minor modifications were made in the wake of the pre-tests. The surveys were self-administered.

To enhance response rates, a reminder/thank-you letter was mailed out three weeks after the initial mail-out of the survey. The original closing date for the survey was extended twice because of the low response rate. With the response rate still low, we called every non-respondent and asked them to complete and return the questionnaire.

2.3.4 In-Depth Interviews

The purposes of the interviews were: 1) to gain a better appreciation of how CCIF carried out its activities; 2) to assess qualitatively the extent to which CCIF had achieved its objectives; 3) to learn what could be done to make similar programs more successful; 4) to investigate the strengths and weaknesses of the program and its projects; 5) to grasp the issues that are still outstanding in the child care field today; and 6) to examine what would happen if the federal government withdrew its involvement from child care.

We began by obtaining lists of potential interview subjects from CCIF staff. From an initial list of potential contacts, we and the scientific authority identified 10 staff at CCIF (including one former CCIF consultant), six provincial government representatives, and 10 child care non-government organization (NGO) agents ².

Concurrently, we designed interview guides to govern the interviews. Somewhat different guides were needed to reflect the different perspectives of each of the above groups.

Interviews, which lasted from 90 minutes to three hours, were completed with all but three NGO representatives, one of whom refused to participate, and two others whom we could not contact. Interviews were held in person with respondents in the National Capital Region and by phone with those outside these areas.

2.3.5 Case Studies

The aims of the case studies were: to get a grass roots perspective of CCIF as it operated in the community — local perception of responsiveness to community need, adequacy of finances, and strengths and weaknesses; to gauge uptake — who actually benefited from the project; and to obtain suggestions for improvements/modifications to a program such as CCIF.

The Terms of Reference for the evaluation stipulated that a number of projects considered by CCIF staff to be worthwhile or successful be selected for in-depth study, with a particular focus on innovative models of service, longer term and more expensive projects. CCIF staff were asked to nominate projects meeting these criteria, with the additional consideration that people knowledgeable about the project were still available. Of the 24 projects nominated by CCIF, 12 were selected with the assistance of the scientific authority to represent a balance of priority areas (e.g., special needs, rural, native, etc.) as well as regional and national projects. A list of the projects selected for study is contained in Appendix B of this report.

The field work activities of all case studies comprised five components: preparation work; a review of project files; interviews with managers, and staff; a review of administrative data; and published literature. In addition, the demonstration projects included a site visit.

2.3.6 Peer Review

The general objective of the peer review was to determine whether the reviewed projects contributed to the objectives of CCIF. There were several purposes: to critique the quality of the projects (i.e., methodological rigor); to assess the relevance of the work — how much it added to the knowledge base; to assess the usefulness of the work — potential applications; to determine its impact on the child care field — how it was perceived and received; and to rate the extent of the project's contribution to the understanding of factors involved in quality child care.

The first step in the process was to select appropriate participants, i.e., acknowledged Canadian experts in the field of child care. The list of experts was compiled by Norpark through discussion with the scientific authority and staff at CCIF.

Then we selected the studies to be reviewed. The project selection process began by identifying the research and demonstration projects from the data base. An important aspect of this process was the involvement of CCIF, which was asked to nominate examples of excellent research and demonstration projects. In addition, we wanted to ensure that there was coverage of topics and geographical regions and tended to select projects with larger budgets.

For the selected projects, we obtained the final reports from CCIF. Experts were sent an outline of the project and a copy of the final report, and asked to read and critique the work for quality, relevance, usefulness, impact on the field, and potential applications. They were given a protocol to help them structure their responses.

2.4 Constraints

An important caveat to bear in mind while reading this report is that most of our informants were heavily involved in child care, and have benefited directly or indirectly from CCIF. Many — e.g., CCIF staff, representatives of NGOs funded by CCIF, child care researchers who received CCIF funding — have a vested interest in a continuing federal role in child care. Almost everyone we spoke to rated CCIF very highly in most regards.

Still, many — including CCIF staff — pointed out areas of weakness. Given their stake in child care and in CCIF, we took these criticisms seriously. This is not meant to diminish the very positive feedback we received from all quarters on CCIF. Rather, it recognizes that an evaluation is much more valuable if it tells policy makers what needs improving, and why it is important to improve.

There were few other constraints in carrying out this study. One of the only noteworthy ones was tracking down project sponsors for the purposes of the survey. Many organizations that conducted a CCIF project were set up for the sole purpose of running the specific project and consequently disappeared when the funding ended. All we could do was use the information from the CCIF administrative system (address and phone) to contact project sponsors. This constraint limited our response rate on the survey to about 40%.

Footnotes

2	One was actually a worker at a municipal government.