Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

This section presents the issues examined by the formative evaluation of the CESG and the approach used to undertake the evaluation.

3.1 Evaluation Issues

The overall objective of the formative evaluation is to provide reliable information on program relevance, design and delivery. It also considers the early signs of program impacts. Within these areas, the following eight evaluation issues were addressed:

Relevance:

1. Role/Need: What are Canadians' expectations and savings behaviour regarding the PSE of their children? What is the role/importance of the CESG in financing individuals' PSE?

Design/Delivery:

2. Profile of participants: Who directly and indirectly benefits from the CESG?

3. Roles and responsibilities: Are the roles and responsibilities of the various partners clearly stated and carried out?

4. Promotion/Marketing: To what extent is information on the CESG Program reaching Canadians with school-aged children? Are promoters, trustees and the Government of Canada providing information about the CESG that is of sufficient quality and quantity? Is the Government of Canada getting sufficient visibility?

5. Satisfaction with the CESG Program: How satisfied are subscribers, promoters and trustees with various aspects of the program?

6. Constraints: Have any constraints or directives been identified that impinge on the ability of the program to achieve its objectives?

7. Administrative mechanisms: What administrative mechanisms have been put in place to monitor the CESG Program? Are sufficient data being gathered in order to measure short, medium and long-term impacts, and identify a comparison group?

Impacts:

8. Early impacts: To what extent does the CESG Program provide incentives for parents to save for their children's PSE? Is the program attracting savings for a child's PSE that would not have been made otherwise? To what extent does the amount of grant provided meet the needs to prepare for PSE studies?

3.2 Evaluation Methodology

Evaluating programs, such as the CESG, raises a number of challenges. As indicated in the logic model, the CESG is designed to encourage early and sustained participation in RESPs as a way to reduce financial barriers to PSE, increase PSE access and participation, and reduce the financial burden of PSE. Achieving these objectives involves working through groups and individuals beyond the direct influence of the CESG Program. As a result, in individual instances, specific outcomes of the CESG are not easy to track and measure. While these challenges will be more of a consideration for the summative evaluation of the CESG, they are also relevant to the consideration of the early signs of program impacts.

The methodology used to evaluate the CESG recognized these challenges and attempted to address them in a number of ways:

• The evaluation approach emphasized the use of multiple lines of evidence thereby allowing for findings from one approach to be corroborated by findings from other lines of evidence;
• Quantitative and qualitative information was gathered from a range of groups (i.e. RESP subscribers, non-subscribers, promoters/trustees, and informants); and
• A review of program documents, administrative data and the literature on PSE financing and access was undertaken to help develop and inform the methodology and lines of enquiries for the various lines of evidence.

The evaluation approach had certain shortcomings that should be noted. First, inaccuracies in the contact information available for RESP subscribers created difficulties in surveying subscribers. Second the study was not designed to measure program incrementality. Thus, the net impact of CESG on RESP take-up and contributions could not be measured. Third, the small sample size of the survey of promoters/trustees and the informant interviews means that the data/information collected from these sources may not be representative.

Survey of Subscribers

The survey of subscribers gathered information from RESP subscribers on their PSE savings goals and strategies and the role of the CESG in providing an incentive to save for children's future education costs. The survey gathered data on subscribers' socio-demographic characteristics. These survey data elements were supplemented with CESG administrative data. Inaccuracies in contact information available for RESP subscribers made it difficult to develop and conduct the survey of subscribers.

To achieve the desired 2,000 interview completions, a sample of 14,000 RESP subscribers was drawn from the CESG administrative database. Telephone survey interviews were completed with 1,998 subscribers.

Subscribers lost to attrition included those whose telephone number was not in service (1,888) and those who did not respond after repeated attempts (5,217). Based on the functional sample of 9,370, the completion rate was 21 percent,⁴ a rate that is not considered low by standards of financial surveys and surveys conducted for other evaluations. Based on the total number called (11,527, which includes those with telephone numbers not in service), the completion rate would be just 17 percent. Considering a population of some 950,000 subscribers, the sampling error for the survey dataset is ±2 percent. Comparison of the survey results with a database on all subscribers revealed that surveyed subscribers are representative of all subscribers along a number of dimensions, including family structure and income, although they are somewhat older.

Survey of Non-Subscribers

The survey of non-subscribers provided information about the characteristics, attitudes, needs, and savings behaviour of non-RESP subscribers, as well as their reasons for not subscribing to an RESP. It should be noted that the evaluation design did not call for the development of a comparison group using the non-subscribers survey.

In order to achieve the desired 1,000 interview completions, a sample of approximately 17,000 telephone numbers was drawn using a random digit dialing process. Individuals lost to attrition included mostly those whose telephone number was not in service (2,148) and those who did not respond after repeated attempts (2,395). Individuals declared ineligible included those who had RESPs, did not have children or grandchildren less than 18 years of age (4,562) and those eliminated because grandparent and female quotas were filled (437). A total of 1,001 non-subscribers completed telephone interviews.

Based on the functional sample of 14,218, the completion rate was seven percent. Based on the total number called (17,205, including those with numbers not in service), the completion rate would be six percent. Considering cooperative contacts (6,000), the response rate is 42 percent as a proportion of the functional sample. Included in cooperative contacts are those who would have completed an interview but were declared ineligible for the reasons noted. The sampling error for the survey dataset is ±3.1 percent.

Survey of Promoters and Trustees

The survey of promoters and trustees provided information about many aspects of the CESG Program including, the effectiveness of the CESG as a savings incentive, perceptions regarding roles and responsibilities, plus operational, legislative, regulatory and jurisdictional constraints.

Contact information was available for 63 liaison officers of participating promoters and trustees for purposes of the survey. Officers not reached were mainly those with out-of-service telephone numbers. A total of 37 promoters/trustees completed the survey interview out of the functional sample of 47, translating into a completion rate of 79 percent. Based on the total initial sample, the completion rate is 58 percent.

With the small sample size and assuming a population of 63 liaison officers, the sampling error is fairly high at ±10.5 percent. Therefore, the results of this survey must be treated with caution. Moreover, the small sample size means that analysis of survey results by institution type would not have yielded reliable results and therefore was not carried out. Another caution to note is that respondents' knowledge of the CESG and therefore their survey responses may not accurately reflect that of their entire organization.

Analysis of Data from the 1999 Survey of Approaches to Educational Planning (SAEP)

The 1999 SAEP, conducted by Statistics Canada and sponsored by HRDC, collected information on how Canadians prepare their children for PSE. The SAEP data provided the evaluation with the means to profile families, including their savings patterns with respect to RESPs and other investment vehicles. Analysis of the survey data identified the factors that influence the level of RESP savings and that distinguish RESP contributors from non-contributors.

Database Linkages and Analysis

Linking the Longitudinal Analysis Dataset (LAD), based on income tax data, with CESG administrative data provided a socio-demographic profile of subscribers and non-subscribers in 1999. The linked CESG/LAD file was used to profile RESP subscribers, assess CESG's role in financing PSE, and examine early program impacts. A comparison group (non-subscribers) was created to test the influence of variables affecting contributions while controlling for other factors. It should be noted that lower income and single-parent families are over-represented among taxpayers relative to their proportion of all economic families as indicated by SAEP and Census data. This did not affect the use of the data in profiling subscribers and observing patterns over time, but does limit their use in measuring the incidence of RESPs among all Canadians.

Informant Interviews

Semi-structured informant interviews were conducted with eight government representatives (six associated with the program), six researchers/non-government organization representatives, and six financial experts. The 20 interviews conducted gathered information on perceptions, opinions and knowledge of various CESG issues (e.g. operational, legislative, regulatory and jurisdictional constraints, roles and responsibilities, and evidence of short-term impacts).

Document/Literature Review

The document and literature review addressed a variety of evaluation topics including estimates of PSE costs, funding sources available for PSE, and financial assistance provided to parents and students in other countries.

Footnotes

4	The response rate based on the number of "cooperative" contacts (which include those who said they had no RESP and who made appointments that were not followed up on because the survey ended before the appointment date) would be 25 percent.