Evaluation of the National Child Benefit Initiative: Synthesis Report - February 2005
| Previous | Contents | Next |
4. Evaluation of the NCB Initiative — The Program Evaluation Approach
The evaluation process was developed to examine the overall impacts of the NCB Initiative and reflects Federal/Provincial/Territorial commitment to joint accountability. The specific arrangements set in place are outlined in Exhibit 6.
Exhibit 6
NCB Initiative Federal/Provincial/Territorial Evaluation Arrangements
Accountability Framework
- Transparent — annual progress reporting
- Reporting on Investments and Reinvestments
- Focus on performance outcomes
- Joint partnership approach to periodic program evaluation of outcomes and effects (based on the three main objectives of the NCB Initiative)
Joint evaluation working group
- Jointly co-ordinated by federal-provincial/territorial co-chairs
- Includes all provinces and territories (Quebec participates as an observer) and includes federal representation from the Department of Finance, the Canada Revenue Agency and Social Development Canada (Indian and Northern Affairs Canada also participates as an observer)
Methodology
- Developed on the basis of an 'evaluability assessment' and a consultation with experts on the evaluability assessment report
- Derived through a consensus of all members of the federal/provincial/territorial evaluation working group and informed by external peer review
- Based on multiple lines of evidence
- Implemented through external contracts with expert consultants and use of peer reviews with several of the studies
The current evaluation focuses on the outcomes resulting from the activities of all jurisdictions, with the exception of First Nations (which are being evaluated separately). The evaluation timeframe is the first three years (1998-2001) of the operation of the NCB Initiative. In a number of cases, analyses in the years immediately preceding the introduction of the Initiative provide a clear context and contrast to the evaluation findings.
Due to anticipated difficulties in assessing the NCB, the evaluation approach focused on the development of multiple lines of evidence in order to provide corroboration of findings from multiple information sources, each partial in coverage, which can serve to strengthen the findings.8
The multiple lines of evidence used in the current evaluation were produced from four separate contracts with external consultants as well as from internal work conducted by the then HRDC. Summary descriptions of the technical reports issued by these contracts and internal work are presented in Annex 19. The following table presents the key methodologies used by each of the documents described in Annex 1.
| Methodologies | Annex 1 Documents |
| Time Series analyses of NCB impact on social assistance caseloads | Documents 1, 2 and 3 |
| Survival analyses of NCB impact on social assistance caseloads | Documents 18, 19 and 20 |
| Gross impact analysis of the NCB Supplement on poverty reduction | Documents 4, 5 |
| Gross impact analysis of the NCB on reducing the welfare wall | Document 21 |
| Net impact analyses of the NCB on poverty reduction and labour supply | Documents 7 and 9 |
| Survey of NCB Supplement recipients' views on the NCB | Document 8 |
| Focus Groups with NCB Supplement recipients | Document 11 |
| Surveys with Managers of the NCB programs | Documents 10, 13, 15, 16, 17 |
| Case Studies of NCB programs | Documents 15, 16, 17 |
| Literature Reviews | Documents 6, 13 and 14 |
| Cost effectiveness analysis of the NCB | Document 12 |
8 As an example of this, the Assistant Auditor General of Canada in a letter dated July 22, 2002 to the Director General of Program Evaluation, Human Resources Development Canada, stated "We note (the) strong support for multiple lines of evidence. We have always been supportive of this type of approach because, when a number of concurrent methodologies are used, they tend to corroborate each other while compensating for their respective limitations."
9 Copies of the background documentation are available upon request by calling 1-888-440-4080; writing to Audit and Evaluation, Social Development Canada, 355 North River Road, Ottawa, Ontario K1A 0L1; or faxing a request to (613) 941-0660.
| Previous | Contents | Next |