Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

In the late 1990s, there was increasing recognition that the federal government should play a role in helping to deal with the problem of homelessness in Canada. Following consultations with Canadian communities, the federal government launched the National Homelessness Initiative (NHI) in 1999. The NHI is a three-year demonstration initiative that aims to ensure community access to programs, services and support for reducing and alleviating homelessness² in urban and rural regions across all the provinces and territories.

This is the final report of the formative evaluation of four components of the NHI managed by Human Resources Development Canada (HRDC), namely, the Supporting Communities Partnership Initiative (SCPI), the Aboriginal Homelessness component, the Youth Homelessness component and the Research component of the Initiative. Evaluation findings on the overall design and coordination of the NHI are also presented.

1.1 Evaluation Objectives and Scope

The original purpose of the evaluation was to meet Treasury Board requirements to assess progress in program implementation and to draw lessons from the design and delivery of the HRDC-based NHI programs, including identifying areas that are in need of improvement. The evaluation also aimed to satisfy the broader government interest in learning from the community-based delivery model and about how to approach similar initiatives in the future.

It was also important for program management to obtain as much information as possible to assist in decision-making regarding the future of the Initiative. As such, it was decided to expand the scope of the evaluation to examine the available evidence on immediate outcomes of the three program components (SCPI, Aboriginal Homelessness and Youth Homelessness). It was also decided to begin assessing the extent to which the investments made through these programs can be shown to be incremental to what was already taking place in communities.

The NHI was structured as an interdepartmental program with components managed by three different departments. The non-HRDC components are being evaluated separately by the departments responsible for them.³ However, the interdepartmental evaluation steering committee decided that this HRDC evaluation should also examine design and coordination issues for the NHI as a whole.

In addition to the program components, the NHI included a small component devoted to research. The evaluation examines the progress achieved in developing a research agenda on homelessness under the NHI Research component managed by HRDC's National Secretariat on Homelessness (NSH).⁴

And finally, the evaluation presents an analysis of issues related to conducting a future evaluation of the Initiative's long-term objectives of reducing and alleviating homelessness. This includes a discussion of measurement and data collection challenges, such as the lack of reliable baseline information on homelessness in Canada.

1.2 Evaluation Methodology

The evaluation began toward the end of the second year of the program (i. e. November 2001) and was concluded in the fall of 2002.⁵

The evaluation methodology is based primarily on a set of in-depth community case studies that help characterize the implementation, early outcomes and immediate incremental impact of the SCPI, Aboriginal and Youth Homelessness components of the NHI. In all, 20 communities have been the object of case studies. In addition to the case studies, information on all 61 SCPI communities has also been collected through the analysis of program data and documents, and interviews with key stakeholders.

1.2.1 Community Case Studies

A case study approach is particularly useful in exploring start-up challenges in recently-introduced programs. It exposes issues and challenges related to the implementation of a program and it allows evaluators to understand unexpected outcomes. Case studies are also effective in addressing implementation issues where there is flexibility in the program's design.⁶

In this context, the case study approach was chosen for the evaluation of the NHI because it allowed for the in-depth analysis of the experiences of selected communities in implementing this new initiative. These case studies provide important information on the effectiveness of the program's design features, the problems and issues encountered during implementation, and indications of the progress made in particular instances. More specifically, the purpose of the community case studies was to characterize in detail what activities took place as a result of the NHI and to identify the factors that influenced the community's response to the Initiative. The goal was also to assess in those communities the early impacts of the Initiative, including the extent to which SCPI funding built upon pre-existing levels of investment in homelessness (i.e. , the incremental impact of SCPI funding).

The case study sites were selected in such a way as to reflect the variations among the 61 SCPI communities. The variations were along the following lines: the extent of progress of community plans and projects, the size of community, the type of delivery model adopted, the presence of a significant Aboriginal population and the geographical location (see Appendix B for a complete list of criteria and selected case study sites).

This sampling strategy prohibits evaluators from making statistical generalizations of the evaluation findings to all 61 SCPI communities. However, given the similarities in relevant known characteristics between the case study sites and the other SCPI communities, it is reasonable to suggest that the findings from the 20 case studies are likely to be applicable to the majority of SCPI communities. At the same time, there was considerable variation in how communities proceeded with the implementation of the Initiative. Variations existed between communities as well as between provinces and regions of the country, and there was an equally diverse range of issues faced by communities in addressing homelessness. Consequently, it is important to note that any given finding presented below may not apply to all SCPI communities, or even to all case study communities.

Within each community case study site, one or two funded projects were identified for review. The purpose was to document the many kinds of projects undertaken, the extent to which they were being implemented as planned, and any preliminary indications of their benefits to the community. An effort was made to cover a diverse range of projects in terms of investment type, activity, size, scope, approach, and client group. At each site, HRDC staff and community planning leaders were consulted for the selection of projects.

Three research methods were used to obtain information for the case studies:

• Review of documents: community plans; project plans; community and project reports; documents describing community and government investments in homelessness-related facilities and services prior to the NHI, and since the start of the NHI;
• Interviews: community planning leaders; organizations participating in community planning; service delivery organizations, including those not directly involved in NHI projects; clients of the projects selected for project case studies; provincial and municipal officials in relevant policy areas; local federal government officials, including HRDC staff. A total of 478 key informants were interviewed. The great majority of interviews were conducted in person, often on site at shelters or other facilities, or in the offices of government and community planning leaders. (See Appendix C for the categories of informants that were interviewed.)
• Review of data on project operations: descriptive information on project operations; available information on demand and take-up of new/enhanced services.

Analysis of the incremental impact of program funding

There is considerable evidence that prior to the introduction of the NHI, there was activity underway to address homelessness in many of the participating communities. For this evaluation, an assessment of the incrementality of SCPI, Aboriginal and Youth Homelessness funding was conducted in order to understand the magnitude of the federal government investment in relation to existing investments. The goal was to determine whether the program built upon these prior levels of activity, or whether it simply displaced, or substituted for, such activity as the federal initiative was implemented.

By assessing the actual contribution made by SCPI, Aboriginal and Youth Homelessness funding and identifying any displacement effects that have occurred, the results of this analysis will help managers enhance future program design and ensure that adequate program safeguards are put in place.

The analysis of the incremental impact of SCPI, Aboriginal and Youth funding at the national level is based on information from 15 of the 20 communities that served as case studies for the evaluation of the SCPI. This information was obtained from various financial records, as well as through key informant interviews. While the 15 communities are not necessarily representative of the total 61 SCPI communities in every respect, the results provide a perspective on the likely incremental impact to date accompanying this program approach.

Only municipal and provincial expenditures related to homelessness are used in the calculation as they are the most accessible and reliable figures available. Therefore, investments made by the private or voluntary sectors are not included in this analysis. The analysis focuses specifically on expenditures that can be characterized as program and grant/contribution expenditures on homelessness (as opposed to expenditures on social housing, or on-going health and social service programs).

The calculation is performed by averaging the levels of investments in the communities during the two years prior to the federal program (i. e. 1998-1999 and 1999-2000). This average is taken to be the minimum level upon which the SCPI would build in the following three years. This minimum level is then compared to the average levels of actual investments for each of the three SCPI years (i. e. 2000-2001, 2001-2002, 2002-2003). The conclusion is then made as to whether SCPI, Aboriginal and Youth Homelessness funding supplemented or displaced prior levels of investment in homelessness.⁷

1.2.2 Analysis of Project Data from the National Secretariat on Homelessness (NSH)

The NSH has developed and maintained a set of databases to keep track of, among other things, the projects funded and expenditures made through the SCPI, Aboriginal Homelessness and Youth Homelessness. The evaluators analyzed this data to characterize the types of projects being funded and the proportion of total expenditures being allocated to different project types. As well, evaluators created a separate database based on a review of the community plans, which characterized the plans and the communities themselves according to key variables including the substantive nature of the priorities, the number and specificity of the priorities that were set in the plans, the size of the communities, and other variables that allowed for analysis of the linkages between community priorities and actual expenditures.

1.2.3 Review of Community Plans

The community plans were reviewed to analyse their content, to identify characteristics and trends in the plans that might assist in understanding evaluation findings, and to prepare the evaluators to conduct the community case studies.

1.2.4 Interviews with Key Informants at the NSH and Other Federal Departments

At various points throughout the evaluation, interviews were conducted with senior management, heads of sections and staff at the NSH, as well as managers and staff at Canada Mortgage and Housing Corporation (CMHC), Public Works and Government Services Canada (PWGSC) and the Privy Council Office (PCO). These interviews focused on the design and operations of the NHI and its component programs, coordination of the Initiative, the activities being undertaken by the NSH and by HRDC staff in the regions, and respondents' assessments of the progress being made and the factors influencing progress. Most NSH managers were interviewed several times during the evaluation. In addition to these formal interviews, evaluators met frequently with NSH managers and staff to discuss particular issues that arose, to present and discuss preliminary findings at various stages, and to arrange for access to NSH data. Finally, evaluators took part in regular Evaluation Steering Committee meetings, and those meetings were often occasions to discuss evaluation and operational or policy issues with key participants.

1.2.5 Review of Research Literature and Available Evaluation Methods

The evaluation included an assessment of the feasibility of conducting an evaluation of the longer-term impacts of the SCPI at some future date. This involved a review of available research literature on the nature and extent of homelessness, with a focus on potential methods for the evaluation of impacts, and the identification of issues to take into account and potential barriers to overcome. Regional and NSH officials were also consulted with regard to the policy implications of possible evaluation approaches, and the practical considerations involved in conducting an outcomes evaluation.

1.2.6 Note on the Reporting of Findings from Key Informant interviews

Unless otherwise stated, the findings presented in this report represent the views of the majority of the relevant key informants interviewed for the evaluation, ranging from community service providers, municipal government representatives to federal government officials. The strength of the findings that are based on key informant views is found in the convergence of opinions from these diverse sources, and in the fact that the evaluation consulted with a large sample of the individuals and organizations active in addressing homelessness.

1.3 Organization of the Report

Chapter Two provides a brief overview of the National Homelessness Initiative and its various components. The third chapter presents the findings on the overall design and coordination of the NHI. Chapters Four, Five and Six present design- and delivery-related findings for the SCPI, Aboriginal Homelessness and Youth Homelessness components, respectively, while the findings on the progress of the Research component are discussed in Chapter Seven. The eighth chapter presents the evaluation findings on the outcomes to date for the HRDC-managed components. Chapter Nine summarizes the main findings and conclusions from this evaluation, and Chapter Ten presents an overview of the feasibility study on evaluating program impacts in the future.

Footnotes

2	In the context of the NHI, homelessness is defined as living on the street or in temporary shelters.
3	For details, see CMHC, "Shelter Enhancement Program Evaluation," March 2002; CMHC, "Shelter Enhancement Program Evaluation Youth Component," March 2002; CMHC, "Evaluation of the Residential Rehabilitation Assistance Program (RRAP), Rental RRAP, Rooming House RRAP, Conversion RRAP, Homeowner RRAP, RRAP for Persons with Disabilities and the Emergency Repair Program Final Report," November 2002; and CMHC, "Evaluation of the On-reserve Residential Rehabilitation Assistance Program Final Report," March 2003. PWGSC, "Surplus Federal Real Property for Homelessness Initiative Formative Evaluation Report" (pending).
4	See Appendix A for the full list of issues addressed in this evaluation.
5	All quantitative data provided in this report reflects program implementation to July 2002.
6	United States General Accounting Office, Case Study Evaluations, November 1990, Chapter 3.
7	Additional information on the data sources and the method used to calculate the incremental impact of SCPI funding can be found in Appendix E.