Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The present evaluation has focused primarily on program design and implementation issues. Given the short program period, it included an assessment of only some of the immediate outcomes of the Initiative. While it was not possible to assess longer-term impacts on communities and on homeless individuals, it is clear that such an assessment would be needed should the federal initiative be renewed. This section explores some of the challenges that a future evaluation of outcomes would face. It also discusses potential evaluation issues, approaches and design options.

10.1 Anticipated Evaluation Issues

The issues that a future evaluation would address are drawn from the long-term policy objectives of the NHI and the SCPI. The issues identified relate to the following four areas: 1) community capacity building and sustainability; 2) incremental and leveraging impacts of federal funding; 3) impact on homeless individuals; and 4) progress of communities in reducing homelessness. These issues are discussed below, along with evaluation methods, potential data sources, and anticipated challenges.

10.1.1 Community Capacity and Sustainability

One of the policy objectives of the NHI is to help communities develop sustained capacity to address homelessness. It will therefore be important to develop appropriate methods to measure community capacity and sustainability, and to determine the extent to which the NHI has contributed to their development.

Through the present evaluation, a number of indicators of community capacity have already been identified. These include the extent to which community-based planning and decision-making processes have been put in place; increases in partnerships, service coordination, and community awareness; and mobilization of service providers, governments and other stakeholders. One of the challenges in evaluating the NHI's impact in terms of sustained community capacity will be to refine measures of community capacity.

New indicators

In the literature on the well-being of communities, the concept of "social capital" is often used in place of "community capacity" to try to explain differences in well-being between communities. The extent of civic involvement in communities and the strength of civic associations are two frequently used measures of social capital. These in turn are measured by such factors as voter turnout, memberships in clubs and other organizations, and trust in public institutions.

It is possible to use these concepts to examine the extent to which the development of associations related to homelessness, and the extent of public involvement on homelessness, has changed over time. Defining a concept such as "social capital" or "community capacity" in relation to homelessness involves attaching relative values to community assets and liabilities, and there are bound to be disagreements as to how this is done. However, the NHI has developed some operational definitions that serve as a starting point. The community plans themselves can be seen as an important asset, and their quality and support in the community can be measured in various ways. The extent to which the planning process has helped establish new community planning and decision-making structures that are sustained over time can be measured once those terms are clarified. Other measurable assets could include the number of agencies serving homeless people or participating on homelessness coalitions; and the number of staff and volunteers working within those agencies on the problem.

Sustainability

It will be important to measure the sustainability of homelessness-related activities in communities. Sustainability can be seen as the degree of stability of community-based associations working to address homelessness. It may also be important to measure levels of public involvement, relative to the extent of the homelessness problem (since public involvement might be expected to diminish as the problem is reduced), as a related indicator of sustainability.

Attribution

It will be necessary to examine the extent to which change in community capacity over time can be attributed to the investments made by the NHI. It may not be necessary to measure the NHI's contribution in absolute terms, but the Initiative's contribution should be measured relative to inputs from other sources, taking into account qualitative assessments of their relative value.

10.1.2 Incrementality and Leveraging

Based on data collected from 15 case study communities, it has been determined that the national impact of federal SCPI funding is fully incremental. This indicates that all of SCPI funding added to, rather than displaced, municipal and provincial investments in homelessness that existed in the community prior to the implementation of the Initiative.

Given that the current incrementality picture is based on only governmental investments, a future outcomes evaluation should provide a more complete picture of the investment levels before and during the program period by including non-governmental investments in the calculation.

In addition to demonstrating the full incrementality of federal SCPI contributions, the figures from the 15 communities also indicated that further additional municipal/ provincial investments have occurred. In other words, both provincial and municipal governments invested more funds in homelessness-related activities during the SCPI period than they did in the pre-SCPI period. The initial analysis of incrementality, however, cannot confirm whether those additional investments were actually leveraged by the federal Initiative, i.e. that the additional investments were attracted as a direct result of the program, or whether they would have occurred regardless of the SCPI. It is possible that the SCPI may have had a leveraging effect. The implication is that further evaluation work should draw a more complete picture of the full impact of SCPI funding on communities by including an assessment of the leveraging effect of the Initiative.

10.1.3 Impacts of Specific Projects/Programs on Individuals

An outcomes evaluation of the NHI will focus primarily on assessing the extent to which, and how, homeless and at-risk people have benefited from facilities and services funded through the Initiative. This could include focusing on the outcomes of specific projects and programs, for example, by measuring outcomes for individuals as one indicator of overall progress. Additionally, the evaluation could test assumptions about the continuum of supports approach and identify effective approaches with a view to planning future interventions.

Most evaluation research on homelessness focuses on the evaluation of specific programs, rather than a broad multi-program initiative. This has the obvious advantage of making the scope of the research narrower, and making the attribution of benefits more feasible. A future outcomes evaluation of the NHI could incorporate the evaluation of a sample of projects of various types, across a sample of communities. Project/program selection would be based on representativeness of communities and investments, and on the availability of reliable information. The purpose of this type of research would be to examine improvements for individual clients as a result of the programs that are being funded through the NHI, and to assess the sustainability of progress over time.

It is apparent from homelessness research that program participants tend to vary greatly in personal histories and characteristics. Consequently, measures of "success" need to be designed carefully. For an evaluation of a federal homelessness initiative, it will be important to focus on measures that are meaningful to a range of clients and that pertain directly to homelessness, rather than to the complexities of broader social problems or psycho-social conditions. Measures such as increasing independence of living circumstances, or stability in maintaining a housing arrangement, might be appropriate types of measures that could apply across a range of program types.

Evaluations of programs for homeless people typically involve multiple lines of evidence, including quasi-experimental quantitative approaches and qualitative methods that assist in the interpretation of quantitative findings. Experimental designs are theoretically ideal in that they present the possibility of more direct attribution of benefits. As in many social service environments, however, ethical considerations would mitigate against withholding services to some individuals in order to establish a control group.

It would also be possible to conduct a "pre/post" analysis, examining changes over time with individual clients, provided that the measures pertained directly to homelessness. These, too, must be carefully planned and pre-tested, however, because of the challenges of tracking homeless people, and the increased risk that non-program influences may be responsible for changes identified over the period of study.

Homelessness interventions often work in close connection with other interventions. Emergency shelters or transitional housing facilities often have programs such as addictions treatment, psycho-social counselling, life skills training and other services that help clients move out of the shelter system and into more independent living situations. Jointly evaluating a group of interventions that share clientele can be an effective way to determine what works best under which circumstances, and to identify factors that contribute to change in individual clients. This approach allows for the study of different combinations of interventions within one research strategy. Such research is more complex and expensive and requires considerable planning and pre-testing. In the context of the NHI, however, where new and enhanced services are being introduced into patterns of existing services, this approach would be appropriate for both research and program assessment purposes.

A challenge in undertaking research is the capacity of service providers to collect data of sufficient quality for analysis, and the inevitable tension between research requirements and service and administrative considerations. For example, intake forms for shelters or specific services are usually limited in scope, in part because clients may be reluctant to provide information, but also because staff are concerned about privacy issues and building the client-staff relationship. On the other hand, the intake process is the most obvious point at which to collect critical "pre-program" information, without which evaluation is difficult. The Homeless Individuals and Families Information System (HIFIS) offers an opportunity to establish some national standards for the recording of intake data, service provision, program participation and exit data, and the system has the potential for expansion beyond its current focus on shelters.

10.1.4 Impacts of Community Efforts as a Whole

The NHI emphasizes community-driven planning to address homelessness, and NHI funds are being spent mainly on facilities and services across the continuum of supports. It would be of interest in evaluating NHI outcomes to broaden the perspective from individual interventions and groups of collaborative interventions, to look at how communities as a whole are progressing. Such research would focus on indicators such as:

• numbers of homeless people;
• numbers of people moving from the shelter system to transitional and supportive housing and then on to more independent living situations;
• numbers of new clients at emergency shelters;
• the role of mainstream programs (social services, health, housing, justice) in preventing/alleviating homelessness, and the interaction between homeless service providers and mainstream agencies; and
• other such measures of the extent to which the system of services for homeless people is helping to reduce the prevalence of homelessness (as opposed to the impacts of programs on specific individuals).

The research literature in Canada and the U. S. has documented the challenges in counting homeless people, but there are well-accepted methods available, including methods for "street counts" to include those not making use of shelters. These methods may not be perfect, but they would provide a reliable indication of the overall prevalence of homelessness. Initial research through the NHI has already begun to adapt some approaches to the Canadian context in some communities. Other studies focus more on housing availability, evictions, applications and take-up at rent banks and other measures related to the "entry point" into homelessness.

In the case of the NHI, the selection of communities for the research would need to take into account the objective of obtaining results that would be generalizable nationally. Given the range of circumstances even within the current 61 SCPI communities, sampling would have to be carefully conducted to take into account key criteria such as community size, geographic distribution, maturity of community planning, and the state of homelessness services prior to the federal initiative and at the time of the research.

A drawback to these macro-level studies is that they do not provide information that can establish a causal link between changes in homeless rates, for example, and the NHI. To increase the reliability of the chosen indicators as measures of success for the NHI, evaluators would need to develop an approach that recognized local economic factors and the potential impact of non-NHI related government policies and programs. Furthermore, many community interventions not funded or only partially funded through the NHI will have had an impact on homelessness rates. Attributing observed changes to particular interventions will inevitably be a subjective process at this macro level of analysis.

10.1.5 Impacts of Research and Communications

It will be important to assess the extent to which a future initiative will have contributed to the broadening of knowledge about homelessness through research, and the communication of that knowledge to increase awareness across the country. Areas of interest relating to the impact of research include the research strategy adopted and the specific research projects; the quality of the research conducted; and the extent to which key research questions are answered by the research findings.

With regard to communications, the evaluation would focus on the following measures:

• activities undertaken to compile, organize and disseminate research findings;
• the reach of those dissemination efforts in terms of communities and target audiences;
• the extent to which information has been made accessible to its target audiences; and
• the extent to which knowledge from the research has been integrated into planning, resource allocation, and service methods for homeless people.

The evaluation may also wish to assess the extent to which the general public, or specific groups in Canada, are more aware of homelessness than they were prior to the Initiative.

Evaluation methods to address research issues run a wide gamut, from simple reviews of activities and strategies, to peer reviews of research reports, to content analyses of research findings in the context of key research questions. On the communications side, research would likely focus on surveys of potential users of the research to assess awareness of findings and to examine how the information was accessed. It could also involve a review of community plans, projects and programs to assess whether they reflect research findings, accompanied by qualitative research to determine the extent to which the research was taken into account. Public opinion survey research may also be required.

10.2 Options for Future Evaluation

Based on the evaluation issues and possible approaches outlined above, three options for a future evaluation of NHI outcomes are proposed. The first option presents a basic evaluation model, involving the least expense and effort, and providing a basic amount of data. The second builds on the first model, and outlines a more elaborate evaluation approach. This second model requires more resources, but also provides stronger data upon which to base future homelessness program strategies and draw conclusions about the link between observed benefits and the NHI. Likewise, the third evaluation option builds on the first two models, and provides even more extensive data, with a corresponding increase in required resources. Within each option, there is a considerable range as to the number of programs and communities that could be involved.

Option 1 - Focus on investments, community capacity and research

This option builds on the current evaluation, but would provide substantially more detail on the community capacity outcomes of the NHI, and the extent to which communities' allocation of investments followed the continuum of supports approach. It would also strengthen the analysis of incremental impacts, and include an analysis of the leveraging effects of the NHI. Finally, it would allow the evaluators to assess the extent to which any future research program has contributed to increased knowledge and awareness. It would not attempt to measure the impacts of expenditures on homeless individuals. It would be the least expensive and labour-intensive of the three options. It would require some design work prior to the evaluation, but not an intensive period of research design and pre-testing. The methods would include:

• Qualitative assessment of progress in reviewing and updating community plans, and in community planning and decision-making processes;
• Analysis of the evolution of community plans over time as a measure of progress in addressing priorities;
• Analysis of changes in community capacity over time based on specific measures of capacity;
• Analysis of project expenditures against community priorities;
• Analysis of community priorities and expenditures in addressing the continuum of supports;
• Analysis of the incremental nature of federal investments based on a national set of definitions and data collection methods;
• Analysis of the sustainability of investments, and the amount of leveraged funds per NHI-funded project;
• Analysis of research results and peer review to assess the research quality;
• Survey, qualitative review and key informant interviews to assess awareness and use of research findings.

Option 2 -Focus on macro-level measures of homelessness

The second option would incorporate the analyses in Option 1, but would have as a primary emphasis a macro-level assessment of the impacts of the NHI on the prevalence of homelessness in participating communities. It would involve community-wide assessments of change over time in the number of homeless people and the types of people who are homeless, and targeted assessments of selected sub-populations such as Aboriginal people, youth and (in some communities) immigrants. It would require at least one year of research design and planning and substantial pre-testing in several communities. It would need to be undertaken in a sizable sample of participating communities, including a mix of communities by size and by stage of progress in community planning and implementation. This option would be substantially more expensive than Option 1 but would provide the first national-level data on the extent of homelessness and progress in reducing the prevalence of homelessness. Attribution of any progress to the federal initiative would be examined qualitatively, relying on the assessments of key informants and the prima facie evidence of new facilities and services intended to reduce homelessness. It would include:

• Pre-post counts on various measures of homelessness in selected communities, overall and for selected sub-populations;
• Analysis of macro-level economic indicators and policy and program factors (i. e. , major non-NHI related factors) and application of that analysis to changes in homelessness rates;
• Qualitative review and financial analysis of investments to assess the role of the NHI in changing homelessness rates relative to other homelessness-related inputs.

Option 3-Focus on specific programs and services

Option 3 would include the elements described in the first two options, but would also include a major element focusing on the evaluation of the effectiveness of individual programs and services. This would further the analysis of changes in the prevalence of homelessness, to try to identify specific interventions and sets of related interventions that appear to be effective. It would provide a greatly enhanced indication of the linkages between NHI investments in facilities and services and success in alleviating homelessness for individuals. It would also provide much enhanced guidance about the kinds of interventions that appear to work for different groups of clients, and the types of circumstances that appear to be conducive to successful intervention.

This option would be the most comprehensive, time consuming and expensive. It would require an intensive period of at least a year of research design and planning. This stage would involve numerous service-providing agencies across the country, pre-testing of methods, and several years to conduct the research itself. Ideally, it would be designed to carry forward into the future to provide longitudinal data for longer-term analysis.