|
|
|
|
|
|
|
|
|
 |
 |
 |
 |
 |
|
|||||
|
|
|
|
 |
 |
|||||
|
 |
|
 |
|
 |
|
2005-614 Evaluation Framework for PWGSC's Involvement in the Surplus Federal Real Property for Homelessness Initiative (SFRPHI), Final ReportSeptember 8, 2006 Table of Contents
Executive SummaryThis evaluation framework for Public Works and Government Services Canada's (PWGSC) involvement in the Surplus Federal Real Property for Homelessness Initiative (SFRPHI) was developed under the authority of the 2005-2006 Audit and Evaluation Multi-year Plan of PWGSC. SFRPHI is a component of the National Homeless Initiative (NHI), a government-wide strategic horizontal initiative, led by Human Resources and Skills Development Canada (HRSDC). SFRPHI was created to help communities overcome the high capital costs that many homelessness projects face, like buying land or buildings. Non-profit organizations and other levels of government are eligible to receive federal buildings or land to assist homeless people if their community has a demonstrable homeless population. The objective of the evaluation is to assess the success and effectiveness of PWGSC's involvement in SRFPHI in order to provide departmental management with information on departmental results. An interdepartmental tripartite committee composed of representatives from HRSDC, Canada Mortgage and Housing Corporation (CMHC) and PWGSC manages SFRPHI. PWGSC's contribution to SFRPHI is to manage the disposal and transactions of surplus federal real properties, and the Affordable Housing Property Fund Program (AHPF), a $9 million fund to compensate custodial federal departments (including PWGSC) for the fair market value of properties the NHI has secured. HRSDC is also undertaking a summative evaluation of the NHI which will include the results with respect to community partnerships and investment achieved through SFRPHI. The evaluation issues are categorized within the following areas as per Treasury Board Secretariat (TBS) policy:
Evaluation Issues and QuestionsSuccess Issue 1: To what extent has PWGSC contributed to the SFRPHI objectives? Q1. What is the extent of SFRPHI stakeholder satisfaction with PWGSC involvement? Q2. Do stakeholders have sufficient information to better prepare proposals and finalize SFRPHI transfers? Q3. Have any potential SFRPHI transfers been ended as a direct result of PWGSC involvement? If so why? Q4. Could improvements to PWGSC's SFRPHI processes contribute to a greater number of transfers? Q5. Have PWGSC SFRPHI activities contributed to unintended benefits? If so, what are they? Issue 2: To what extent were the policies, practices and ethics guiding the disposal of surplus federal real properties upheld in dealing with the SFRPHI transfers? Q6. To what extent have surplus real property transfers as a result of SFRPHI been realized in an equitable, fair and transparent manner? Q7. To what extent have the interests of communities and local governments been considered? Q8. Were existing real property disposal processes modified to accommodate SFRPHI? Cost-effectiveness and appropriate alternatives Issue 3: To what extent has public value been maximized through the SFRPHI transfers? Q9. Are Canadians getting the best value in SFRPHI transfers? Q10. Are there more effective alternatives to having PWGSC provide surplus real property disposal services for SFRPHI? If so, what are they? Rationale/Relevance Issue 4: Is there a continuing need for PWGSC involvement in the SFRPHI? Q11. Does SFRPHI fit strategically with the strategic concerns of the government and the direction and priorities of PWGSC? Q12. Does PWGSC's role in SFRPHI address knowledge or capacity gaps in the government's homelessness strategy? Methodology The Audit and Evaluation Branch (AEB) suggests an approach that will study 4 case studies of SFRPHI transfers or clusters of SFRPHI transfers in St. John's Nfld., Montreal, Quebec, Ottawa, Ontario and Strathmore, Alberta. The estimated costs of approximately $65,000 in professional services and travel, and 50 AEB person-days for the proposed evaluation provides a general order of magnitude in terms of the level of effort to carry out the evaluation work. It is recommended that the Assistant Deputy Minister (ADM), Real Property Branch (RPB), accept this evaluation framework.
1. IntroductionThis document presents the evaluation framework for the summative evaluation of the Department of Public Works and Government Services Canada's (PWGSC) involvement in the Surplus Federal Real Property for Homelessness Initiative (SFRPHI). This joint initiative involving Human Resources Skills Development Canada (HRSDC) and the Canada Mortgage and Housing Corporation (CMHC) is a component of the National Homeless Initiative (NHI), a strategic horizontal initiative of the Government of Canada (GoC) led by HRSDC. 1.1 Approach and MethodologyThe evaluation framework will ensure that appropriate evaluation issues are identified and addressed and that the appropriate data, information sources, and strategies for gathering data will be in place to conduct a subsequent evaluation in accordance with the Evaluation Policy of the Government of Canada, and meet the evaluation standards of the Audit and Evaluation Branch (AEB) of PWGSC. TBS' methodology on program evaluation and the AEB professional practices will be used as guidelines for the development of this Evaluation framework. The methodology utilized for the preparation of this evaluation framework included the following:
1.2 BackgroundIn December 1999, the Government of Canada launched the three-year, $753 million National Homelessness Initiative (NHI), a community focused horizontal initiative designed to alleviate and prevent homelessness in communities located in all provinces and territories. The NHI is lead by HRSDC. Other partners include the CMHC, PWGSC, the Department of National Defence (DND), Health Canada (HC), Indian and Northern Affairs Canada (INAC), Citizenship and Immigration Canada (CIC) and Justice Canada. SFRPHI was created to help communities overcome the high capital costs that many homelessness projects face, like buying land or buildings. Government departments are encouraged to identify such properties and receive compensation for lost revenues when the property is transferred to a group for the purpose of assisting homeless people. Non-profit organizations and other levels of government are eligible to receive federal buildings or land to assist homeless people if their community has a demonstrable homeless population. Based on the successes and the lessons learned, the NHI was extended for an additional three years (2003-2006) with an investment of $405 million. This extension includes the following program components and funding allocations:
HRSDC, CMHC and PWGSC act as partners at national and regional levels in implementing the SFRPHI. PWGSC's contribution is to manage the disposal and transactions of surplus federal real properties and the Affordable Housing Property Fund Program (AHPF), a $9 million fund to compensate custodial federal departments (including PWGSC) for the fair market value of properties the NHI has secured. A formative evaluation of the SFRPHI was completed in September 2002. The study's key findings were that:
During the initial 3-year period of the initiative, the delay between declaration of property as surplus and actual transfer to recipient groups was often cited as a source of frustration by custodian departments. Under Treasury Board policy, a surplus property must be circulated to all federal departments and agencies, as well as provincial and municipal levels. Interested parties have 120 days to agree to the transfer and must complete the transfer as soon as possible thereafter. If no interest is expressed within that 120-day period, the property is then offered for sale to the public. Under the terms of the Homelessness Initiative, HRSD acts as the interested federal agency for properties that could be appropriate for use in alleviating homelessness. During the first phase of the Homelessness Initiative, the process to complete the transfer of properties averaged over 500 days. In normal market conditions a sale is expected to occur within 180 days. Custodian departments are responsible to maintain properties until transfer out of their inventory, and in the case of the Homelessness initiative, they have been funding Operating and Maintenance (O&M) activities well beyond what would normally be expected for a routine disposal. [*]. Final evaluation and selection of successful proposals from across Canada and monitoring of the fund is done by a tripartite committee comprising representatives from HRSDC, PWGSC and CMHC. The tripartite committee is also responsible for evaluation and recommendation of requests for reimbursement of eligible costs under this framework. 1.3 Overall Risk AssessmentThe risk level for PWGSC's role and accountabilities for the SFRPHI is moderate based on the analysis of the risk factors associated with materiality, renewal commitments, public interest, and delivery complexity. PWGSC's overall risk level is increased as a result of the political risks it shares with its other SRFPHI partners and by extension as a partner in the larger horizontal National Homelessness Initiative. There are two principle reasons that make this summative evaluation of particular interest to the Audit, Assurance and Ethics Committee (AAEC). Firstly, [*]. Secondly, this evaluation engagement provides PWGSC with an opportunity to learn how the department implements program evaluations of its accountabilities in strategic horizontal initiatives.
2. Component Profile2.1 The Renewed Surplus Federal Real Property for Homelessness InitiativeThe primary objective of SFRPHI is to contribute federal real property and technical support to enhance the capacity of communities to provide sustainable facilities (i.e. service delivery, emergency shelters, transitional and supportive housing, affordable housing) to help stabilize the living arrangements of the homeless and those most at risk of becoming homeless. SFRPHI is managed by an interdepartmental tripartite committee composed of representatives from HRSDC, CMHC and PWGSC, with support from the National Secretariat for Homelessness (NSH). During the first phase of SFRPHI (1999-2002), 50 properties worth $9.2 million were approved for transfer that resulted in the creation of 214 transitional and affordable housing units. For the second phase of SFRPHI (2003-2006) an additional $9 million of surplus federal real property, at the rate of $3 million per year, was made available to support community-based activities to alleviate and prevent homelessness. As of the summer of 2005, 17 properties valued at $2.7 million had been transferred and a further 18 proposals for properties valued at $4.4 million were in the process of preparation. [*]. The payment of non-recoverable PWGSC program administration expenditures (up to an annual maximum of $100,000), incidental service costs and extraordinary costs incurred by federal property custodians due to significant program delays are also funded through AHPF. Under SFRPHI, community providers develop proposals, with assistance provided by HRSDC and CMHC regional offices, to alleviate homelessness through investments in facilities and services for homeless individuals or those at risk of homelessness. Community proposals are evaluated and recommended by the tripartite committee based upon a thorough assessment of the potential recipient's capacity to implement the proposed project as well as the financial viability of the submitted business case. The tripartite committee has systems, procedures and resources in place to ensure due diligence prior to recommending approval of the non-cash contributions of real property. This due diligence process includes verifying eligibility and entitlement, and the management and administration of the program. Authority to approve, sign or amend contribution agreements under SFRPHI is shared between HRSDC and PWGSC. PWGSC provides real property expertise, information and assistance concerning available surplus federal real properties as well as carrying out the property sales. Transfers of surplus federal real property to eligible recipients under SFRPHI are subject to the Treasury Board Secretariat Policy on Transfer Payments, with a notable deviation from that policy in that the payments are in the form of non-cash contributions of real property rather than cash contributions. Key stakeholder groups include federal property custodians, other levels of government and non-governmental organizations that provide community facilities and assistance in support of homeless individuals and families as well as those at risk of becoming homeless. 2.2 Program DeliveryThe processes and authorities for disposing of surplus federal real property are defined in the Treasury Board Policy on the Disposal of Surplus Real Property. The objective of this policy is to establish a system for the disposal of surplus real property that ensures:
When federal government properties are no longer required for program purposes, the disposal of these surplus properties by sale or transfer is subject to one of two processes: routine or strategic. Surplus real properties subject to routine disposal are generally properties with lesser value that can be sold easily without any substantial investment. These properties are normally sold on the open market in their "as is" state by the custodian, its agent (PWGSC), or a private sector firm. Surplus real properties subject to strategic disposal are properties or portfolios of properties with potential for significantly enhanced value, those that are highly sensitive, or a combination of these factors. Surplus properties that are subject to a strategic disposal process are sold to the Canada Lands Corporation (CLC) Limited, the government's disposal agent for strategic properties. Although properties subject to a strategic disposal are generally not offered for sale or transfer on a priority basis for public purposes to other jurisdictions, [*]. Appendix A provides an overview of the federal real property disposal processes.
3. Logic Model: Activities to OutcomesExhibit 1, illustrates how SFRPHI activities undertaken by PWGSC contribute to the achievement of its stated objectives and desired outcomes. The logic model presents an overview of the entire initiative including activities and outcomes for which PWGSC is not accountable. The following section provides an explanation of these activities and accountabilities. 3.1 Activities and OutputsActivity 1, Manage federal surplus real property disposals, PWGSC PWGSC provides expertise related to real property and information and assistance concerning available surplus federal real properties, as well as carrying out property, sales and administration of the funds. Outputs: PWGSC maintains an accounting framework for SFRPHI funds, including the compensation of federal entities and the identification of foregone revenues. The funding framework includes making payments to agent Crown Corporations in cases where their surplus federal real property is approved for contribution; and, non-agent Crown Corporations only in cases where surplus federal real properties being disposed of under the strategic process outlined in the Treasury Board policy are being pursued. Once a property has been recommended and approved for transfer under SFRPHI, PWGSC undertakes the property transfer from the custodian department to the homelessness related community service provider. Real property transfers include: negotiations with custodians, other jurisdictional governments and community providers; undertaking of due diligence reports (legal, environmental and technical); property appraisals; and standard property transfers. In the case of strategic properties, the SFRPHI funding framework provides for payments to the Canada Lands Corporation. PWGSC is a co-signatary with HRSDC of the contribution agreements with homelessness related community group. HRSDC has the responsibility of monitoring the contribution agreements once in place. Outcomes: Disposals made through SFRPHI have been completed efficiently, equitably, with fairness and transparency. Federal surplus real property is disposed in accordance with Public Service values and ethics. Better value to Canadians in disposals of surplus federal real property. Exhibit 1 (View detailed figure by clicking on link) Activity 2, Develop and maintain partnerships with other federal departments, PWGSC, HRSDC and CMHC Fundamental to the SFRPHI's design and operationalization is the importance of partnerships, particularly across the federal government. Outputs: The National Homelessness Secretariat (NHS) of HRSDC provides the support role to these activities which are primarily focused on the interdepartmental tripartite committee that oversees SFRPHI. This includes the development of the SFRPHI framework, strategic priorities and plans, as well as program terms and conditions. As a member of the tripartite committee, PWGSC provides input on submitted SFRPHI proposals and on custodian reimbursements. NSH provides additional support and national co-ordination. Given that the Federal Coordinator on Homelessness holds accountability for demonstrating overall results for the NHI, the NSH assumes responsibility for collecting the performance measurement data for SFRPHI and undertaking evaluation activities examining the extent of sustained capacity communities have as a result of the NHI (and SFRPHI specifically). Outcomes: Coordination among federal departments to provide continued guidance and more properties under SFRPHI to address homelessness. This includes: Continued and expanded efforts at the federal level to realize projects, including the continuation cooperation of the tripartite committee; and, the commitment of other federal custodians to make available their surplus properties through the SFRPHI process. Activity 3, Community support and capacity building, National Secretariat on Homelessness / Human Resources Skills Development Canada (HRSDC) supported by CMHC Partnerships and relationship building are at the core of SFRPHI and are crucial to the development of the sustainable capacity of communities to address homelessness. The following outputs and immediate outcomes are expected as a result of community support and capacity building activities. Outputs: HRSDC's regional and local staff provides expertise concerning specific local needs and the level of community support for proposals submitted to the tripartite committee. HRSDC supported by CMHC assists potential recipients in the development of SFRPHI proposals as required, and advises on their viability as part of the process of reviewing proposals and recommending contributions. Outcomes: Increased understanding of community priorities. Broader engagement of partners to address homelessness. Necessary partnerships are in place to sustain community efforts to address homelessness beyond the timeframe of SFRPHI. Enhanced supports and services available to meet the needs of homeless individuals and families and those at risk of homelessness through the provision of individual projects. Services and supports are available for individuals at-risk to move from unstable living conditions, and other unsafe conditions, to transitional or supportive accommodation that would give them the opportunity to live independently in a stable environment. Activity 4, Managing the investments, National Secretariat on Homelessness / Human Resources Skills Development Canada (HRSDC) SFRPHI's core objective is to facilitate the transfer of surplus federal real properties to support community-based initiatives, to address the needs of homeless people and those at risk of homelessness. SFRPHI aims to contribute to the NHI's objectives of moving more individuals and families into stable living environments with access to the services and supports needed to move to self-sufficiency, and prevent others from becoming homeless. The following outputs and immediate outcomes are expected as a result of community investment activities: Outputs: Contribution agreements between Her Majesty the Queen in Right of Canada, represented by the Minister of Public Works and Government Services and Minister of Labour; Outcomes: Increased local capacity and resources to deal with homelessness through the provision of projects and formulation of partnerships to improve services and facilities for homeless people (from transitional facilities to prevention). The increased supports indicate an enhanced community ownership of solutions. Longer-term and ultimate outcomes for all of SRPHI: Communities have in place, as a result of SFRPHI, the transitional and supportive services and interventions necessary to move people out of homelessness, as well as the services needed to help prevent homelessness, which together could eventually lead to a reduction in homelessness in Canada. SFRPHI proponents develop a range of funding partners to ensure their efforts and investments are able to address the range of needs of homeless people in a sustainable, long-term approach.
4. Evaluation Strategy[*]. The RMAF indicated that HRSDC will include a focus on the SFRPHI's contribution to building the capacity of communities to address homelessness as part of their NHI Evaluation. The RMAF also states that the PWGSC's evaluation should focus on how the program functions in relation to policies and frameworks guiding the disposal of surplus federal real properties. Given the concurrent evaluation activities of HRSDC and PWGSC, both evaluation teams will work closely together to coordinate the evaluation work in order to reduce any duplication. Consequently, AEB does not recommend questions related to the ongoing need of the SRFPHI and the NHI as this question should in principle be adequately addressed in the NHI Evaluation. 4.1 Evaluation IssuesThe objective of the evaluation is to assess the success and effectiveness of PWGSC's involvement in SRFPHI in order to provide departmental management with information on departmental results. The evaluation issues are categorized within the following areas as per Treasury Board Secretariat (TBS) policy:
To this end, Exhibit 2, presents the Evaluation Matrix which outlines the evaluation issues along with more specific evaluation questions to examine, indicators and data sources. 4.2 MethodologyIn developing the approach for this evaluation, AEB was influenced by a number of factors, the most important of which being that PWGSC's role in SFRPHI is driven by individual property disposal transactions. Exhibit 2 - Evaluation Matrix
Each effective transfer or contribution agreement brings together unique participants who have either very limited experience with PWGSC and SFRPHI or they have developed a certain degree of expertise such as the case of stakeholders in St. John's Nfld., where there has been 22 SRPHI property transfers since the program was started in 2000. As a result of these unique variables, combined with the fact that there are only about 35 to 50 SFRPHI property transfers over a three year period, it is clear that three separate lines of enquiry: case studies, documentation and administrative file review; and, key informant interviews, supplemented by some degree of cost benefit analysis will be sufficient to answer the evaluation issues put forth in this framework. The estimated costs are approximately $65,000 in professional services and travel, and 50 AEB person-days for the proposed evaluation. This estimate provides a general order of magnitude in terms of the level of effort to carry out the evaluation work. The amount may be adjusted depending on the exact nature of the coverage required at the time the evaluation work is commenced. 4.2.1 Case StudiesCase studies of representative transfers or group of transfers will be undertaken in order to provide more detailed information on the proposed evaluation issues. The criteria to determine which transfers for inclusion in the case study portion of the evaluation include: the property type, the market value, the level of experience of all stakeholders involved in the transfer, and the perceived degree of success within the community. Key documentation will be reviewed for each case study. This information will be supplemented by interviews with all key partners at the national and regional level responsible for the implementation of the SFRPHI transfers. A synopsis will be prepared for each of the case studies outlining key findings. The four case studies AEB plan on evaluating are:
4.2.2 Documentation and File ReviewRelevant documents, studies and files will be reviewed in order to obtain contextual information relative to the development, structure and achievements of the property transfers. Documentation related to previous studies and performance reports will also be reviewed to identify common themes and issues related to the implementation and immediate outcomes. This information will be used in the development of data collection tools for the key informant interviews. Documents will also be reviewed in order to obtain quantitative measures which may be used to support the qualitative data collected in the case studies. 4.2.3 Key Informant InterviewsInterviews with individuals involved with the development and implementation of SFRPHI transfers represent a rich source of information regarding PWGSC activities, achievements, impacts and opportunities for service delivery enhancements. Key informants include individuals from the following stakeholder groups:
Key informants will be selected in relation to their roles and responsibilities, their familiarity with SFRPHI transfers and PWGSC activities, their involvement in the development of proposals, and their knowledge of the key issues surrounding PWGSC's implementation and development of SFRPHI. 4.2.4 Determination of Best Public ValueAll programs aim to produce benefits that outweigh their costs. Peter Drucker suggested that efficiency is about "doing things right" and effectiveness is about "doing the right things". In SFRPHI terms, doing things right means achieving the best public value for surplus real property. The TBS guide on program evaluation[1] identifies cost benefit and cost effectiveness analysis as two common methods used to address the program evaluation requirement for measuring cost-effectiveness. Cost-benefit analysis (CBA): CBA compares monetary benefits (outputs or outcomes) with investment costs (inputs).
Cost-effectiveness analysis (CEA): CEA compares benefit results (outputs or outcomes) with costs, but results are not necessarily transformed into monetary units.
In order to address the public value issue presented in this framework, AEB will conduct research to determine if there are any existing studies that have transformed into monetary terms the costs of homelessness and/or the societal costs of one homeless person. This value will be used along with costs estimates of the case study SFRPHI transfers that will be obtained from existing management information in HRSDC and PWGSC and/or from estimates obtained from interviews with key informants. These values will be considered with data being collected by the NSH, on the amounts each SFRPHI project has managed to leverage from sources other than the GoC. Once these estimates of the costs and benefits will have been gathered, AEB will present the information in a manner which allows decision-makers to better understand the value of the SFRPHI in monetary and economic terms.
5. RecommendationIt is recommended that the ADM, Real Property Branch, accept this evaluation framework and the strategy outlined as the basis for undertaking the summative evaluation of PWGSC's involvement in SFRPHI.
Appendix A - Federal Real Property Disposal ProcessAppendix A (View detailed figure by clicking on link)
Appendix B - List of Interviewees
Footnote:[1] Program Evaluation Guide, Treasury Board Secretariat, 1991
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||||
|
|
|||
|
