Current Reports

2004-610 Evaluation Framework for the Alternative Forms of Delivery (AFD) Initiative for Real Property Services, Final Report
2005-05-26

• Executive Summary
  
  
• 1.0 Introduction
  • 1.1 Purpose
  • 1.2 Methodology
  
  
• 2.0 Background
  • 2.1 Environment
  • 2.2 AFD 1998-2005
  
  
• 3.0 Component Profile
  • 3.1 PWGSC Real Property Role
  • 3.2 AFD Objectives
  • 3.3 AFD Delivery Approach
    • 3.3.1 Performance Fee Payment
  • 3.4 AFD Logic Model
    • 3.4.1 Logic Model Diagram
    • 3.4.2 Detailed Description
    
    
• 4.0 Evaluation Strategy
  • 4.1 Evaluation Approach
    • 4.1.1 Interim Evaluation
    • 4.1.2 Final Evaluation
  • 4.2 Evaluation Issues
  • 4.3 Data Requirements
  • 4.4 Identification of Performance Indicators
    
  
  
• 5.0 Recommendations
  
  
• Appendix A: Estimated Evaluation Costs
  
  
• Appendix B: Documents Reviewed, Key Stakeholder Interviews and Workshop Participants
  
  
• Appendix C: Acronyms

Executive Summary

Authority

This evaluation framework for the Alternative Forms of Delivery (AFD) Initiative was developed under the authority of the 2004-2005 Audit and Evaluation Annual Plan of Public Works and Government Services Canada (PWGSC).

Objective/ Scope

The purpose of this document is to present an evaluation framework for formal evaluations of the Alternative Forms of Delivery (AFD) Initiative to commence April 1, 2005. PWGSC's business plan endeavors to use Alternative Forms of Delivery to deliver property management and project delivery services in many of its office and common use facilities across Canada, which will enable PWGSC to expand its capacity and focus internal resources on strategic real property activities. The objective is to develop an evaluation framework, which will ultimately be used to assist in assessing the Initiative's relevance, success, and cost effectiveness.

Background

PWGSC is one of the principle custodians of real property for the Federal Government. As such, the Department is charged with representing ownership interests and managing office buildings and special-purpose facilities across Canada.

In 1998, the Department initiated the large-scale outsourcing of property management and project delivery services in most of its office facilities across Canada. PWGSC is continuing to contract the private sector for the provision of high quality services to its clients. For its part, as steward of the facilities, the Department will retain overall responsibility for asset and client management. It will also continue to set standards and performance requirements related to its real property assets. The Department aims to achieve the following objectives with the AFD Initiative to commence April 1. 2005

• Harvest the full value of private sector expertise and develop world class public-private relationships;
• Achieve the best value for money in providing real property services;
• Continue to provide high quality services to its clients in a framework of continuing improvement; and,
• Ensure the condition and value of federal real property assets on an ongoing basis.

Outline of the Evaluation Framework

This Evaluation Framework describes the AFD Initiative's objectives, rationale, implementation processes, and intended outcomes. The framework consists of a background of the AFD Initiative, a component profile which describes the AFD's three key activities, their outputs, and expected outcomes and its one page illustration in the form of a logic model. It also includes an evaluation strategy that outlines evaluation options, potential evaluation issues, performance indicators that address these issues, and data sources, collection methods and timing of evaluations.

The proposed evaluations for the AFD Initiative will provide an in-depth, periodic opportunity to examine important questions with respect to how the Initiative has performed to date and to suggest what possible changes or directions might be made to further enhance the achievement of the intended AFD objectives and outcomes.

The evaluation issues are categorized within the following areas as per Treasury Board Secretariat (TBS) policy:

• Rational / Relevance;
• Program Design and Delivery;
• Success
• Overall Cost Effectiveness and appropriate alternatives

The proposed evaluation options include an interim evaluation, to be carried out in the start up phase of the AFD Initiative and a final evaluation to be conducted early in year three to allow for contract renewal options to take place in a timely manner. An estimate of evaluation costs is presented in Appendix A.

The proposed data collection methods and sources for the AFD Initiative include the following:

• Document review;
• File review;
• Administrative data review;
• Key informant interviews; and,
• Surveys.

Recommendations

It is recommended that:

1. The ADM, Real Property Branch, accept this evaluation framework and the strategy outlined as the basis for undertaking future evaluation work, including that the proposed final evaluation in year three would chart out any subsequent evaluation work, should it be PWGSC's intention at that time to extend the contract past the four-year base period.

2. The ADM, Real Property Branch, ensure that performance information required for future evaluations, as specified in Exhibit 1 of this evaluation framework, is incorporated into real property business line information processes and collected on a timely, ongoing basis.

3. The ADM, Real Property Branch, in conjunction with the ADM, Service Integration, the ADM, Acquisitions, and the ADM, Finance, Accounting, Banking and Compensation, identify all component costs associated with this AFD Initiative and establish and implement any necessary mechanisms required in order to collect and compile all significant costs of this AFD Initiative.

1.0 Introduction

1.1 Purpose

The purpose of this document is to present an evaluation framework for the formal evaluations of the Alternative Forms of Delivery (AFD) Initiative that will commence on April 1, 2005. The evaluation framework herein is being proposed to be used to evaluate the effectiveness and results of this AFD Initiative.

1.2 Methodology

The methodology utilized for the preparation of this evaluation framework included the following:

• A review of documents related to the AFD Initiative;
• An initial round of interviews with key stakeholders which resulted in the identification of issues, challenges, benefits, objectives, success factors, and lessons learned regarding the AFD Initiative;
• Workshops with key stakeholders directly involved with the development and implementation of the AFD Initiative;
• Input from interviews and workshops were analyzed to develop;
  • A logic model that graphically depicts the AFD Initiative
  • Significant Evaluation Issues/Questions
  • An Evaluation Strategy;
• Preparation and submission of a preliminary draft for review and comment;
• Submission of a revised draft for acceptance by the ADM RPB;
• Issuance of a final report.

2.0 Background

2.1 Environment

PWGSC is the common service provider to more than 140 Government of Canada departments, agencies and Crown corporations. With a workforce of nearly 14,000 people across Canada, with locations in North America and Europe, the goal of PWGSC is to provide the best value for taxpayers' dollars in common and central services for the Government of Canada, with due regard for the important values of prudence, probity and transparency.

The Real Property Branch (RPB) of PWGSC is the Government of Canada's (GOC) real property service delivery manager and strategic advisor. As such, the RPB is the departmental entity mandated with delivering the real property business line, which includes acquiring, managing, operating, maintaining, building, repairing and disposing of federal real property. This inventory ranges from large, multipurpose complexes in major urban centres, to small buildings in towns and villages across Canada, to industrial buildings, laboratories and other specialized facilities.

The GOC has pursued continuous improvement and innovation in the delivery of its programs and services. The Treasury Board Secretariat's Alternative Service Delivery Policy was put into effect to pursue new and appropriate organizational forms and arrangements, including partnerships with other levels of government and other sectors, in order to improve the delivery of programs and services. Anticipated benefits from innovative organizational arrangements for delivering government programs and services are:

• More cost-effective, responsive delivery to Canadian citizens;
• Changes in organizational culture and management practices so that the organization performs more effectively; and
• The granting of greater authority to managers, thus moving decision making closer to the point of delivery, to the communities served and to Canadian citizens.

These concepts are considered the cornerstones of PWGSC's AFD Initiative.

2.2 AFD 1998-2005

The contracts for the AFD Initiative currently in place were signed on March 18, 1998 and are due to expire on March 31, 2005. Over the course of that first term of this Initiative, there has been substantial scrutiny. The most relevant studies for the purposes of the development of the Evaluation Framework for the AFD Initiative to commence April 1, 2005 are briefly described below.

The Evaluation Framework for the Initiative currently in place was completed in September 1998 (# 97-613). The objective of that framework was to provide a basis for the formal evaluation of the effectiveness and results of the AFD Initiative. The Report of the Auditor General of Canada was published in September 1999. Chapter 18, titled " Public Works and Government Services Canada Alternative Forms of Delivery: Contracting for Property Management Services" addressed PWGSC's contracting process and the on-going management of the contracts. The report produced conclusions and some lessons learned. In the summer of 2000 Consulting and Audit Canada produced a report titled " Review of the Implementation of the Alternative Forms of Delivery for Property Management Services Stakeholders Perspectives" (# 330-0890). The focus of this review was on specific development and implementation issues and some design and delivery issues that related to the implementation of that first Initiative.

A Revised Evaluation Framework was completed in 2002, just prior to beginning the summative evaluation for the current AFD Initiative (# 2002-640). Four specific revisions to the Framework are worthy of special note: two revisions responded to issues related to the adequacy of PWGSC's financial systems and the control process for capital expenditures; a third area of revision related to the cost savings generated by the AFD and the availability of supporting data; the final area of revision related to the bringing forward of the timing of the summative evaluation from 2004 to 2002. The summative evaluation was finalized in March 2003 and formulated four primary recommendations. In the ADM RPB's " Real Property Branch Audit File Action Plan /Closure Analysis" it states that recommendations 1, 2 and 4 have been addressed through improvements to the SOW and related operational documents for the AFD Initiative to commence April 1, 2005. In response to recommendation 3, a champion has been named and work on the consolidation and provision of readily available program wide management information is on going.

3.0 Component Profile

3.1 PWGSC Real Property Role

With respect to real property, PWGSC is committed to:

• Safe, healthy, secure, affordable and environmentally sustainable real property solutions and workplaces that contribute to productivity and to the success of customer programs and objectives;
• Responsible stewardship of government assets and maintenance of their value on behalf of all Canadians; and
• A high level of customer satisfaction based on timely delivery of integrated and customized services.

3.2 AFD Objectives

The existing AFD contracts are due to expire on March 31, 2005 and PWGSC is enhancing the AFD approach to service delivery with an AFD Initiative to commence April 1, 2005. This Initiative aims to:

• Harvest the full value of private sector expertise and develop world-class public-private relationships;
• Permit the Department to focus its resources on the core elements of real property management and client relationships;
• Achieve the best value for money in providing real property services;
• Maintain quality of service to our clients; and
• Ensure the condition and value of federal real property assets.

In order to fulfill these objectives, the Department is seeking to attract bids from dynamic, solution-oriented private sector real property service providers who are committed to long-term collaborative relationships and strategic alliances for the delivery of high quality services to PWGSC clients. The contracting process is underway and contract award is expected in the third quarter of 2004-05.

3.3 AFD Delivery Approach

The AFD contracts commencing April 1, 2005 will provide for the outsourcing of property management and project delivery services for more than 300 federal buildings across Canada, representing some 2.9 million square meters of rentable space. This new procurement is designed to replace thirteen existing contracts and two provincial agreements that expire on March 31, 2005. This AFD Initiative comprises eight real property services contracts, with a potential life span of up to ten years each. Each contract will include an option, for the benefit of the Crown, to extend the base four-year term of the contract by periods of two years or multiples thereof, but not to exceed six years in the aggregate under the same terms and conditions.

Properties included in this Initiative have been grouped into eight Portfolios, giving rise to an equivalent number of contracts. These Portfolios are grouped along regional lines and comprise the following approximate volumes:

* The Western Portfolio includes buildings in Saskatchewan
** The Pacific Portfolio includes buildings currently being managed by the British Colombia Building Corporation (BCBC) under a federal-provincial agreement.
*** Projections for year 2005-2006 based on Operations & Maintenance and Projects valued at less than $200,000, which include Tenant Service Work.

In this AFD Initiative, the Department retains overall responsibility for asset management. It also continues to set standards and performance requirements related to its real property assets. Contractors will be required to deal with various government organizations and officials in order to effectively deliver the services identified in the Statement of Work. Of principal importance are the many tenants in PWGSC buildings who rely on PWGSC to provide a productive work environment. The contractors will be required to respond to tenants' service needs and concerns in a manner that satisfies PWGSC's goals for tenant satisfaction. For each contract, PWGSC will have a Technical Authority responsible for matters relating to performing the Statement of Work and a Contracting Authority responsible for matters related to the administration of the contract. The development of a Governance Structure and an Accountability Framework are in progress at the time of writing of this report.

The deliverables for this AFD Initiative include eight categories of services. The three primary service categories of property management, project delivery, and optional services are linked to a system of Key Performance Indicators (KPI) for accountability and reporting. The eight categories are:

1. Requirements related to all services;
2. Planning, budgeting and expenditure management;
3. Property management services;
4. Project delivery services;
5. Information management and reporting;
6. Optional services;
7. Contract initiation and completion; and,
8. Excluded services.

The nature and scope of each service category is briefly described below.

I) Requirements related to all services consist of:

• Tenant relationship management;
• Quality management;
• Subcontracting management;
• Occupational health and safety;
• Environmental protection and conservation;
• Contract relationship management;
• Management of PWGSC contracts;
• Novation and assignment of existing contracts;
• Stewardship of heritage character and design quality of buildings;
• Critical incidents; and,
• Risk management.

These eleven requirements apply to all portions of the Statement of Work, whether property management, project delivery or optional services. With regard to tenant relationship management, Service Integration Teams (SIT) act as the focal point for providing integrated real property service delivery to tenants. SIT's negotiate service delivery agreements, develop tenant strategies, forge national and regional tenant relationships, ensure overall tenant satisfaction, and achieve financial viability and other inventory management targets. In support of the SITs' role, the contractor assumes responsibility for managing the day-to-day relationship with tenants.

Since the federal government operates within a set of guidelines with health and safety being paramount, it is also expected that the contractors will meet the highest standards of health and safety in delivering their services. Finally, because PWGSC is strongly committed to leadership on environmental issues, the contractors will be expected to meet targets that demonstrate this commitment.

II) Planning, budgeting and expenditure management consist of:

• Annual planning and budgeting; and,
• Expenditure management.

With respect to the work planning function, PWGSC expects the contractors to display analysis, creativity and innovation. Every year contractors will be required to submit a comprehensive Building Management Plan and budget for each facility, including revenue projections, expected operating and maintenance costs, a project plan for repair and capital improvements, and an overall management analysis. In the management analysis, PWGSC expects the contractor to provide good ideas as to how the Department can get better performance from the properties and how contractors can offer better value for money by managing an entire Portfolio. As with all prudent business arrangements, contractors will be required to manage, control, account for and report on expenditures within an approved budget.

III) Property management services consist of:

• Building operations (operations, service calls, energy management, maintenance, cleaning, materiel management, other building services); and,
• Physical security & emergency planning.

As an institutional owner in the public sector, PWGSC's main focus is on:

• Providing productive workplaces for PWGSC's government tenants;
• Protecting the Crown's investment in real property; and,
• Reflecting broader government priorities in the management of PWGSC's assets.

IV) Project delivery services consist of:

• Repair and improvement projects;
• Individual projects up to $200,000;
• Associated commissioning activities; and,
• Tenant alteration projects.

Contractors are required to manage a program of repair and improvement projects. It is expected that the private sector entrepreneurial spirit and skills will be showcased in these projects. All parties expect projects to be delivered faster, better and for greater value. Contractors are asked to manage a program of projects up to a maximum value per project of $200,000 through all stages of project delivery, including project planning, definition, design, implementation, and commissioning. Contractors will also be asked to take on all commissioning activities for projects undertaken within the buildings they manage, including large projects above $200,000 being delivered by PWGSC. Contractors may also be asked to perform tenant alteration projects.

V) Information management and reporting consists of:

• Comprehensive data sets;
• IT Management practice (e.g., security, backup and disaster recovery);
• Conforming to departmental data formats; and,
• PWGSC systems track performance.

In terms of information management and reporting, PWGSC expects that contractors can deliver accurate and timely information in whatever electronic format PWGSC requires. The contractor shall utilize an electronic maintenance management system to facilitate the delivery of maintenance management services and maintain and update maintenance management data in order to keep information on this system current and complete.

VI) Optional Services consist of:

• Facilities management services;
• Lease administration services;
• Commercial operations related services; and,
• Delivery of projects over $200,000 up to a limit of $1,000,000.

VII) Contract initiation and completion consists of:

• Ensuring a seamless continuation and initiation of Statement of Work services.
• Outlines the activities required of the contractor prior to contract completion.

VIII) Excluded services

There are five key activity areas (core services) that have been retained by the Department. These are:

• Asset management and strategic investment decisions;
• Occupancy instruments;
• Acquisitions and disposals;
• Payments in lieu of taxes; and,
• PWGSC's National Service Call Centre.

Although PWGSC will continue to prepare comprehensive asset management plans, it is expected that the contractors will provide professional input to these plans. Other examples of the excluded responsibilities in the AFD contracts are: the negotiating of occupancy instruments with government tenants - equivalent to leases; the buying, selling and leasing of real estate; making payments in lieu of taxes to municipalities; and, operating PWGSC's own 24-hour-a-day, 7-day-a-week National Service Call Centre. It should be noted that with respect to the interface between this call centre and PWGSC and the contractors, given the contractors' role as property managers, it is expected that the contractors will respond to tenant requests for service that originated from PWGSC's Call Centre.

3.3.1 Performance Fee Payment

Relating to service categories I to VII noted above, the Contractor's Property Management Service Fee, Project Delivery Service Fee and Optional Services Fees are set out in the Basis of Payments schedule of the Contract. While these fees are shown for the full amount of invoices, they are payable at 85% of the fee component of the approved invoice with the remaining 15% of the fee payable at the end of the fiscal year, subject to the Contractor's performance determined in accordance with the Performance Management Framework in Annex "C" of the Contract. This latter 15% of the fee is the performance fee portion for each of the Property Management Service Fee, Project Delivery Service Fee and Optional Services Fees and is further apportioned in relation to each applicable Key Performance Indicator as set out in the Contract.

These above noted fees will be contingent upon meeting thresholds for service delivery as set out in a framework of Key Performance Indicators (KPIs). The three KPI's for these contracts are:

• Asset Integrity: looks at success in operating, maintaining and improving PWGSC building assets throughout their life-cycles, given PWGSC strategic objectives;
• Satisfaction: measures both the Department's satisfaction and its tenants' satisfaction with the contractor's performance; and,
• Financial: focuses on contractors' performance in preparing and managing budgets, their ability to provide reasonable estimates for planning as well as their ability to forecast accurate year-end expenditures. It also evaluates the contractors' performance in achieving commercial operation objectives, where applicable.

On an annual basis, a Contractor Incentive Program (CIP) discretionary award will be available to reward the Contractor's contribution to the achievement of outstanding results in the pursuit of GOC targets and priorities or the discharge by PWGSC of its Departmental mandate. As a prerequisite to being considered eligible for a discretionary award, Contractors must first have earned 100% of available Contract fees during the fiscal year.

3.4 AFD Logic Model

The Logic Model for this AFD Initiative was developed based on a review of the 1998 Evaluation Framework for the current AFD Initiative, the 2003 evaluation of the current series of AFD contracts, review of current documents and input from key stakeholders through a series of interviews and workshops carried out in the conduct of this engagement.

Definitions:

Activities - These are the key activities performed by those involved in the AFD Initiative. These activities are intended to contribute to the achievement of the outcomes.

Outputs - These represent the direct results of the key activities. They demonstrate that the activities have been undertaken. Outputs are measurable and tangible. Products or services generated by the key activities and outputs provide evidence that the activity did occur. The organization is accountable for them and directly controls them.

Outcomes - There are various levels of outcomes, which reflect a chain of results that is expected to occur. Outcomes represent the intended consequences of the activities and outputs. The organization does not control the outcomes, but seeks to influence their occurrence by carrying out certain key activities and delivering certain outputs. Outcomes are typically influenced by other factors as well and managers must maximize the possibility that the services and products that they do control adequately and reasonably contribute to the final outcomes sought. Outcomes are typically articulated at three levels - immediate, intermediate and final as follows:

Immediate outcomes - These are the short-term results that stem from the activities and outputs. Outcomes represent the intended consequences of the activities and outputs.

Intermediate outcomes - These are the next links in the chain of results that occur, flowing from the activities and outputs and occurring after the immediate outcomes have been achieved. These are typically considered to be medium term.

Final outcomes - These are the expected final results of the policy, program or initiative reflecting why the key activities are being engaged in. These are generally results that take a longer period to be realized, are subject to influences beyond the policy, program or initiative itself, and can also be at a more strategic level.

3.4.1 Logic Model Diagram

View a larger version of the above image (pdf 73kb)

3.4.2 Detailed Description

The logic model depicts three main activities for the AFD Initiative along with the key outputs and outcomes they are expected to produce. In the Activity ==> Output ==> Outcome chain, activities are major (in terms of resources) and/or significant (in terms of importance) areas of work. These are the primary purposes for which resources are used. Outputs are the goods or services provided by the Initiative to its customers or clients. These are key because they are the basis on which clients judge the performance of the Initiative. Outcomes are the intended results (or consequences/impacts) of the Initiative's activities and outputs.

The AFD Logic Model consists of the following three main activities:

Activity 1  Plan and Implement the AFD Contracts
Activity 2  Manage the AFD Initiative
Activity 3  Conduct Independent Reviews of the AFD Initiative

Activity 1: Plan and Implement the AFD Contracts

This activity consolidates all the activities involved in preparing to put in place the AFD contracts. These activities begin with PWGSC internal strategic planning regarding whether to continue with AFD as a mechanism for delivering specific types of real property services within PWGSC mandate and continues through to setting in place all activities required for the awarded contracts to function as intended.

Outputs Associated with Activity 1

There are 7 main outputs deriving from this Activity.

Business Case. This includes various feasibility and other studies, reviews, cost benefit and other analyses including risk assessments, position papers, lessons learned, that may have been conducted to support the strategic rationale and scope for the AFD Initiative.

Statement of Work (SOW). This includes the complete scope of the contractor's work, the KPI framework, and associated timelines. The SOW derives from an appropriate business case and extensive consultations with industry and departmental subject matter specialists to ensure that the private sector was in a position to respond to PWGSC's specified requirements. The SOW is ideally designed in such a way as to attract only top quality bidders with proven track records and references. It is required that all government wide and PWGSC policies, practices and values are reflected in the design of the SOW.

Bid Solicitation and Evaluation. This includes the involvement of various groups within PWGSC to develop a Request for Proposal (RFP) and evaluation criteria that clearly reflects the business case and the SOW for this AFD Initiative. The instruments and processes developed for solicitation and evaluation are required to meet government values and priorities.

TB Submissions and Approvals. This includes all preliminary approvals concerning the concept and scope for this AFD Initiative, as well as final submissions and approvals involving contract award.

Contracts Award. This includes the process for awarding the contacts and the actual award to a successful bidder or bidders within the terms and conditions set out in the RFP.

Transition Plans and Activities. This output includes the full range of activities that are required to ensure that the contracts that are awarded can operate within the intended strategic context; that the contracts can be managed to ensure that the terms and conditions of the contracts are being met; that the changeover from the existing contract to the new contracts is as smooth as possible for the clients/customers/tenants, and that the new contractors can be fully operational as soon as the contract commences. These activities include the establishment of a governance structure with clearly defined roles and responsibilities, the establishment of business processes; knowledge transfer of guidelines/policies/regulations, etc.; the establishment of standards and baselines; clearly articulated and clearly communicated reporting requirements and protocols and the identification of associated systems implications. The full range of transition plans and activities require development and implementation to ensure:

• continuity of tenant programs and operations;
• planning of Statement of Work implementation;
• a thorough understanding by the contractor of the Department, its structure, practices, regulations and operations, as well as specifics of the Portfolio and its component buildings;
• planning and coordination leading to the contract operational start date;
• continuity of building operations;
• finalization of contractor management processes and operational procedures;
• contractor staffing;
• establishment of contractor relationships and lines of communication with key parties, including the Technical and Contract Authorities, Service Integration Teams and tenants; and,
• development and testing of contractor information systems and protocols.

Outreach Activities. This includes training, briefings, information sessions, documentation, communications, etc., that are geared toward knowledge transfer and coordination and readiness planning to establish clear expectations and an understanding of this AFD initiative. A range of stakeholders may be involved in these activities, including existing contractors, new contractors, PWGSC employees, clients/customers/tenants, and central agency employees.

Immediate Outcomes for Activity 1

The development of the SOW and the Bid Solicitation and Evaluation process was designed to meet government priorities and values related to the demonstration of a fair, open, accessible, transparent, and competitive process. PWGSC's expectation is that its process results in a transparent and equitable contracting process that establishes a long term public/private relationship built on mutual respect and trust. A high level of competitive bidding was expected from top quality firms in the private sector, with bid prices that reflected a fair market value for the services. With this fair market value as an indicator of the cost of the services requested in the contracts, government would have a solid level of confidence that affordable real property services were to be garnered from the competitive awarding of contracts to the successful bidder or bidders.

The transition activities are focused on putting in place all the ingredients of an improved framework for information and reporting for this AFD Initiative that is consistent with the PWGSC business case and relevant TBS policies. The outreach activities are aimed primarily at the increased stakeholder understanding of AFD requirements, however, in conjunction with the transition activities, are intended to create a smooth, seamless, timely changeover to the new contracts.

Activity 2: Manage the AFD Initiative

The major activities have been categorized into 3 areas: i) Planning and Reporting; ii) Contract Management; iii) Relationship Management. Each sub-activity has its associated outputs.

Outputs Associated with Activity 2

i) Planning and Reporting

This includes 3 main outputs:

• The consolidation of all the transition activities is to result in an AFD Management Framework that includes standards and protocols that can be applied consistently across all contracts. The Management Framework supports the provision of functional direction at the national level.
• Approved plans and budgets from the national portfolio level to the asset level, generated by various levels within PWGSC are required in order to establish plans for the contractors.
• Quality assurance protocols and systems include a range of reporting mechanisms associated with PWGSC responsibilities for internal reporting, including reporting upward and reporting to central agencies. Quality Assurance Protocols and Systems also include protocols and mechanisms that are required for ensuring contractor compliance with all contract terms and conditions regarding quality.

ii) Contract Management

This includes 3 main outputs:

• Performance monitoring covers all the KPI reports, and any other type of financial reports, meeting reports, building related reports, etc. to determine whether all contractual terms and conditions have been met regarding the delivery of services. This output includes contract issue management, contract amendments, change orders, mechanisms for penalties for non-achievement of the contractual requirements as well as mechanisms for corrective or remedial action.
• Payment certification involves all the management and financial responsibilities and controls required for payment certification and cheque issue.
• Contractor incentive program assessment involves a range of mechanisms to evaluate whether the contractors have pro-actively assisted PWGSC in achieving outstanding results beyond the performance baselines set out in the Contract, such that a contractor can be considered for the CIP.

iii) Relationship Management

This includes 3 main outputs:

• Client/customer consultations can occur to inform or to obtain input from clients/customers/tenants. These can take the form of informal feedback, meetings, interviews, surveys, etc.
• Client/customer guidesare written materials with the intention of informing and clearly outlining processes for working within this AFD Initiative.
• Agreements can take the form of occupancy instruments, service agreements, memoranda of understanding, specific agreements with respect to optional facilities management requests, etc.

Immediate Outcomes for Activity 2

With the appropriate development and implementation of the 3 main areas of outputs, PWGSC expects an overall enhanced AFD contract management process. The extensive performance monitoring and reporting system that is already embedded in the contracts, the new quality assurance protocols and systems that will be developed, and the focus on relationship management will ensure cost effective, timely, responsive delivery of integrated and customized real property services, and enhanced contractor performance and accountability. It is the expectation that assurance of these will permit PWGSC to realize a reduction of PWGSC costs in contract management, contracting, finance, and administration related to the AFD Initiative and that this can provide potential opportunity for an enhanced allocation of resources to permit a greater focus of PWGSC personnel on strategic real property activities. The performance monitoring system, quality assurance protocols, and the requirement for contractors to achieve the ISO 9001:2000 certification will ensure that the Department is receiving quality asset management.

Intermediate Outcomes for Activity 1 and Activity 2

PWGSC expects that the reasonable achievement of the identified Immediate Outcomes for both activities will ensure that this AFD Initiative is delivering safe, healthy, affordable and environmentally sustainable real property solutions and workplaces. Effective relationship management and responsive, customized real property services will result in a high level of client/customer satisfaction. The SOW, with its associated performance monitoring and reporting systems, and the ISO 9001:2000 certification for contractors will result in a continuation of enhanced contractor performance and accountability. It is expected that with this nature of a performance- based contract, associated performance measurement systems and the Contractor Incentive Program, increased leveraging of private sector knowledge and expertise and innovative real property practices will result.

Activity 3: Conduct Independent Reviews of the AFD Initiative

In addition to the ongoing monitoring and day- to- day management of the contractors' performance, PWGSC relies on periodic internal and external reviews and audits to assess the effectiveness of this AFD Initiative. These efforts also provide valuable inputs for downstream options assessments and future design improvements for the Initiative. There are 4 main outputs deriving from this Activity.

Evaluations. Evaluation will comprise the systematic collection and analysis of information on the performance of this Initiative in order to make judgments about its continued relevance, progress or success and cost-effectiveness, and/or to inform future programming decisions about design and implementation.

Audits. Various audits will be conducted over the course of the Initiative, responding to the requests and needs of the Audit and Ethics Committee, the AFD Initiative, and central agencies.

Lessons learned. Based on the results of on-going monitoring and evaluations, PWGSC will be able to apply any necessary changes to the current or future AFD Initiatives.

Special studies. Various special studies may be conducted during the course of this Initiative to address a range of issues, including quality assurance, cost effectiveness, efficiency, and relevance.

Immediate Outcomes for Activity 3

With the results of these reviews available to management, it is expected that improved accountabilities for the AFD process and better reporting of results will occur.

Intermediate Outcomes for Activity 3

Incorporation of the recommendations from these reviews will result in objective, timely information for decision making and an overall improved AFD design and performance.

Final Outcomes for all Activities

The final outcomes are the overall results expected of this AFD Initiative and reflect the rationale for the key activities. Influences of other departmental policies and initiatives are integral in producing these results, and the AFD Initiative is one partner of many that contribute to the Department's ability to demonstrate an enhanced recognition of PWGSC as the GOC's expert in real property management, the maintenance and preservation of government real property asset value, and to realize an optimized return on government investment.

4.0 Evaluation Strategy

PWGSC requires, on an on-going basis, information on the following:

• how well the initiative is operating;
• how effective the monitoring is;
• how satisfied the client/customers/tenants are with the quality and timing of service; and,
• whether the AFD Initiative is meeting its performance targets.

Much of this information will come from administrative data, existing program files, and opinions of key stakeholders. However, attention is required to ensure that the data collected is organized in a way that supports management and evaluation needs.

The Evaluation Strategy component of the Evaluation Framework for the AFD Initiative consists of the following four subsections: 1) evaluation approach; 2) evaluation issues; 3) identification of data requirements and 4) identification of performance indicators.

4.1 Evaluation Approach

The AFD Initiative is significant to the PWGSC business strategy and would best be evaluated at critical points in its development. It is proposed that a two phase Interim Evaluation be conducted in the start up period to ensure that effectiveness is not comprised and to obtain preliminary information on the Initiative's performance, and that a Final Evaluation be conducted early in year three to assess longer-term effectiveness before the end of the four-year base contract period. An estimate of evaluation costs is presented in Appendix A.

4.1.1 Interim Evaluation

Putting in place this new Initiative is complex and care will need to be executed to avert obstacles during the early stages of implementation. PWGSC will require early feedback to ensure that effectiveness is not compromised and to determine whether the Initiative is being carried out as designed and intended. It is recommended that an Interim Evaluation be conducted in the start up phase that would focus on the identification of any problems being encountered which may compromise the initiative's effectiveness, and determine corrective actions. This Interim Evaluation would take place in two phases. Phase 1 would occur between periods 4 to 6 of year one and address whether transition plans and activities resulted in a smooth changeover for clients/customers/tenants, and whether the appropriate management instruments (including both Governance and Accountability Structures) were in place to permit contracts to operate within the intended strategic context. Phase 2 would take place in periods 12 to 14 at the end of the first fiscal cycle and would examine how well the Initiative is taking ground. Issues such as whether the AFD Management Framework put in place is effective, whether appropriate data/information are being generated from the monitoring, reporting and quality assurance systems and whether relationships are operating as intended, would be the focus of Phase 2.

4.1.2 Final Evaluation

The final evaluation would address how well the Initiative has achieved its objectives (success), whether it's program rationale has continued relevance, whether the Initiative has proven to be cost effective, and whether there are more appropriate alternatives to consider to deliver the specified services. In order to support departmental analyses of the initiative and facilitate informed decisions about extending the contract for the optional six-year period (renewal options for the contract need to be exercised 365 days prior to the end of the initial four year term), a recommended start date for this evaluation is early in year three.

4.2 Evaluation Issues

TBS categorizes evaluation issues as follows:

• Rationale/Relevance;
• Program Design and Delivery;
• Success
• Overall Cost Effectiveness and appropriate alternatives

Further descriptions of the evaluation issues follow:

Rationale/Relevance

The evaluation questions associated with this issue will help provide insight into the Initiative's consistency with government and department wide priorities. Addressing this issue from the federal government's perspective requires input from a variety of stakeholders including PWGSC, central agencies, clients/customers/tenants as well as the private sector industry.

Program Design and Delivery

Evaluation questions on the design and delivery address the AFD Initiative's implementation processes and compare its design and delivery with accepted practices/ procedures. Issues and questions in this area address the clarity of the AFD Management Framework; roles and responsibilities; adequacy of resourcing; adequacy of the quality assurance process; and the transition from current contract to new contracts.

Success

Evaluation questions related to success address the results achieved or the degree of progress toward the attainment of the Initiative's objectives and intended outcomes.

Cost Effectiveness

Evaluation questions concerning cost effectiveness address whether government investment was optimized, whether this initiative can be considered cost effective, and whether there are appropriate alternatives that could be considered for the provision of these real property services.

Exhibit 1 lists the specific evaluation questions to be addressed for each evaluation issue, as linked to the Interim and /or Final Evaluation.

Evaluation Issues and Questions - Exhibit 1

4.3 Data Requirements

Data collected at the beginning and throughout the life of the Initiative will be required to properly manage this AFD initiative and to provide a basis for meaningful studies, evaluations, and audits. The data collection methods and sources for the AFD Initiative include the following:

• Document and file review;
• Administrative data review;
• Consultations with key stakeholders; and,
• Information generated by systems.

Document and file review

Document and file review includes studies, planning documents, position papers, research reports, performance frameworks, progress reports, central agency documents, previous evaluations and audits, etc.

Administrative data review

Administrative data includes data collected on a day-to-day basis by the initiative. This includes correspondence, individual or project file information, monitoring data and reports, financial data and reports, etc.

Consultations with key stakeholders

Consultations include interviews (face to face or by telephone; one on one or in groups), focus group sessions, surveys (formal or informal; written or verbal), on site visits and observations, expert opinions, etc. Consultations can occur with individuals and groups who have an interest in the evaluation or are affected by the evaluation. Key stakeholders may include personnel directly involved in the initiative, other government employees, decision-makers, central agency employees, members of the industry, contractors, etc.

Information generated by systems

Systems include electronic or manual systems that contain performance, financial, condition, or status data.

4.4 Identification of Performance Indicators

Performance indicators need to be identified which will show whether an output was produced or a specific outcome was achieved. These can be quantitative or qualitative in nature. Wherever possible, data that is being regularly collected by RPB and other Branches to manage the program and to monitor and report progress should be used, with multiple lines of evidence. Performance indicators are identified in order to gather data to respond to each evaluation issue/question. Exhibit 1 summarizes the suggested performance areas/indicators that support each of the Evaluation Issues and Questions for the AFD Initiative.

5.0 Recommendations

It is recommended that:

1. The ADM, Real Property Branch, accept this evaluation framework and the strategy outlined as the basis for undertaking future evaluation work, including that the proposed final evaluation in year three would chart out any subsequent evaluation work, should it be PWGSC's intention at that time to extend the contract past the four-year base period.

2. The ADM, Real Property Branch, ensure that performance information required for future evaluations, as specified in Exhibit 1 of this evaluation framework, is incorporated into Real Property Services information processes and collected on a timely, ongoing basis.

3. The ADM, Real Property Branch, in conjunction with the ADM, Service Integration, the ADM, Acquisitions, and the ADM, Finance, Accounting, Banking and Compensation, identify all component costs associated with this AFD Initiative and establish and implement any necessary mechanisms required in order to collect and compile all significant costs of this AFD Initiative.

Appendix A: Estimated Evaluation Costs

The estimated costs for the proposed evaluations provide a general order of magnitude in terms of the level of effort to carry out the evaluation work. The figures below may be adjusted depending on the exact nature of the coverage required at the time the scope of the evaluation work is being developed.

Using the Audit and Ethics Branch (AEB) current service delivery approach, the cost of professional services and associated internal AEB person-days are gauged to be approximately:

i) $100,000 in professional services and 80 AEB person-days combined for the two-phased Interim Evaluation;

ii) $200,000 in professional services and 100 AEB person-days for the Final Evaluation in year three to assess the overall Initiative and to support and facilitate informed decisions about extending the contract for the optional six-year period.

Appendix B: Documents Reviewed, Key Stakeholder Interviews and Workshop Participants

Documents

• Key documents related to the current and the AFD Initiative to commence April 1, 2005 were reviewed
• Key pieces of correspondence were reviewed.

Key Stakeholders Interviewed

• Béland, Claude, Director of Expenditure and Management Reviews - Capital Asset Review, TBS
• Billings, Jane, Assistant Deputy Minister, Acquisitions, PWGSC
• Boissoneault, Jean, Regional Manager, Asset and Facilities Management, Quebec Region, PWGSC
• Brazeau, François, Director General, AFD New Procurement, Real Property Branch (RPB), PWGSC
• Campbell, Mark, Director, Client Service Team, Service Integration Branch, National Capital Area, PWGSC
• Chiasson, Marcel, Senior Director, Expenditure & Management Reviews, TBS
• Cormier, Alphonse, Assistant Deputy Minister, RPB, PWGSC
• Couture, Normand, Regional Director General, Quebec Region, PWGSC
• Davy, Gerry, Regional Director General, Ontario Region, PWGSC
• Edwards, Rosalee, Manager, Strategic Business Advisory Services, Resourcing & Strategies/Strategic Initiatives Directorate, PWGSC
• Gagné, Régis, Director, AFD Operations, AFD New Procurement, PWGSC
• Hawkes, Mike, Director General, Financial Management, PWGSC
• Hum-Hartley, Sue, Director General, Property and Facilities Management, RPB, PWGSC
• Jackson, Laura, Director General, Resourcing and Strategic Management, RPB, PWGSC
• Jen, Shirley, Director, Asset and Facilities Management Directorate, RPB, PWGSC
• Lemieux, Sylvie, DG, Accommodation and Portfolio Management, RPB, PWGSC
• Lorimer, Bruce, DG, Parliamentary Precinct & Major Crown Projects Sector, RPB, PWGSC
• McGrath, Tim, Executive Director, RPB, PWGSC
• Mongeau, Pierre Marc, A/Regional Manager, Quebec Region, PWGSC
• Peden, Heather, Regional Director General, Western Region, PWGSC
• Pownall, Wayne, Manager, AFD and Quality Management, AFD New Procurement, RPB, PWGSC
• Saint Pierre, Liliane, Director General, Service Integration, NCA representing herself and John Shearer, ADM, Service Integration Branch, PWGSC
• Strong, Garnet, Acting/Director General, Architectural and Engineering Services, RPB, PWGSC
• Trépanier, Alain, Senior Director, PWGSC Portfolio and Services Division, Government Operation Sector, TBS
• Willmer, Dwayne, Regional Director, Architectural and Engineering / Asset and Facilities Management, Western Region, PWGSC

Note: Position titles are reflected as per date of interviews.

Workshop Participants

• Campbell, Mark
• Gagné, Régis
• Herd, Kathy
• Hum-Hartley, Sue
• Jackson, Laura
• Jen, Shirley

Appendix C: Acronyms