Audit Report 2003

IMPLEMENTATION OF THE PROJECT APPROVAL AND MANAGEMENT FRAMEWORK
PROJECT #03/04 01-04

Prepared by
Audit, Evaluation and Review Directorate

March 2004

	Executive summary
	1.0 Introduction   1.1 Background   1.2 Objectives and scope   1.3 Terms of reference
	2.0 Status of the revision of the PAMF
	3.0 Implementation of the PAMF   3.1 Projects subject to the PAMF   3.2 Implementation of the project approval framework   3.3 Implementation of the project management framework   3.3.1 Relevance of policies, procedures and practices   3.3.2 Assignment of responsibilities
	4.0 Implementation of the risk management framework
	Appendix A - List of projects subject to the PAMF
	Appendix B - Management action plan

EXECUTIVE SUMMARY

The objectives of this audit project consisted in ensuring that the policies, procedures and practices (PPPs) connected with the Project Approval and Management Framework (PAMF) have been brought up to date in response to the observations and recommendations arising out of audits carried out in 2002 by the Audit, Evaluation and Review Directorate and the Office of the Auditor General. Additionally, this audit project was designed to ensure that our PPPs are operational and systematically applied to all Agency projects, particularly with respect to risk management.

The Project Management Group conducted a revision of the PAMF by developing an action plan that integrates the findings of the abovementioned audits with the results of consultations held with other stakeholders. Considerable effort has been devoted to streamlining the project approval process and clarifying responsibilities and deliverables in each phase. We encourage the Project Management Group to continue its efforts to rationalize and incorporate into its action plan measures for ensuring that all stakeholders in the project management process understand the scope of the Project Approval and Management Policy. Corporate policy must apply to all capital projects, regardless of value, and should extend to all other related activities so as to ensure that we have a management framework that is up to the task of seeing these activities through to a successful conclusion.

Over the past year, more Project Management Group resources were directed to improving the project approval process and implementing the risk management process, rather than to improving the project management framework. For these reasons, confirmed by our review, the situation reported by the Auditor General still persists, in that our project management practices, while generally rigorous, are not applied consistently, and project documentation is not always in compliance with our PPPs. The Project Management Group will therefore have to continue its rationalization efforts with a view to making our PPPs more responsive to the needs of stakeholders and the realities surrounding our projects.

A special review of the project risk management process provided reassurance that our practices and procedures in this area are, generally speaking, effective and followed as they should be.

1.0 INTRODUCTION

1.1 BACKGROUND

In March 2000, the Treasury Board (TB) approved an increase in the delegated project approval authority of the Canadian Space Agency (CSA), raising it from $1 million to $5 million, based on the implementation of the CSA's Project Approval and Management Framework (PAMF). This audit project is the second and final contribution of the Audit, Evaluation and Review Directorate under the plan for implementing the PAMF.

In February 2003, in a proposal aimed at raising the Agency's spending authority in connection with the MVIS project, a proposal that received conditional approval from the TB, the Agency reiterated it commitment to auditing the implementation of the PAMF.

1.2 OBJECTIVES AND SCOPE

Whereas the objective of the first audit project was to assess the effectiveness, efficiency and compliance of PAMF policies, procedures and practices, the second audit focuses on the evaluating the degree to which the PAMF has been implemented.

The second audit has the following specific objectives:

a) To ensure that PAMF policies, procedures and practices (PPPs) have been updated further to:

• the internal audit of project management policies and practices, with a view to assessing the effectiveness and compliance of PAMF-related PPPs (report #01/02 issued in February 2002);
• the December 2002 report of the Auditor General on implementing the Canadian Space Program;

b) To ensure that PPPs related to the PAMF are operational and systematically applied to all Agency projects.

1.3 TERMS OF REFERENCE

Policies of the Treasury Board Secretariat

The Treasury Board Secretariat (TBS) has developed a set of seven policies addressing key aspects of the overall management of federal government projects and acquisitions planning. According to the Treasury Board Manual, "these policies require departments to obtain authority from Treasury Board to ensure that major capital planning and expenditure efforts are consistent with government objectives. Departments, however, remain responsible and accountable for the overall effective and efficient management of their programs and capital projects..." The following four policies, and their objectives, are of particular importance:

• Long-term Capital Plans: To retain or acquire only essential assets required to deliver government programs efficiently;

• Project Approval: To ensure that projects proposed for approval by the sponsoring minister or, where required, by the Treasury Board, receive informed and effective consideration;

  Under this policy, departments must normally submit projects valued at over $1M to the TB. In March 2000, the threshold value for Agency projects was raised to $5M, on the condition that the Agency implements an effective project management framework.

• Project Management: To achieve effective and economical management of projects with visible and clearly established project leadership;

  This policy requires that "Departments are expected to approve sound internal policies, guidelines and practices to be followed by project leaders, project managers and other staff responsible for identifying, planning, approving/budgeting, defining, and implementing projects..."

• Management of Major Crown Projects: To deal with the sensitivity and risk inherent with large and complex projects by elaborating the basic project management policy.

Definition of "project"

The TBS defines a "project" as "a set of activities required to produce certain defined outputs, or to accomplish specific goals or objectives, within a defined schedule and resource budget. A project exists only for the duration of time required to complete its stated objectives."

Scope of application

The relevant TBS policies "apply to any project for which the federal government will be

• the owner of the end-product; or,
• the prime user of the end product; or
• the entity responsible for delivering the end product, including managing a contract or contracts for that purpose."

More specifically, the Project Approval Policy "...applies to projects and to capital, lease and information technology projects..." and the Project Management Policy applies to "... projects and capital projects..." where capital project is defined as "... a project specifically intended to acquire or improve a capital asset."

Agency's terms of reference

To ensure compliance and sound management, in March 2000, the Agency committed itself to developing and implementing a management framework that would meet TBS requirements. The Agency thus developed the PAMF and integrated it into its overall management framework, which covers the selection, approval, planning, execution and monitoring of projects. The PAMF is based on business tools and processes and the assignment of responsibilities. The Agency adopted a Project Approval and Management Policy, which is an official statement of our policies, practices and procedures (PPPs) and the scope of their application. Projects are initiated through a Project Approval Document (PAD), which sets out the roles and responsibilities of participants and must be updated in order to obtain the necessary approval to subsequently carry out each of the phases in the project's life cycle.

2.0 STATUS OF THE REVISION OF THE PAMF

In 2002, an initial audit of the PAMF was carried out by the Audit, Evaluation and Review Directorate (report # 01/02 issued in February 2002). The purpose of the audit was to make sure that the PAMF was complete and in line with TB policies and that workable, easy-to-use project management policies, procedures and practices had been developed internally.

Further to this audit, the Project Management Group identified actions to be taken in response to recommendations and held consultations with all stakeholders in the CSA's project management process with a view to developing a revision plan for rationalizing the PAMF and making it more effective.

The results of these consultations and the actions identified further to the internal audit were incorporated in an action plan (the Streamlining Plan), which helped significantly improve the PAMF over the original March 2000 version. The improvements were mainly designed to clearly spell out the responsibilities and deliverables attached to each project phase and to streamline the project approval process according to the project's dollar value. An important step was taken in May 2003, when the Executive Committee approved a revision of the project approval process and a more precise redefinition of project phases and their corresponding deliverables. Steps were taken in October 2003 to discuss the proposed changes with the TBS with a view to their eventual implementation.

When the second audit was initiated, the Project Management Group was still endeavouring to improve the PAMF in preparation for implementing it for all players in the project management process. These efforts are designed to set out the organization's requirements and guidelines for project approval and management. These efforts must continue in order to adapt the PAMF to and ensure its compliance with recent changes to the Agency's management framework that will be brought about by the new division of responsibilities arising out of the creation of the position of Vice-President, Science, Technology and Programs and the revision of the terms of reference of the Agency's steering committees currently under way (the Executive Committee, the Program Review Advisory Board [PRAB/CCEP], the Core Function Co-ordination Panel [CFCP/CCFC], etc.). These changes will likely have a significant impact on the project approval process.

Further to the internal audit of project management policies and practices (report # 01/02 issued in February 2002), a review of the actions submitted by the Project Management Group showed that the recommendations had been implemented or were in the process of being implemented. TABLE 1 summarizes the actions completed or in the process of being carried out as of November 30, 2003.

TABLE 1

RECOMMENDATIONS

Since the implementation of the PAMF has yet to be finalized and consequently will require adjusting and updating to better define the organization's requirements and project approval and management guidelines, we recommend that the Project Management Group take the following steps:

1. continue the revision of the PAMF by completing the implementation of its action plan while taking into consideration recent events affecting the Agency's overall management framework and the findings and recommendations of this audit report;
2. update PPPs so that they reflect changes made to the PAMF;
3. obtain TBS approval for changes to the PAMF, since the TBS approved certain aspects of it.

3.0 IMPLEMENTATION OF THE PAMF

Before undertaking an evaluation of the PAMF's implementation, we first had to determine which projects are subject to the PAMF. In accordance with the scope of the PAMF's application as defined by the TBS and incorporated in the Agency's policy, the PAMF "... applies to all Capital, Lease and Information Technology Projects conducted throughout the entire Agency."

3.1 PROJECTS SUBJECT TO THE PAMF

On October 28, 2003, there were 16 active projects to which the PAMF applied. The projects ranged in value from $800,000 to $430 million, for a total value of $715.6 million. We identified these projects based on the proposed reference levels held by the Financial Planning Division in the IPS system. Active projects subject to the PAMF are defined in this report as capital projects in Phase A, B, C or D whose total value represents expenses incurred or planned for those same phases since the project began. Appendix A contains a detailed list of these projects.

Although the relevant TBS policies as reflected in Agency policy do not apply to the Radarsat-2 project, the TBS has stipulated that this project must nevertheless be treated as a Major Crown Project (MCP), thereby making it subject to TBS policies and the Agency's PAMF.

TABLE 2 below gives the number and value of projects subject to the PAMF as of October 28, 2003, as well as their degree of completion in terms of the phase reached in the project's life cycle.

TABLE 2 - PROFILE OF ACTIVE PROJECTS

¹ Pre-PAMF projects began before the implementation of the PAMF in February 2000. Each of these projects costs more than $5 million, except for one project valued at less than $1 million.

TABLE 3 shows the levels of annual budgets allocated to the execution of projects. According to budget estimates in the Financial Planning Division's IPS database, planned spending totalled $45.3 million for the current fiscal year (ending March 31, 2004) and $65.7 million for the next (ending March 31, 2005). The following is a breakdown of those estimates:

TABLE 3 - ANNUAL BUDGETARY ALLOCATIONS

The following considerations made it difficult to determine projects subject to the PAMF:

1. Although the definition of "project" was revised to bring it in line with the definition proposed by the TBS, this definition is still not well understood by some stakeholders because of the accounting principles defining capital expenditures. Thus, as of October 28, 2003, there were 73 "projects" recorded as such in the Project Knowledge System (PKS) database maintained by the Project Management Group, whereas the auditor found only 16 to which the PAMF should apply. The Project Management Group will record any project or activity in the PKS as long as their staff is involved in it.

   The PKS database includes activities that are not projects subject to the PAMF, although in some cases the PAMF is used as a management tool for ensuring the successful execution of those activities, for example:

   • grants and contributions programs, such as CASSIOPE;
   • research and development activities, including opportunity appraisals and pre-feasibility studies (Phase 0), such as ALOS;
   • operational (Phase E) or completed projects; and
   • recurrent activities not producing end products that can be capitalized.

   Upon reviewing the Streamlining Plan developed by the Project Management Group, we observed that it had been recommended that application of the PAMF be optional for projects under $5 million in Phase A and projects under $500,000 in Phase B, C or D. Furthermore, in a request to the TBS dated October 2003, it was recommended that the threshold for applying the PAMF be raised from $500,000 to $1 million.

   Under the PAMF, as approved by the TB and officially adopted in Agency policy, the framework is supposed to apply to all capital, lease and information technology projects undertaken by the Agency, be they projects involving space operations or systems or projects related to the Agency's infrastructure, support or operations. TBS project management requirements are intended to foster sound management practices and should not be restricted in any way by the monetary value of a project. Practices may, however, be tailored to project's scale.

2. The PAMF requires that projects be approved at each phase of the development through the PAD process. We observed that the various versions of the PAD were not systematically given to stakeholders, including the Project Management Group. This was especially the case for projects in Phase A and projects in Phase B, C or D for which sectors other than Space Programs were responsible. This situation made it impossible to update data (current phase, financial data, project status, etc) in the PKS.

As a result, the Agency's project management policy is perhaps not being applied to some projects to which it should apply, thus weakening the management and monitoring of these projects. Conversely, the policy and everything associated with it might be applied to activities that do not require such a framework. Updating PKS data would allow the Program Management Group to improve project management resource planning and strengthen monitoring of the PAMF implementation process.

Nevertheless, we acknowledge that, in certain cases, the PAMF is the preferred management and monitoring tool for securing the success of non-capital projects. In the interests of increasing effectiveness and promoting sound management, the Agency would benefit from expanding the scope of its Project Approval and Management Policy.

RECOMMENDATIONS

To ensure that the Project Approval and Management Policy is applied to all projects and activities requiring a rigorous management framework that is in keeping with the project's scale, risks, value and other relevant considerations, the Project Management Group should:

1. ensure that all stakeholders understand the scope of the PAMF's application as currently defined, especially with respect to the distinctions to be made between capital projects and other related activities;
2. revise the Project Approval and Management Policy so as to expand the scope of the PAMF's application, allowing it to be used as a tool for managing and monitoring capital project-related activities that require a rigorous management framework;
3. ensure that the PAMF includes a mechanism giving all stakeholders access to up to date versions of Program Approval Documents at all times;
4. ensure that information in the Project Knowledge System (PKS) database is always useful and accurate by keeping it as up to date as possible.

3.2 IMPLEMENTATION OF THE PROJECT APPROVAL FRAMEWORK

Background

The Agency's has a project approval framework that ensures that, at every phase of a project, there is an approval process in place to determine if the Agency should commit resources to the project or not.

The Project Approval Document (PAD) forms the basis of the approval process. The initial approval framework, as approved by the TB in March 2000, required that the PAD for any project in Phase B, C or D be, depending on the project's value, approved or reviewed by the Program Review Advisory Board (PRAB), the Senior Financial Officer (SFO) or his/her representative and, where matrix support is used, the Core Function Co-ordination Panel (CFCP). For projects in Phase 0/A, the PAD needed CFCP approval only if matrix support was used to carry out the project.

Additionally, a project brief was required for any project over $500,000, and projects over $5 million had to be submitted to the Treasury Board for approval.

Rationalization of the approval process

Further to the initial internal audit carried out in 2002, the Project Management Group revised the PAMF to make improvements, particularly to the project approval process. This exercise resulted in a new PAD-based approval process, which was approved by the CSA's Executive Committee in May 2003. The revised process is set out in TABLE 4.

TABLE 4

(*): CFCP approval required for matrix implementation

Since May 2003, the submission of a project brief for projects under $5 million is no longer required. The PAD is the only approval document required.

The project approval process and the format of the PAD are still being studied, and suggestions for improving them and increasing their effectiveness are still being considered. For example, changes proposed in May 2003 would, under certain conditions, make the preparation of a PAD optional, substituting the PAD with a "PAD Light". However, it would appear that tightening up the approval process has resulted in the abandonment of such options in favour of a mandatory PAD with standardized sections for all projects. Classification of projects would, nevertheless, allow certain sections of the PAD to be filled out summarily.

Organizational changes arising out of the creation of the position of Vice-President, Science, Technology and Programs and the revision of the terms of reference of steering committees (Executive Committee, PRAB, CFCP, etc) are recent events that must be taken into consideration.

At the time of the audit, changes made to the initial approval framework had not yet been formally approved by the TBS.

Observations

The May 2003 review of the revised approval process, including the approval levels presented in TABLE 4, uncovered the following facts:

1. Application of the PAMF and the submission of a PAD are now optional for all phases of projects under $500,000. Although we support the efforts made to make application of the PAMF more effective for users with respect to small-scale projects, the Agency's Project Approval and Management Policy, in compliance with TB policy, requires a project approval and management framework for all capital projects undertaken by the Agency.

   Furthermore, a formal, standardized written document describing a project's scope and content and the planned strategies for carrying out that project must be used to help managers make decisions about projects subject to the PAMF, whatever their value.

2. The revised approval framework says nothing about the responsibilities of the Senior Financial Officer (SFO). As the SFO is responsible for confirming a project's compliance with the Financial Administration Act (FAA), the availability of funds and the effect of these expenditures on the Five-Year Risk Assessment and Source of Funds Plan, we believe the SFO plays an essential role in the project review and approval process.

3. The revised approval framework does not make any mention of the decision makers that must approve and review for recommendation Phase A, B, C or D projects whose value is under $1 million. We believe that all projects subject to the PAMF, whatever their value, should be reviewed and require formal approval to confirm their compliance with CSA strategies and the FAA, their roles and responsibilities and the sources and availability of funding.

4. Apart from the PAD, the revised approval framework does not mention any other formal documentation required for the project approval process, such as the project brief or, for projects over $5 million, submissions to Treasury Board. The lack of any mention of these documents could lead PAMF users to mistakenly believe that such documentation is no longer required.

We also examined PADs from 16 projects (see Appendix A) and made the following observations:

• Instructions for preparing PADs were found to be complete and adequate for the task of assisting managers in the preparation of PADs.

• All PADs examined were complete and contained all the information required under the instructions currently in force, thereby facilitating managers' decision making.

• There was no information in the PADs concerning the accounting treatments associated with financial transactions or the financial coding structure to be adopted. The PAD should include specific information about the nature of agreements with our partners in terms of property and responsibilities connected with our participation. Such considerations determine the accounting treatments to be applied, which parliamentary appropriation to use, etc.

  Information concerning supply and the awarding of contracts are presented in a succinct manner in the section "Implementation Approach", but the strategies used to make decisions were generally not well documented in the PADs.

  These situations are due to the fact that project managers still use a standard format to present the required information in the PAD, based on instructions for preparing PADs that were set in March 1999, before the PAMF existed.

• A review of PAD approvals and recommendations for projects begun before March 2000 shows that, in general, the projects had been approved at the specified level of authority. All projects in Phase B, C or D with a value over $5 million were subject to formal approval by the Treasury Board, with the exception of the SmallSat/MicroSat project, whose PAD was still in the process of being drafted at the time of the audit.

  Nevertheless, as shown in Appendix A, we identified five PADs that did not receive all the required authorizations. Such projects represent 45% of all projects that are subject to the PAMF and began after March 2000. The missing authorizations are set out below in TABLE 5.

TABLE 5

Note 1: The PADs for these projects were discussed by the PRAB and the CFCP but were not formally signed by the representatives of these committees.

This shows that the necessary follow-up on authorizations for projects subject to the PAMF are not always carried out adequately. This increases the risk of decisions being made on the basis of inaccurate information.

RECOMMENDATIONS

To ensure an adequate and effective approval process, the Project Management Group should continue its efforts to rationalize the PAMF, taking into account the following considerations:

1. amend the Project Approval and Management Policy to include a more comprehensive approval matrix that sets out the parties responsible for approving projects or reviewing them for recommendation and identifies all documents required for these purposes;
2. ensure that the approval framework also applies to projects under $1 million;
3. revise the format of the PAD to ensure that all project management-related elements specified in the PPPs are covered, including the accounting treatment and financial coding structure to be used and supply-related information.
4. have the TB approve the changes made to the initial PAMF of March 2000.

We also recommend that all PADs be signed by representatives of the committees involved in the approval process, as set out in the PAMF.

3.3 IMPLEMENTATION OF THE PROJECT MANAGEMENT FRAMEWORK

In her December 2002 report, the Auditor General stated, "The Agency's project management and risk management practices are applied inconsistently in projects over which it has prime responsibility and control. Project documentation, cost projections, and risk assessments do not always meet the criteria set out in the Agency's Project Approval and Management Framework."

Based on our discussions with staff, the documents we consulted and our review of five individual projects subject to the PAMF (see TABLE 6), we observed that the Auditor General's statement still holds true. Even though the practices set out in the PPPs are not always applied as they should be, we identified project management practices that demonstrate that these projects are managed adequately.

TABLE 6

Our review of the five projects consisted in identifying the practices used to manage the projects, based on the tools and deliverables set out in the PPPs. The projects included the following management practices to ensure sound project management:

1. Appropriate definition of objectives and specifications;
2. Use of a Work Breakdown Structure describing all tasks to be carried out;
3. Documentation and monitoring of budgets and establishment of a timetable;
4. Appropriate management of requests for changes and of information related to work carried out;
5. Adequate quality management through the documentation of quality assurance plans and audit tests;
6. Identification, quantification and monitoring of risks specific to the project;
7. Project monitoring and periodic reporting of results through statutory reports to senior management.

3.3.1 RELEVANCE OF POLICIES, PROCEDURES AND PRACTICES

The PAMF is still new to the Agency, and the Project Management Group continues to work on its development. Up to now, efforts have mainly focussed on improving the approval framework and risk management rather than the project management framework. The following observations explain the non-compliance of some practices examined with those set out in the PPPs, and the lack of consistency among projects:

1. There are no minimum requirements or guidelines concerning tools, deliverables or methods for ensuring sound, consistent project management that takes the individual characteristics of projects into account. It would appear that the best practices set out in the PPPs are not necessarily applied to every project. Some best practices, however, are essential elements of sound management and should therefore be considered minimum requirements that project managers must comply with.

2. The procedures manual made available to project managers contains a list of best practices (grouped into nine functions) based on project management theory. In many respects, the tools and deliverables listed there are not suited to the realities and individual characteristics of projects carried out by the Agency. The project managers we met with told us that they have doubts about the use and effectiveness of several specific tools and deliverables, in that those tools/deliverables are generally not documented or are documented in a very limited manner:

   • Project plan
   • Project performance evaluation plan
   • Work authorization system
   • Content management plan
   • Schedule management plan
   • Cost management plan
   • Quality management plan
   • Supply management plan
   • Contractor performance evaluation
   • Project performance measurement plan

   The objectives for sound management that these tools and deliverables are supposed to attain are nevertheless met by other tools and deliverables used, as we observed when we reviewed the projects in question.

3. c) Project managers did not always have access to procedures describing the required deliverables and standards; rather, they only had the theoretical concepts, which often meant that they often had to redefine and adapt tools and deliverables to each project. For this reason, the procedures manual was rarely consulted, and management practices varied from project to project, especially since current PPPs allow project mangers to adapt them to the specific characteristics of a project.

4. d) The PPPs do not indicate if it is the contractor or the members of the CSA project team who should be responsible for providing certain deliverables. The following examples were observed:

   • For several projects, monitoring of the project schedule is carried out by the contractor with respect to his/her own activities, while internal CSA activities concerning project management are not documented in a project schedule or diagram (Gannt chart). Instead, the project manager monitors the project schedule using the milestones stipulated in the contract.
   • The contractor is sometimes responsible for drawing up project, quality assurance and safety plans, while the CSA project team takes care of validating the information provided by the contractor.

   This situation increases the risk that responsibilities of the CSA project team will fall to the contractor, making the Agency too dependent on the contractor's own systems and checks. These systems and checks may not meet the CSA's project management requirements.

5. e) When the Agency developed the PAMF, it did so in a context in which large projects (Radarsat, Mobile Servicing System, etc) were monopolizing our resources. However, the Agency's operational framework has since changed, the emphasis now being on taking on more numerous, but more modest projects. When we reviewed the PPPs, we found that they did not take the new operational framework into account. Apart from the scale of the projects, there are other changes we observed, such as the following:

   Subcontracting

   Current projects are smaller in scale than in the past and rely more on subcontracting, while contractors hired by the Agency, under the supervision of Agency project managers, are assigned important components of project management to work on themselves (for example, project development plans, schedules and test specifications. This increases the risk that project management practices will not be applied in a consistent manner.

   Contractors' roles and responsibilities in the project management process are not addressed in the Agency's PPPs, which instead focus on general management practices for large-scale projects carried out entirely using internal resources. These practices are based on general principles set by the Project Management Institute.

   Similarly, no reference is made of deliverables that contractors are required to produce, be they in the areas of planning or testing or the communication of results throughout the course of the project.

   International partnerships

   Staff members we met with commented that most projects are carried out in partnership with organizations from other countries, and that these projects entail special risks and require a special form of management. This dimension, however, is not specifically addressed in the PPPs.

   Change management

   In an environment in which subcontracting and international partnerships play a dominant role in the project management process, change management requires specific coverage in the PPPs to ensure consistent management of change and the associated risks. Change management concerns the appointment, makeup and responsibilities of committees responsible for change management and the documentation, communication and approval of proposed changes. Such issues affecting change management are not addressed in the current PPPs.

RECOMMENDATIONS

To ensure that the PPPs meet stakeholders' needs in an effective manner, the Project Management Group should take the following recommendations into account in its rationalization efforts:

1. set minimum requirements and guidelines with respect to tools, deliverables and means of ensuring that CSA projects are managed in a rigorous and consistent manner;
2. adapt PPPs, tools and deliverables to the particular characteristics of CSA projects (eg, subcontracting, international partnerships, change management);
3. set and standardize the format and content of tools and deliverables considered to be minimum CSA requirements;
4. identify responsibilities of the project team and contractors with respect to the production of deliverables considered to be minimum CSA requirements.

3.3.2 ASSIGNMENT OF RESPONSIBILITIES

Senior management at the Agency has adopted a number of measures in response to the Auditor General's comments in her December 2002 report, including the assignment and confirmation of responsibilities associated with each project phase. The President reaffirmed that phases B, C and D will be assigned to a single entity, the Space Programs sector, for all space projects, thereby facilitating the consistent application of PAMF procedures and practices and ensuring that experienced managers are assigned to projects.

An analysis of responsibilities showed that, with the exception of projects begun before implementation of the PAMF in February 2000, only three projects subject to the PAMF had a project manager who did not belong to the Project Management Group: ATEN, SmallSat and Victoire. In the case of SmallSat, phases C and D will be handed over to a private contractor through the CASSIOPE contribution program. The Victoire project, a non-space project with an estimated value of less than $1 million, is the responsibility of the Administration sector. In accordance with the PAMF, the Project Management Group was involved in Victoire, providing support throughout the entire approval process. As for ATEN, senior management decided that management of the project should be left to the Space Science sector, as was the case with other projects that had begun before the assignment of responsibilities had been confirmed and had progressed too far to be handed over to the Space Programs sector.

We acknowledge that centralizing all projects (space and non-space) in phases B, C, and D under the responsibility of the Space Programs sector is not desirable, given the expertise that other sectors may possess. For example, a significant portion of the operations of the Security and Facilities Division and the Information Technology Division is carried out in the form of projects, including, on occasion, capital projects.

We are nevertheless of the opinion that, to promote sound management, the entities responsible for carrying out projects must use an adequate project management framework, as prescribed by the Agency's Project Approval and Management Policy. The two entities mentioned above as examples developed their own project approval and management framework. In this audit, we did not attempt to evaluate the compliance of these frameworks with corporate policy in this area.

RECOMMENDATION

The Agency adopted a corporate policy on project approval and management to emphasize the importance of linking it to an appropriate management framework. To facilitate the application of the Project Approval and Management Policy, the Project Management Group should:

1. clarify the roles and responsibilities assigned to sectors and stakeholders. Clarifications should distinguish between the approval process and the management process, depending on whether it is a space- or non-space project, and should also take into account the phases in the project's life cycle;
2. spell out the role and responsibilities of the Project Management Group as office of primary interest in such matters, especially with regard to administration of the Policy, dissemination of knowledge and staff support.

4.0 IMPLEMENTATION OF THE RISK MANAGEMENT FRAMEWORK

In December 1999, the Agency integrated a risk management framework into its management practices in order to provide funding for space programs and cover the associated risks based on Agency reference levels.

Process

The risk management process calls for:

• identifying and evaluating risks affecting Agency programs and projects;
• developing mitigation plans for medium- and high-risk programs and projects;
• identifying sources of funding to cover identified risks;
• monitoring and controlling risks on a regular basis to ensure that risks are developing as predicted in the mitigation plans;
• establishing and maintaining a Five-Year Risk Assessment and Source of Funds Plan showing the funds frozen to cover the risks of programs and projects for the next five years, by service line.

Policies and procedures

Risk management was taken into consideration when the PAMF was created. The PPPs for risk management identify the following procedures and responsibilities:

• The Project Manager is responsible for managing a project's risks, including the tasks of identifying and evaluating risks, preparing a response plan for identified risks and monitoring and controlling risks.
• Each project's risks are documented and updated by the Project Manger using the PAD and the Work Breakdown Structure (WBS).
• Risk are identified according to standardized categories: cost, deadlines, technical performance, security and programmatic risks (this last category includes public relations, human resources, policy and the Agency's mission).
• Information about funds for risk management is provided by Project Leaders and Project Managers, thereby keeping the Five-Year Risk Assessment and Source of Funds Plan up to date.
• Project risks to be included in the Risk Information and Assessment System (RIAS) database and the Five-Year Risk Assessment and Source of Funds Plan are identified by the Project Leader, in co-operation with the Project Manager and Risk Manager.
• Only the PRAB may authorize commitment levels in the Five-Year Risk Assessment and Source of Funds Plan and the freeing up of frozen funds or reserves when risks materialize.
• The Agency's Risk Manager ensures that risk management procedures are applied consistently.

Findings

A review of project risk management showed that the practices and procedures in place are generally followed. We made the following observations:

1. Each individual project's risks are characterized, assessed and quantified using a consistently applied methodology under the supervision of the Agency's Risk Manager.
2. Each individual project's risks are documented in the PAD, regularly revised and made available to the relevant stakeholders through the RIAS database.
3. Quantified and weighted risks in the RIAS are associated with a long-term commitment (through funds frozen to cover identified risks), which is documented in the Five-Year Risk Assessment and Source of Funds Plan under the responsibility of the Financial Planning Division. This commitment is updated regularly and formally reviewed three times a year by the PRAB;
4. A process requiring the PRAB's authorization has been implemented to approve the use of commitments when a specific project risk materializes.
5. Generally speaking, the risks of projects subject to the PAMF are periodically reviewed by management in each sector responsible for managing the projects in question and are documented in the Five-Year Risk Assessment and Source of Funds Plan. The following project risk management activities were noted:
   
   
   • In the Space Programs sector, the risks of projects under its responsibility are reviewed on a quarterly basis by the Consolidated Risk Advisory Board (CRAB) under the supervision of the sector's Director General. The Board was set up for major Crown projects carried out by the CSA and has remained active ever since, carrying out formal tracking of risks associated with projects for which this sector is responsible.
   • In the Space Science sector, although most of the projects for which it is responsible began before the PAMF was implemented, risk management for these projects is carried out using a methodology similar to the one described in the PAMF. Risk assessment, the management plan and risk tracking are carried out in co operation with the contractors responsible for carrying out the work contracted out by the Agency. Risks are identified, and a risk management plan is completed to assess the likelihood and consequences of each identified risk, establish what actions should be taken to mitigate those risks and track developments regarding those risks and the strategy for mitigating them. Financial reserves are established as required and documented in the Five-Year Risk Assessment and Source of Funds Plan.

Nevertheless, we found that, contrary to the Risk Management Framework, the risks associated with certain projects subject to the PAMF were not documented in the RIAS database and consequently had not been reviewed by the Agency's risk manager, this despite the fact that the risks specific to these projects had been duly documented and characterized in the PADs. These projects, all of which were in sectors other than Space Programs, included the following:

TABLE 7

The Project Management Group indicated that these projects had not yet been incorporated in the RIAS database because the risk management process had been applied first to projects in Phase B, C or D for which the Space Programs sector was responsible. Nevertheless, the Group's managers said that the above-mentioned projects would be incorporated in the CSA's risk management process in the near future, allowing us to conduct a formal risk assessment, document them in the Five-Year Risk Assessment and Source of Funds Plan and hold funds to cover identified risks, if need be.

Because of these circumstances, the Agency cannot be certain that all risks associated with projects subject to the PAMF have been quantified and updated or that the funds held, as documented in the Five-Year Risk Assessment and Source of Funds Plan, are sufficient to cover all identified risks.

RECOMMENDATIONS

We recommend that the Project Management Group:

1. incorporate all projects subject to the PAMF into the risk management process to ensure that these projects are assessed in a consistent manner and documented in the Five-Year Risk Assessment and Source of Funds Plan and that the necessary funds are set aside to cover the risks identified;
2. formally document risks associated with projects subject to the PAMF and regularly update such information in the RIAS database.

APPENDIX A - LIST OF PROJECTS SUBJECT TO THE PAMF

APPENDIX B - MANAGEMENT ACTION PLAN