24.19 We were concerned that the existing large IT projects would not or could not make adjustments to reflect this revised framework for some time. Accordingly, we continued the work started last year with a view to underscoring the urgent need for improving the way the existing projects are being managed. Further details on the scope and approach of this year's audit are included at the end of the chapter in the section About the Audit .
24.20 In Chapter 12, we referred to the Treasury Board Secretariat's list of 25 large government IT projects with a combined budget of more than $2.1 billion. Our work has shown that this list is incomplete: it does not include all the projects and all the costs.
24.22 In the three other projects that we reviewed, project risks were not being managed effectively:
24.24 The Public Accounts Committee endorsed the notion of effective project sponsorship in its May 1996 Report. The Treasury Board also endorsed this concept in April 1996 in its Enhanced Framework for the Management of Information Technology Projects.
24.25 We have observed that the chance of success is greater when the project leader, sponsor and manager are trained and experienced, and have proven project management capability ( Exhibit 24.2 ).
24.26 Strong leadership, effective communication and appropriate delegation of responsibility in the TCCCS project helped to mitigate the risks caused by the geographical separation of the project teams and by the complexity of the project. Given the nature of the project, senior management's support to the project team came from the project leader. Overall, the project team in place had a high level of technical expertise and displayed strong management skills. The project team encouraged and created an environment of open and honest communication among all the members of the project and with the stakeholders. There was also a judicious delegation of responsibility to the technical staff.
24.27 By contrast, for the first five years of the CAATS project, the project leader and project sponsor were only minimally involved in the decision-making process. As a result, senior managers did not take a central role in decision making. In December 1994, the Deputy Minister assumed the role of project sponsor, the Assistant Deputy Minister assumed the role of project leader and the project adopted a more businesslike approach.
24.28 CAATS is an extremely complex software development project ( Exhibit 24.3 ) that, like TCCCS, entails managing geographically dispersed teams. However, in the case of CAATS, effective communication and appropriate delegation of responsibility to the project staff were missing prior to December 1994. Although improvements are under way, communication is still not effective between the Project Office and the Air Navigation System (ANS), the client group for CAATS.
24.29 The lack of appropriate delegation of responsibility to the CAATS technical staff to resolve technical issues as they arose made decision making cumbersome and slow. In all cases, decisions affecting work being done in British Columbia had to be referred to senior project managers in Ottawa or to Transport Canada headquarters. Beginning in late 1994, a higher priority has been put on expediting decisions.
24.30 In the spring of 1995, the Deputy Minister took an active role and the Assistant Deputy Minister took a hands-on approach by changing the reporting relationship of the project manager to a direct one. This has been instrumental in closing some of the open items and shortening decision cycles. However, problems persist as a more results-oriented management approach is put in place.
24.31 As part of this audit of systems under development, we reviewed the U.S. government's professional development program that offers senior project executives training in the procurement and the management of large system projects. The Treasury Board Secretariat is working on implementing a professional development program aimed at Canadian project managers. Such a program could help create a pool of experienced and expert project managers in Canada. However, we have concerns whether the Canadian program will address the need for qualified project leaders at the assistant deputy minister level to lead large development projects.
24.32 In the U.S., recent policy and practice do not allow a large development project to proceed until a qualified project leader is appointed to lead the project full time. In Canada, there is no such policy. Government departments generally assign part-time project leaders who have no specific training or experience in managing large information technology projects.
24.34 The government runs unnecessary risks when it enters into long-term, fixed-price contracts before a clear understanding of what will be built is reached by all parties. The fixed-price development contract ought to be let only when the government and the developer have reached a common understanding and definition of the requirements, the criteria to be used to determine success and the milestones for the work to be done. Combined with the "manageable component" method described last year, this approach can help avoid the high risks currently experienced. Similar to long- and short-term financial planning, projects need to be planned as a whole; but project phases and contracts need to be planned in more detail for 6 to 18-month planning horizons to allow a higher level of control in achieving results.
24.35 We selected two projects to illustrate what can happen when a contract is awarded before requirements are clearly understood.
24.36 In the case of the Canadian Forces Supply System Upgrade, $15 million was divided among three firms for what was called a "funded definition phase". This approach was a deliberate risk reduction strategy to enable potential contractors to develop a full understanding of a very complex requirement and thus develop precise proposals for the solution. The three firms worked competitively and concurrently to prepare their bid. The winning firm received an additional pre-implementation contract of $2.5 million to update its implementation proposal. However, this process of using a series of contracts to define the project failed to produce a clear set of requirements.
24.37 After being awarded a four-year development contract, the winning contractor needed the first nine months to confirm with National Defence its understanding of the requirements. By the end of that time, it had become clear that the project would take longer and cost more than expected unless the requirements were reduced. Establishing and confirming the system's requirements over shorter periods of time and identifying the specific deliverables before awarding the final contract would have helped the Department design a better contract and establish better development plans.
24.38 In the case of CAATS, the project evolved from another project started in 1986, the Flight Data System Modernization Program. The contract was signed in 1989. A year later, at the first major milestone, the contractor presented a description of what the new system would look like and do. However, Transport Canada rejected the document because it did not correspond with its own understanding of the requirements.
24.39 In June 1994, the contractor and the Department had not yet reached a common understanding of the requirements and, as the requirements for the latest milestone had not been met, the government stopped all progress payments ($230 million to that point). In May 1995, an agreement was reached on a reduced set of requirements and new approval was obtained from the Treasury Board in September 1995. Resolving a number of high-risk functions and other items that were removed from the contract is now the Department's responsibility. A further description of the renegotiated contract can be found in Part II - CAATS: Some Matters of Concern.
24.40 In May 1996, six years into the contract and two years before the system's final delivery date, there are still 11 unresolved issues relating to some operating concepts and to what will actually be built according to the renegotiated contract.
24.41 In both the CFSSU and CAATS projects, many dedicated people in the project management offices have struggled to keep the projects moving ahead, only to have to redo their work because requirements weren't understood or were poorly communicated. Clearly, all parties must thoroughly understand the system requirements as a prerequisite to managing the risks associated with any software development project. While it is true that requirements can legitimately change over time and that some cannot be detailed in advance, it is important that all parties agree to specified requirements for reasonable and realistic time horizons. Action plans can be created to deal with any vague or outstanding need that extends beyond predictable time horizons.
24.43 What limited success there has been in developing government information technology projects has been the result of outstanding contributions by individuals. However, when those individuals leave, the ability of an organization to repeat the success is often lost. By contrast, a "mature" organization possesses an organization-wide ability to manage software development, software acquisition and lifetime software maintenance. Its success does not depend on chance or on individuals.
24.44 Mature software development organizations work hard to implement standards of quality that are well accepted by industry and governments, such as the International Organization for Standards models (ISO 9000) or, more specifically in the area of software development, the Software Engineering Institute's Capability Maturity Model (CMM). The Treasury Board Secretariat has included in the Enhanced Framework for the Management of Information Technology Projects the requirement to use the Capability Maturity Model ( Exhibit 24.4 ). The Secretariat sees the Model as the cornerstone of an approach to identifying and managing project risks in software development projects. In the longer term, the Model may help the government to improve software development capability by supporting continuous improvement in development processes. In the Framework, the Secretariat further suggests that the achievement of a specified maturity level eventually become mandatory for government departments and contractors. As organizations meet higher standards of quality in their development processes, the probability of obtaining the intended results in large information technology projects is increased.
24.46 Typically, the next variable to be compromised is time. Either the time allowed for completing the project is increased or the project is implemented in stages or phases, which, in effect, extends the original completion date.
24.47 The third variable, cost, is usually the last to be compromised because the Treasury Board must normally approve any change in funding. Attempts to adhere to the project budget often translate into more compromise of time and requirements.
24.48 To adhere to the original project budget approved by the Treasury Board, costs are often adjusted by transferring direct project costs to other departmental budgets. For example, when the CAATS project was redefined in 1995, Transport Canada assumed a number of responsibilities that previously had been part of the CAATS contract. Equipment that was to be purchased as part of the original contract amount is now to be leased by Transport Canada at a total additional cost of $176 million over seven years. These new costs will be paid out of departmental budgets.
24.49 All the projects that we examined faced critical decisions on the compromises necessary to complete them. The time to decide which is the most important element - requirements, time or cost - is when the project starts, before major contracts are signed. Based on our work to date, single, large, lengthy, fixed-price contracts are not suitable for large information technology projects because they appear to limit management's ability to adjust to changed conditions. There is an urgent need to improve the procurement and contract processes for information technology projects.
24.50 As reported last year, one way to achieve more flexibility is to break down large projects into smaller, more manageable components. This position was endorsed by the Public Accounts Committee in its report on Chapter 12 of our October 1995 Report and by the Treasury Board Secretariat in its Enhanced Framework.
24.51 Implementing large projects in smaller pieces must be well managed. For example, the set of 11 projects in Real Property Services (RPS), when taken together, represents a very large system development effort. Individually, each project constitutes a manageable component of the whole. However, the benefit of dealing with the entire set of projects in smaller, more manageable components was lost when, for reasons of time and of funding from operational budgets, all the projects were run concurrently. A shortage of experienced and expert staff resulted. RPS senior management has determined that the implementation date (time) of the system is the highest priority, cost is the second priority and requirements is where the greatest flexibility can be exercised. Unfortunately, those priorities were not made clear to all the stakeholders and project teams. When the project priorities are not clearly understood at the start, time and effort are wasted later on clarifying understandings, reducing overlaps and reconciling organizational agendas.
24.53 In 1992, the Treasury Board required all project leaders to use a project performance management system to manage all major capital projects (projects larger than $100 million). In 1994, this requirement was extended to all projects. We note that in 1993, government and industry adopted C/SPMS (Cost and Schedule Performance Management System) as the national standard. It conforms to international standards and provides a balanced measure of performance by evaluating performance against both planned cost and schedule. The Treasury Board is strongly recommending that all project leaders use C/SPMS.
24.54 In the case of the CAATS project, progress figures were produced as required under the 1995 renegotiated contract but management did not use the data as a tool for managing the project (for reasons outlined in paragraph 24.61). In Real Property Services, progress indicators such as C/SPMS were not used. Two other projects that we reviewed had mixed experiences with the use of performance data.
24.55 The CFSSU project adopted the C/SPMS standard. Management's analysis of the monthly performance figures showed rising costs and a slippage in schedule. Efforts were made to take corrective action; however, senior management did not recognize the full implications of these figures until our audit confirmed the project management's analysis of the situation. The trends in the data showed that the project would take twice as long and cost over $100 million more than expected if the project were to continue to completion with productivity at the current level. However, the trends showed productivity declining rather than staying constant. This indicated that the project would never be completed unless management took significant corrective action. At the time of our audit, project management was looking at alternative ways to meet the project objectives.
24.56 The Treasury Board Secretariat received the same monthly reports, as a courtesy, from the CFSSU project manager. Secretariat officials attended senior project meetings. However, the Secretariat considered the information gained from the meetings and reports to be "unofficial". Treasury Board Secretariat policy requires that the project leader formally communicate project performance information to ensure that accountability for project delivery clearly remains with the department. Despite having the information and a responsibility to monitor large information technology projects, the Secretariat did not assess departmental performance in meeting the objectives of the major capital projects policy, nor did it escalate the issue of the schedule slippage to the Treasury Board ministers. The Secretariat views its monitoring responsibilities as related to the effectiveness of its policies rather than to project management, which is a departmental responsibility.
24.57 In the case of TCCCS, the project predated the adoption of C/SPMS. Nevertheless, performance data conforming to the standard were provided by the contractor to National Defence.
24.58 The system that generated the performance data for TCCCS was separate from the one used to generate data for internal management purposes. As a consequence, the reporting system and the management system often presented different views of the status of the project. Efforts to change the system would have been costly and the overall project might have been 75 percent complete by the time the changes were made. Consequently, because the reported data were incomplete or contradicted other sources of information, the TCCCS management team had to develop alternative, time-consuming methods of tracking progress. As a result, management obtained performance data that were over two months old by the time they were assembled, which compromised their usefulness for decision making. The TCCCS management team compensated for the time lag with an ongoing analysis of the various project risks it had identified and with technical reviews.
24.59 Two other major software projects that we examined, CAATS in 1996 and ISPR (the Income Security Program Redesign) in 1995, were major capital projects that used the C/SPMS standard but only partially. In both cases, performance was tracked in relation to schedule but neglected in relation to cost. In the case of CAATS, Transport Canada is not using the cost performance indicators, and the method used to calculate the figures for schedule performance has severely compromised their usefulness for management purposes.
24.60 Cost performance is a critical indicator. It can show management that resources are being spent at a rate that will put the project under severe financial pressure before it is completed. We have observed that fixed-price contracts created a false sense of security for some managers and they paid little attention to cost performance indicators. A fixed-price contract does not necessarily mean that costs are managed. Project costs can increase dramatically while contract costs remain fixed. As mentioned in paragraph 24.47, financial pressure can cause compromises in the schedule or in what the final product can do.
24.61 In the CAATS contract renegotiations of 1995, the Crown agreed that the previous five and a half years of work would be deemed to be 100 percent "on time". This approach introduced severe distortions that have hidden the true schedule performance since July 1995. Even serious slippages in the schedule cannot be clearly identified until major deadlines are missed. Status reports from the prime contractor indicate that the CAATS project is very unlikely to be delivered on time and at currently specified cost and requirements (as per October 1995 amended contract). This is further confirmed by contract negotiations under way as of August 1996.
24.63 The first challenge for the Treasury Board Secretariat is to determine the actual size of investments in information technology projects. The Secretariat does not have a comprehensive list of all projects and their full costs. Only two of the systems that we examined this year are on the Treasury Board Secretariat's list of 25 projects: CFSSU and CAATS. The cost of these two projects totals $953.7 million, as approved by the Treasury Board. However, cost is presented only as $710 million on the Secretariat's list, or 26 percent of the total $2.7 billion. The other two projects that we examined, TCCCS and RPS, were not on the list. The Treasury Board approved TCCCS at $1.9 billion, including over $1 billion in communications equipment. The set of projects in RPS has a value of at least $50 million, by our estimate, but has never been submitted to the Treasury Board. A complete listing of all government projects with an estimate of the full life cycle costs does not exist. This information may be useful to inform Parliament of the total investment in these projects, as recommended by the Public Accounts Committee.
24.64 We estimate that technology projects in government represent well over $5 billion in investments over the life of these projects. Using the statistical findings of a recent study in its revised Framework, the Treasury Board Secretariat estimates that changing the way these projects are managed could help save over $1 billion in cost overruns on the $2.7 billion worth of projects on its current list. Assuming the same factors were applied, there could be possible savings of close to $2 billion on the current $5 billion worth of projects.
24.65 The Treasury Board Secretariat plans to develop the policy changes and the government-wide practices required by the Enhanced Framework by April 1997. As of May 1996, departments had been directed by the Treasury Board to implement the new Framework over the next few years. However, we are concerned that the lack of clear departmental action plans will allow many large information technology projects already under way to lose the benefits of the new Framework over those years. Until the Framework is firmly established in the departments, the Secretariat will need to provide support and monitor compliance very closely if the government is to reap the expected benefits.
24.67 Since we reported last October, the development of the Common Departmental Financial System (CDFS) has been concluded and the system is being implemented throughout Public Works and Government Services Canada. In addition, Correctional Service Canada and eight small agencies have adopted CDFS as their financial system. Adjustments are still being made to the system to make it fit better with other existing systems and to improve its performance.
24.68 The Integrated Departmental Financial and Materiel Management System (IDFS) has been implemented at Transport Canada. In addition, a group of users has been formed and is aiming to upgrade the system by April 1997. The cluster group includes Transport Canada, the National Film Board, the National Research Council, Environment Canada, Fisheries and Oceans and the Vancouver Airport Authority.
24.69 As reported last year, the Public Service Compensation System (PSCS) was terminated.
24.71 In our audit this year, we focussed on six key factors that we believe greatly influence the success of systems under development in government:
24.74 We are pleased that the Public Accounts Committee has taken a keen interest in the results of our 1995 audit of systems under development. The Committee stressed three areas for action in systems development projects:
24.76 To support the work started by the Treasury Board Secretariat and continued by the Public Accounts Committee, the following initiatives should be taken by the Secretariat and the departments:
Departments are implementing the Enhanced Framework for the Management of Information Technology Projects approved by the Treasury Board. Work is proceeding in a review of the information technology procurement process, in developing a project manager and project leader development and support program and in identifying improved practices, methodologies and tools for managing projects, including project size, project review, monitoring and approval, risk assessment and management, and change management. As mentioned by the Auditor General, the Software Engineering Institute's Capability Maturity Model is being studied as a model for government.
Many existing projects, like those audited by the Auditor General this year and last, suffer from inadequate past practices that cannot be fully corrected retroactively and are being carried out under contracts that limit departments' ability to negotiate changes. Departments are managing these projects closely but they still pose high risks. The Enhanced Framework will help departments to manage their existing high-risk projects. However, its primary benefit is expected to be significant improvement in the government's success for new information technology projects.
We appreciate the continuing support of the Auditor General's Office and look forward to its continuing involvement in the work of improving the government's successful completion of large technology projects.
Department of National Defence's response: The Department generally concurs with the content of this report.
For all major projects, the Department assigns both project sponsors and project leaders in addition to the project manager. For some projects, the project sponsor and leader roles may be assigned to the same person. These people normally have extensive project management experience in addition to the formal training provided to all departmental project managers.
On the CFSSU project, the findings of the Auditor General served to confirm departmental management's assessment of the problem areas identified several months before the audit. Senior management, including the interdepartmental Senior Project Advisory Committee, was fully involved and had engaged a highly regarded consulting firm to assist in confirming appropriate corrective action.
Public Works and Government Services Canada's response: The Department generally concurs with the findings of the Auditor General and shares the concerns expressed with respect to the risks of systems under development.
Public Works and Government Services Canada (PWGSC) agrees with many of the Office of the Auditor General's (OAG) conclusions; however, the proposed remedies are less persuasive, as these recommended improvements have been used in the past. The recommendations focus on processes such as reporting, risk analysis, performance indicators, and scorecards; this belies the human factor in the successful completion of projects. The Department believes that competent individuals and project management skills are fundamental to the success of projects. Therefore, the following paths should be considered by the government:
PWGSC continuously aims to improve its software and systems development processes through various initiatives. It is the first department to use the Software Engineering Institute (SEI) software risk assessment methodologies and tools to facilitate systems under development (SUD) audits; furthermore, the full suite of SEI tools is being considered for facilitating audits and for transferring this technology to the business line organization. In addition, PWGSC has adopted this risk management approach to be implemented in each SUD project. The Capability Maturity Model (CMM) mentioned by the Office of the Auditor General was adopted by a departmental branch as a roadmap to improving its software development processes.
More specific to the PWGSC systems reported in the Auditor General's chapter, while there are 11 projects under way within the RPS Systems Framework, only 9 are application development projects. PWGSC agrees that concurrent development presents more risk than if projects are developed one at a time; however, it presents less risk than if the projects had been defined as one mega project. It should be noted that these projects were thought out and planned within a strategic framework, the purpose of which was to mitigate the risk of developing these systems concurrently. Although the timing of the development effort overlapped, it is a result of a need to proceed with these systems within a specified time period in order to meet our business and client needs. In order to mitigate future risks, PWGSC is currently revising its Strategic Information Technology Plan with respect to the RPS systems. At the same time, PWGSC is reviewing the legacy systems and systems under development to rationalize and streamline these initiatives. An initiative is currently under way for situations where departmental staff are involved in a risk assessment audit of a RPS software development project.
The Department recognizes a need for RPS systems to be developed in a timely fashion within a finite budget; however, time, cost and requirements are not mutually exclusive. There are occasions where project timelines may need to be extended to ensure adequate functionality. It is PWGSC's objective to obtain a high-quality product that meets all functional requirements on time and within budget.
With respect to the CAATS project, PWGSC was responsible for the contractual and legal elements of the project.
The renegotiation of the CAATS contract was a complex process that took into account many factors, including contractor claims, outstanding differences between the Crown and the contractor and changes in scope of the work package. As a result, the amended contract was not prepared on the basis of integrating all of the elements in the original contract into the new contract. Rather, it was structured based upon available funds and a redefinition of the required work.
While the cost of putting the contract in good standing could have been presented in a different manner for greater clarity in documentation, PWGSC fully described the complex nature of the negotiation exercise to the government. PWGSC did, to the best of its ability, clearly disclose fully all aspects of the contract restructuring.
24.78 In March 1995, the Minister of Transport met with the Auditor General to advise him of concerns related to the CAATS project that had recently come to the attention of the Minister. He indicated that, in his view, the project merited our attention. This view was later reiterated publicly by the Minister in statements to the House of Commons. The project, which we were already monitoring, was also high on our list of audit priorities, and the conjunction of our priorities and the Minister's request resulted in the decision to do this audit.
24.79 Because the project was one of the larger systems under development in the government, we made an early decision to link our work with that of the Systems Under Development (SUD) audit team. It, in turn, was interested in examining CAATS as a case for its audit.
24.81 Accordingly, we decided to focus most of our audit attention on matters for which the lessons to be learned would remain valid for future government projects of this scale. At the same time, there were certain matters related to the project that lay outside the normal focus of our work on systems under development that we believed would be of interest to Parliament. We report on those matters here.
24.83 These findings led to significant changes in the management of the project. A new Project Director was appointed, the Assistant Deputy Minister (Aviation) was designated the Project Leader and the Deputy Minister assumed the role of project sponsor. From this point on, they assumed ongoing leadership roles in the project, chairing key internal committees and communicating regularly with the senior management of the prime contractor.
24.84 In January 1995, under the leadership of a new Project Director, the Project Office and the contractor began discussions on a possible "rebaselining" or restructuring of the project. In March 1995, at the request of the government, the contractor submitted a proposal indicating that it would require a contract price of $764 million (including the $230 million already spent) to complete the project (the existing contract then stood at $419 million). The requested amount was not acceptable to the government.
24.85 Faced with this situation, the Department believed that it had two options. It could terminate the contract or seek to negotiate a reduced system at an affordable price. An affordable price was one that would stay within the original overall cost ceiling - the effective project approval (EPA) - of $659 million established by the government for the project, and would result in a contract price of not more than $500 million. Should a renegotiated contract be possible, continuing with the project would mean that the Department faced a further cash outlay of $270 million on the contract. The Department believed, however, that there would be a reasonable prospect of receiving a functional system at the end of the contract - albeit, a system with less capability than it had originally hoped to get. Alternatively, if the Department decided to terminate the contract, it believed it faced the prospect of receiving very little of value for the $230 million already spent, and the likelihood of being sued by the prime contractor for an amount equal to or greater than the cost of finishing the project.
24.86 After considering the alternatives, the Department decided to enter into negotiations with the prime contractor. The main objective was to maintain a project that would be operationally sound, while restructuring the contract so that the original overall cost ceiling (the EPA) of $659 million would not be exceeded.
24.87 Those negotiations were concluded in September 1995 with a rebaselined contract. The price for the new contract was $500 million, the total cost of two contracts: the main contract for CAATS with a price of $478 million; and a contract for the voice switch for CAATS, priced at $22 million. The voice switch was part of the original contract and was separated from it during negotiations. When we refer to the post-rebaselining contract, these two contracts are included together.
24.88 Transport Canada reported that, coupled with other changes to the project (discussed in the next section), the revised contract permitted the Department to complete the project while remaining within the overall approval level of $659 million. In Part III of its Estimates for 1996-97, the Department reports the cost of CAATS at this approved value ($659 million) plus an additional $75 million in associated costs for training, installation and leasing.
24.89 Exhibit 24.6 presents a breakdown of project costs. The first column presents the original EPA cost ceiling as approved by the government for CAATS in November 1989. The second column gives the EPA costs as proposed by the Department and approved by the government in September 1995. It also includes certain additional costs to arrive at a fair statement of the total project cost for the reapproved project, calculated on a basis consistent with the original EPA. These additional costs represent two types of costs: first, those costs included in the original contract and thus in the original EPA; and second, costs that must be incurred in order for the CAATS system to operate.
24.90 The most significant item in these additional costs is a pair of leases for the hardware and commercial software for CAATS valued at $176 million (see line 10 and note 3). Under the original contract, the prime contractor was required to design, assemble and supply these items as part of the contract deliverables. Under the renegotiated contract, the contractor was relieved of the obligation to pay for the items as part of the contract and instead the Department was to lease them from a third party. The capital price of the items to be furnished under the main lease was capped at $103 million and the contractor is required to furnish them at that price. According to the Department, should the cost of the items, the requirements of the software or any other changes not explicitly directed by the CAATS Project Office cause the price to rise above the cap, the excess costs would be on the contractor's account. Additionally, the Department has informed us that the lease was structured in three stages in order to minimize the financial exposure of the Crown in the face of both the likelihood of the project being transferred to Nav Canada, and the continued technical risk of CAATS.
24.91 The leased hardware and software are essential to the operation of CAATS and, in our view, the omission of these costs and other like costs from the revised EPA results in the true cost of CAATS to the Crown being understated by $217 million (line 15). In preparing Part III of its Estimates, the Department followed the convention of reporting for Major Crown Projects the following: the effective project approval ($659 million) and only those additional costs that would be incurred to the end of the main contract ($75 million). However, in doing this, the information provided suggests that the renegotiated contract was achieved with no material increase in the cost. As a consequence, this presentation results in Parliament not being fully informed of the real cost of the decision to rebaseline the CAATS project.
24.93 As indicated earlier, one of the major changes in the contract, resulting from the rebaselining, was to take the purchase of the hardware and some related commercial software out of the contract and to set up a lease for the computers, work stations for the controllers and some of the software. In addition, a number of features were removed, such as the ability of the system to propose possible resolutions to potential conflicts between aircraft. The scope of the contract was also reduced significantly by the decision to implement CAATS at 23 towers instead of the 60 originally proposed. The Department believes that the removal of certain of the features, such as conflict resolution, from the contract, had the benefit of greatly reducing the overall technical risk associated with the CAATS project.
24.94 The items removed were presented by the departments to the government as having a price of $227 million (line 14). In arriving at this figure, the price of the hardware removed from the contract was shown as approximately $92 million (note 2). We were concerned about the extent to which that figure represented the correct price of the hardware removed. When we asked for a listing of the hardware prices, based on the price of the items at the time they were contracted for, we were informed that none was available.
24.95 We would have expected that before entering into negotiations to remove the hardware from the contract deliverables, the Project Office would have established the original price of the hardware in the contract and the price of hardware added though contract amendments. We anticipated that this would have been done so that the Department could ensure that it received full credit from the contractor for the deletion. We found that this had not been done and that the Department did not know the original hardware prices.
24.96 Accordingly, we used available information and a variety of approaches to estimate the price of the hardware at the time it was put into the contract. Our analyses yielded a range of estimated prices and, to be conservative, we have chosen to use the lowest figure ($103 million). This means that the price of the items removed from the contract is understated, we estimate, by $12 million (line 19 and note 2).
24.97 The decision to lease the hardware and certain items of the COTS (commercial off-the-shelf) software rather than purchase them through the contract affected not only the price to be removed from the contract, but also the pricing of several of the new engineering change proposals (ECPs) that were added to the contract. We found that while items priced at $19 million (line 20) were removed from ECPs to be leased instead, the price of those items was not removed from the ECP prices (line 2) that were used to justify the contract price. As a consequence, the price of the new ECPs (line 2) is overstated by $19 million in the material presented for the government's consideration.
24.98 At the same time, changes in the approach and in the needs of the Air Navigation Service led to new elements and costs being added to the contract. These included costs such as $54 million to be incurred as a result of changing the approach to the transition from the current control system to CAATS. In total, the cost of these additions to the contract was reported to the government as $364 million (line 5). However, this amount includes $187 million for an engineering change proposal - ECP 232 (line 3). Our inquiries have shown that during the negotiations for the rebaselined contract, a decision was made to reject this proposal.
24.99 According to the Department, ECP 232 is not properly an ECP, but rather is a claim to be paid an additional amount for the costs of work arising from an earlier engineering change proposal. That proposal (ECP 60R4) had been incorporated into the contract as part of Amendment 10 at a price of $12 million. The work related to the preparation of a series of what were called operational requirements definition documents. In ECP 232, according to the Project Office, the contractor was asking for additional funding as the work had become broader in scope than anticipated.
24.100 The Department has informed us that it rejected the ECP because it did not wish to accept any liability for the charges made by the prime contractor in the claim. Accordingly, the Project Office made no attempt to verify any of the financial data in the claim in order to assess whether the work done justified the price asked. Nonetheless, in the interests of continuing the contract, it decided that it wished to credit the contractor in the price negotiations with an amount equivalent to the price ($187 million) claimed in the ECP .
24.101 This item, the most costly by far of the items justifying the price increase, was presented to the government as part of a list of ECPs representing new value to be added to the rebaselined contract. In fact, it was not a new ECP; it was included in the settlement made without admission of liability and represented part of the cost of resolving the differences between the Crown and the prime contractor.
24.102 In sum, our review of departmental documentation including the contract price justification, prepared by the departments, led us to conclude that the justification did not present the full price of the hardware and COTS software removed ($92 million as opposed to $122 million for a difference of approximately $31 million (lines 19 and 20)). Additionally, the contract price justification did not adequately disclose the nature of the $187 million addition to the price attributed to ECP 232. This means that our analysis suggests that the government agreed to a contract price of $500 million on a contract where a consideration of the price of the items removed from the original contract and the cost of the items added would only justify a price of $282 million (line 22 and note 2).
24.103 In our opinion, the material prepared for consideration by the government in approving the revised contract should have clearly shown that the revised contract price of $500 million included a component that represented the cost to the Crown of putting the contract in good standing, settling all past differences and disputes among the parties to the contract, and receiving from the contractor a formal release to that effect. While it is not possible to calculate the exact amount, our review of departmental documentation, the price justification and discussion with management leads us to conclude that the settlement cost was approximately $200 million.
24.106 We observed that throughout the project, the approach of the CAATS Project Office to the design validation was responsive, based largely on reacting to the proposals and questions of the prime contractor. Given that the lead role in the contract lies with the prime contractor, who is responsible for preparing and implementing the design, we would have expected a certain level of responsiveness in the role of the CAATS Project Office. However, the Project Office should also have had a plan to work with the prime contractor to achieve closure on the design issues by clearly communicating the requirements of the air navigation system to the prime contractor. When we asked for such a plan, staff indicated that a strategic plan for the Project Office had never been prepared.
24.107 The CAATS project had been under way for nearly five years in the fall of 1995. Yet at that time there were still six major (and many more minor) unresolved design issues. According to the prime contractor, the failure to achieve closure on these issues had significant effects on project progress. The unresolved issues included matters as fundamental as the interface between the radar data processing system (part of the existing and ongoing Air Navigation Service) and CAATS. Given that one of the fundamental purposes of the CAATS project was to integrate the information from the radar data processing system with flight plan information, the failure to resolve such a central issue for the whole system, and similar issues, was an indication of the problems of the Project Office in achieving its objectives in design validation.
24.108 To respond to this problem, the CAATS Project Office began, in late 1995, to establish what it called integrated product teams. This involved assigning Project Office staff to work in tandem with their prime contractor counterparts to try and expedite the resolution of key design problems.
24.109 We observed that in the period since rebaselining, the CAATS Project Office has had some success in resolving many, but not all, of these design problems.
24.111 Performance monitoring . The prime contractor has put in place a project performance monitoring system called an "earned value system". According to the system description provided to the Project Office by the prime contractor, the contractor's earned value system has most of the critical attributes of the types of systems recommended by the Treasury Board Secretariat for such projects. The Project Office receives from the contractor monthly reports on progress against plans for the project as a whole and for subcomponents of the projects. In addition, individual managers receive progress against plan information on a weekly basis. However, the overall project information provided to us by the Project Office was often between six and eight weeks out-of-date by the time it was received and the weekly information appeared to be two to three weeks out-of-date. Project Office staff informed us that they have performed extensive checks on the reliability of the information coming from the system and are satisfied that the information is sound.
24.112 Risk assessment . We were informed that before rebaselining, risk analysis was not part of the management process in the Project Office. However, since rebaselining, the Project Director has appointed a risk manager who is producing a monthly series of risk analyses of the project.
24.114 Under these circumstances, control by the Project Office over the milestones was critical to ensure some degree of correlation between proven progress and the Crown's growing financial exposure. However, problems with milestones were evident from the start of the contract. Milestones were consistently allowed to be passed with significant lists of outstanding items attached. Indeed, some items were passed from milestone to milestone without ever being resolved.
24.115 The Department was to complete a checklist to assess the completion of each milestone. For the first milestone, System Design Review (for which $24 million was paid by January 1991), we were informed that the Department did not use the 38 pages of checklists required for this review, but paid for the milestone nonetheless.
24.116 A similar situation arose at the second milestone. Its checklist shows that half of the milestone, relating to interface requirements, was not completed. However, a certificate of completion was issued and the 20 percent milestone payment was made.
24.117 This practice leads to a "bow wave" effect whereby each successive milestone becomes more difficult to achieve than was intended, and the list of postponed work grows, increasing the likelihood of project failure. At the same time, the gap between the actual value of the work achieved and the amount of work the Crown had paid for grows as well, further increasing the risk of project failure.
24.118 On four significant occasions prior to the rebaselining, 75 percent of the milestone value had been paid for, yet the milestones were not delivered. On each occasion, the government stopped making payments to the contractor. However, the resolution in each instance was ultimately to amend the contract and to restructure the milestones by shifting more of the requirements from the current to future milestones, thereby permitting payments to resume and the project to proceed.
24.119 The initial stop-payment was issued in October 1991, when the prime contractor had reached the 75 percent payment limit of the second milestone. At that time, some $56 million had been paid. The stop-payment was issued in October 1991 and the second milestone was not met until November 1992 when $16 million, representing the balance of its assigned value (less holdback), was paid. This payment was made possible only because of an amendment to the contract that changed the milestone schedule.
24.120 From November 1992 on, the CAATS Project Office began a pattern of paying for one milestone and, at the same time, paying 75 percent of the next milestone because the prime contractor's progress claims already exceeded that amount. An example is the case of the second milestone, as discussed in paragraph 24.119. In November 1992, a payment of $16 million was made based on its completion. At the same time, a payment of $9.1 million was made representing progress claims for 75 percent of the price of the third milestone, which was also eventually late. This payment process continued until 1994 when, after payments had again been suspended, it was decided that the whole contract had to be renegotiated. The cumulative amount paid then stood at $230 million, out of a contract price that was then $419 million.
24.121 Milestones restructured . Under the rebaselined contract, milestones remain the critical point for payment control. As before, each milestone has a pre-established value. However, between milestones, progress payments are based on dividing the value of the milestone into roughly equal monthly amounts. These amounts are then modified by applying an index intended to reflect progress in the work. If progress for the payment period is less than planned, the payment is reduced and the amount of the reduction is held back until the milestone is completed. Once a milestone is missed, payment is stopped until the milestone is completed. In lieu of a cash holdback, the rebaselined contract requires the prime contractor to maintain an irrevocable letter of credit in favour of the Crown in an amount equal to five percent of the cumulative amount paid under the contract.
24.122 Since rebaselining in the fall of 1995, the prime contractor has successfully met a series of smaller milestones. The most recent - milestone 10 - was delivered slightly late, at the end of March 1996, and was deemed complete. However, significant elements of that milestone remain to be resolved. Finally, in the summer of 1996, when it became clear that the next milestone would be missed by several months and that the contracted delivery date of the final product was again in danger, contract renegotiations were begun again. One feature of the discussions was a further restructuring of the milestones.
24.124 Although we discuss verification as the last function of the Project Office, it is by no means only an end-of-the-project task. If proper assurance is to be achieved, we would expect that the verification process would begin in parallel with design validation, and continue through the development of a verification plan by the Project Office. Such a plan would classify program functions by their criticality, establish accordingly the degree of assurance that the Project Office desires from the verification of each function or combination of functions, and establish the actions and tests to achieve that assurance.
24.125 No overall plan. When we asked the Project Office for its plan to achieve such assurance, we were informed that it did not have one. We were further informed that the prime contractor follows a disciplined and well-documented software development process that includes a series of reviews as each unit of code is designed, written and tested. Project Office representatives are permitted to witness all of these activities and do so on a selective basis. The contractor has also developed a series of other plans for testing as program integration takes place, testing once the system is assembled and, finally, testing once the system has been installed. The Project Office representatives review, suggest changes to and approve these plans, and again are permitted to observe all aspects of these tests.
24.126 The CAATS Project Office had not established the desired level of assurance for determining the degree to which the system fulfills the specifications, nor could it show how its review and observations would ultimately lead to achieving such assurance. Accordingly, we conclude that the Project Office has not established the degree of assurance it needs to achieve from its verification activities, nor is it currently able to ascertain what level of assurance it will receive from the activities that it and the prime contractor are, and will be, carrying out.
24.127 Importance of verification for safety-critical systems. Verification is an important activity of the Project Office for any software development project to ensure that the Crown receives value for the money paid. However, the importance of verification rises sharply when the software, or parts of it, are critical to safety. CAATS is a safety-critical system and thus assurance of system safety is of paramount importance. In operation, CAATS will be the controllers' sole source of much of the information they need to safely route air traffic and to prevent losses of separation between aircraft. If that information is wrong, for any reason, then the likelihood of an accident and the potential for loss of lives increases.
24.128 In spite of the fact that CAATS is a safety-critical system, the CAATS Project Office has not been able to provide any evidence that it has initiated any independent safety process or hazard mitigation plan of its own. Although the expertise in air traffic control systems and their safety resides within the Department, we were informed that the Project Office has no documented plan to deal with system safety. While it has a representative on the System Safety Working Group, which is part of the contractor's safety process, the Project Office is relying almost completely on the prime contractor for safety of the delivered system.
24.129 Management support needed. Good safety practices must be supported by top management to foster the culture necessary for an effective safety program. One of the ways management demonstrates its commitment is by appointing a manager with the authority and mandate to ensure that safety is given the necessary attention and priority. In a project of this scope and complexity, we would have expected the appointment of a full-time safety manager, reporting directly to the Project Director.
24.130 Following the August 1995 retirement of the Project Office staff person responsible for performance engineering and system performance, performance and safety responsibilities were transferred to the Software Engineering Manager, and the responsibilities for reliability, maintainability and availability and for quality assurance were transferred to the Systems Engineering Manager. With this division of responsibilities, two individuals working in separate parts of the Project Office have each spent part of their time working on safety-related issues. However, in September 1996, the CAATS Project Office announced the appointment of a Safety Manager.
24.131 Some efforts made. System safety has not been wholly ignored. In 1990, the prime contractor developed an initial safety plan, titled the "Reliability, Maintainability and Availability (RMA) Program Plan". In 1992, the contractor engaged a consultant to evaluate the existing safety of the CAATS project, and to propose a new direction for future work in this area. The consultant's report provided an evaluation of safety for the CAATS project, a short description of state-of-the-art safety analysis, a list of recommendations on how to perform future RMA tasks, and a list of suggestions on how to modify the 1991 version of the RMA Plan. The report also described a number of items that the consultant considered to be responsibilities of the CAATS Project Office and identified specifications that the prime contractor needed from the Project Office in order to define the program safety requirements better. These included such things as: formal specifications for the definition of functions; data flows and data storage; the identification of critical functions with the safety objectives stated for them (by stating controllers' responsibilities, definitions of hazards and severity of hazards, and by performing a hazard analysis on the air traffic control system); a procedure to test the critical functions, verifying that the controllers would actually use the system for its intended purpose; and an "exhaustive" review of documentation to ensure consistent, unambiguous specification of critical functions.
24.132 The CAATS Project Office was made aware of this report in 1993, and we expected that it would have responded by producing the analyses recommended by the consultant. However, the Project Office has not responded in any way to the recommendations.
24.133 The prime contractor's RMA Plan has been amended since the consultant's evaluation, with the most current version (according to the Department) being the August 1995 revision now titled "Safety Case". Our review of that plan suggests that it is quite complete, based on the project description of the day. However, the rebaselining of the project was only partially completed at the time the document was prepared. Several key areas in the document contain notes to the effect that it will need to be revised to reflect the new baseline when that is established. Unfortunately, the rebaselining has substantially changed many parts of the project with the consequence that the 1995 plan will need to be updated to apply to the project as now defined.
24.134 Use of COTS software products increased . The planned use of COTS (commercial off-the-shelf) software products has escalated tremendously as the CAATS project has been redefined. Most COTS software products are designed for non-safety-critical use and come with very limited warranties. Further, many of these products are designed as either stand-alone programs with little, if any, consideration of operability with other applications or as programs designed to operate in a limited environment with a few other programs. It will be extremely important that the individual and integrated responses from performance testing of COTS software products are carefully studied initially and, for each subsequent version released, before implementation in the functioning system. Evaluations of COTS software products must be frequently revisited, since the project duration has already outlived some of the products that were available at the first writing of the RMA Plan.
24.135 Conclusion . The current approach being taken by the CAATS Project Office to system verification and safety appears to be relying very heavily on observing the testing by the prime contractor. Extensive testing of the system may aid in gaining assurance that the required functions exist in the system, but it is unlikely to provide adequate assurance of the installed system's overall reliability and safety.
24.136 The CAATS Project Office has not described any supplementary process to gain assurance that the system will operate reliably, consistently, over a long duration, and at an acceptable level of performance for critical functions.
Transport Canada's response: The Department definitely agrees that CAATS has to date been a very problematic contract. When it was apparent that the CAATS project had encountered serious problems, Transport Canada took decisive action to correct them and is still doing so. While active consideration was given in early 1995 to cancelling the contract, we made a business decision to proceed with a fixed-price contract. This approach was designed to limit costs while still ensuring that there would be a functional product at the end of the contract.
The audit only refers to the role of the Department in managing the contract and does not refer, directly or indirectly, to the role of the contractor. It is the view of the Department that a share of the problems belongs to the contractor. The Department has to date needed to take corrective action not only to get its own house in order but also to ensure timely and effective delivery of milestones and key products by the contractor.
The audit attempts to make a direct comparison of the prices and costs in the contract prior to its renegotiation in October 1995 with the prices in the revised contract. The Department took a fixed-price approach and negotiated a global price within which all aspects of the revised contract had to be considered. The adjustments that had to be made by both sides were defined as a package and, as a result, it is extremely difficult to make a direct comparison of the numbers before and after the rebaselining of the contract.
We acknowledged that the costs associated with putting the contract in good standing could have been more clearly delineated in the contract documentation. However, we believe we fully disclosed the complex changes to the contract, including the settlement of claims.
With regard to the issue of system safety, it is essential to recognize that system safety engineering for air traffic management systems such as CAATS is a new and evolving discipline. As the software functionality is developed, it will be tested against established safety criteria. The bottom line is that the system will not be implemented unless we are assured that it is safe.
In general, we agree with many of the conclusions reached in this audit but we strongly object to the allegations that the facts were misrepresented, and that all the blame seems to belong to the contractee and none to the contractor.
The role of the CAATS Project Office is to have built, on behalf of the Air Navigation Service (ANS), an automated air traffic control system that meets the functional needs of the ANS and that is safe and efficient to use. We have identified three critical jobs that the Project Office must do to accomplish this. First, it must communicate the needs of the client to the prime contractor and assess the contractor's design proposals for their technical and operational suitability and compatibility with the rest of the ANS system. This task is sometimes referred to as design validation. Second, as the government's representative in dealing with the prime contractor, the CAATS Project Office must monitor the progress of the contractor to ensure that it is satisfactory and ensure that funds are paid to the contractor only when they are due under the contract. Finally, the Project Office must be able to assure the Air Navigation Service that the product delivered is safe, fit for use, and fully meets all of the specifications of the contract. This is sometimes called verification.
We examined these areas to be able to report on matters that, in view of the circumstances surrounding CAATS, are of significance to Parliament.
Quantitative information. The quantitative information in this chapter has been drawn from government documents. Historical financial information has been checked for reasonableness but not audited. Cost and future-oriented financial information has been checked against official sources.
For information, please contact Eric Anttila (Part I) or Hugh McRoberts (Part II), the
responsible auditors.