Audit Report 2004

Audit of the Management Framework of the Canada/ESA Cooperation Agreement
PROJECT #04/05-01-01

Prepared by
Audit, Evaluation and Review Directorate

November 2004

	Executive summary
	1.0 Introduction   1.1 Rationale for the audit project   1.2 Objective and scope of the audit   1.3 Background   1.4 Methodology
	2.0 Administrative records   2.1 Audit trail
	3.0 Selection and approval process for optional programs   3.1 Program approval document (PAD)   3.2 Consultation process
	4.0 Contribution accounting   4.1 Commitment of funds   4.2 Advance payment of transfer payments   4.3 Exchange rates   4.4 Financial control of contributions to optional programs
	5.0 Implementation of the RBAF   5.1 Implementation of the RBAF   5.2 Program risk management plan
	6.0 Results-based management and accountability framework (RMAF)   6.1 Implementation of the RMAF   6.2 Content of the RMAF
	Appendix A - audit objectives and criteria
	Appendix B - management action plan

EXECUTIVE SUMMARY

The objective of this audit project is to evaluate the extent to which the systems and procedures used to manage the Canada/ESA Cooperation Agreement in all its aspects are effective, efficient and economical.

The current Canada/ESA Cooperation Agreement was signed on March 28, 2000 and will be in force until 2008-2009, subject to an agreement to renew its terms and conditions effective April 1, 2005. Up to $30 million in contributions will be paid out to ESA each fiscal year throughout the term of the Agreement.

Implementation of the recommendations made in this audit report will allow the manager of the Canada/ESA Cooperation Program to carry out the duties of this position more effectively, especially with respect to compliance with program policies and conditions, decision making and accountability.

More specifically, the audit report makes the following recommendations:

• Complete administrative records for each optional program should be retained to ensure an adequate audit trail and support decision making.
• Program approval documents (PADs), submitted to the Program Review Advisory Board (PRAB) with a view to obtaining approval for optional programs, should contain all the necessary information, as described in the program terms and conditions approved by the Treasury Board (TB).
• A process for holding consultations with industry stakeholders should be implemented with a view to supporting decisions respecting optional programs selected by Canada.
• The rationale for considering the contribution to the General Budget as a generic expenditure related to various activities according to the new Program Activities Architecture (PAA) should be reviewed in line with the actual contribution to optional programs so that more accurate financial information can be obtained.
• Monitoring of accumulated appropriations in ESA accounts should be carried out in early 2005 to ensure that they have actually been used when the next payment call occurs.
• A mechanism for minimizing the risks of fluctuations in exchange rates should be implemented in consultation with the Treasury Board Secretariat (TBS).
• Monitoring of expenditures related to optional programs should be carried out to ensure that cumulative payments match the amounts in foreign currency that the Canadian Space Agency (CSA) committed itself to, thereby allowing the detection of cost overruns.
• The Risk-Based Audit Framework (RBAF) and the Results-Based Management and Accountability Framework (RMAF) should be implemented.

This internal audit was carried out in accordance with the Treasury Board Policy on Internal Audit and the IIA (Institute of Internal Auditors) Standards for the Professional Practice of Internal Auditing. In our professional opinion, the audit procedures followed and evidence gathered were sufficient and appropriate and support the accuracy of the conclusions in this report. The conclusions are based on a review of the situations in question using established audit criteria. The conclusions only apply to the entity examined.

1.0 INTRODUCTION

1.1 RATIONALE FOR THE AUDIT PROJECT

The audit of the Canada/ESA Cooperation Agreement is part of the approved 2004-2005 audit plan. This audit was included in the annual plan further to item 4.2 of the program's Risk-Based Audit Framework (RBAF), which states that an audit of the Canada/ESA Cooperation Agreement shall be carried out in 2004-2005. This review will also be included in the evaluation report required by the Treasury Board (TB) in anticipation of the renewal of the terms and conditions of the program, pursuant to the Policy on Transfer Payments.

This audit project was carried out from July to October 2004.

1.2 OBJECTIVE AND SCOPE OF THE AUDIT

The objective of this audit project consists of obtaining assurances that all systems and procedures currently used to manage the Canada/ESA Cooperation Agreement in all its aspects are effective, efficient and economical.

The audit covered elements of the Canada/ESA Cooperation Agreement's management framework that were in force during the 2003-2004 fiscal year.

1.3 BACKGROUND

The current Canada/ESA Cooperation Agreement was signed on March 28, 2000, and has been in force since January 1, 2000. It will remain in force for 10 years, subject to an agreement to renew its terms and conditions.

Under the Agreement, Canada makes annual contributions to ESA expenditures under the General Budget, with the exception of costs associated with the Scientific Program and Technological Research Programs. Canada contributes to the ESA General Budget at 50% of the GNP-based rate normally used to calculate the contribution of ESA member states. The General Budget relates to all expenditures involved in the overall management of ESA. The contribution to the General Budget is mandatory and provides certain rights and privileges, the most important one being the right to participate in optional programs. These programs relate to all space hardware and technology development initiatives open to member states, which can decide whether or not to participate in a program. The Agreement also provides for a clear obligation to ensure Canada a fair industrial return and guarantees Canada the same rights as ESA member states in optional programs.

The ESA is to receive up to $30 million in contributions each fiscal year over the period covered by the Cooperation Agreement (1999-2000 to 2008-2009).

1.4 METHODOLOGY

This audit was carried out in accordance with audit standards set forth in the Treasury Board Secretariat (TBS) Policy on Internal Audit, which requires that audit objectives be set on the basis of audit criteria. The audit objectives and criteria used are outlined in Appendix A.

Audit standards also require that the audit be conducted in a structured manner, according to a process that includes

• a planning and preliminary review phase,
• an execution phase, and
• the reporting and disclosure of results.

Various audit procedures were used, including interviewing employees and reviewing and analysing documents, records and reports.

The auditor provided periodic verbal reports to keep the entity being audited informed about how the audit project was progressing.

RESULTS OF THE AUDIT

2.0 ADMINISTRATIVE RECORDS

2.1 AUDIT TRAIL

Pursuant to the Treasury Board Secretariat's Policy on Transfer Payments, departments must establish policies and procedures to ensure that proper program and accounting records and other relevant documents are maintained to provide documentary evidence of decisions made and results achieved and to allow for disclosure of the amounts paid to recipients of such payments.

Under the Canada/ESA Cooperation Agreement, contributions are made to ESA's General Budget and to optional programs in which Canada has chosen to participate following an analysis of each individual optional program. The selection and approval process for optional programs is one of the elements covered by the Terms and Conditions of Contributions Under the 2000-09 Canada/ESA Cooperation Agreement (Appendix 1 of the TB decision). The selection and approval process for optional programs typically generates the following types of documents:

• Program Approval Documents (PADs),
• Arrangements,
• Declarations,
• Results of consultations, and
• Correspondence.

To ensure proper monitoring of the entire selection and approval process for optional programs, maintain an adequate audit trail and account for decisions made in connection with these programs, it is essential that complete administrative records be kept.

Our review of the situation showed that records relating to optional programs were not up to date and did not contain all the documents required to ensure a complete audit trail.

For the purposes of our audit, we had to consult several different sources to obtain the required documents:

• Copies of signed Arrangements were obtained from the CSA's Legal Services Directorate;
• Copies of unsigned Declarations were obtained from ESA's electronic records management system (DODIS); and
• Copies of submissions to the PRAB (which takes the place of a PAD) were obtained from the sector's financial analyst.

DODIS-The ESA's Official Documents Distribution and Archiving System

The manager of the Canada/ESA Cooperation Program told us that he relied to a large extent on DODIS to obtain information, such as copies of Arrangements, Declarations or other documents used in decision making, in a timely manner.

DODIS is ESA's electronic archiving system, and not the CSA's system. If, at any time, for reasons beyond the CSA's control, this system were no longer available to the CSA, it would be very difficult to track down certain documents required to maintain an adequate audit trail as described above. This makes it important for the CSA to retain complete administrative records for all optional programs to which the CSA contributes.

RECOMMENDATION

Ensure that complete administrative records are kept for every optional program in order to maintain an adequate audit trail and support decision making.

3.0 SELECTION AND APPROVAL PROCESS FOR OPTIONAL PROGRAMS

3.1 PROGRAM APPROVAL DOCUMENT (PAD)

Item 3.1, Selection Process, of Appendix 1 of the TB decision stipulates that a PAD must be prepared for each optional program under consideration for submission to the Program Review and Advisory Board (PRAB). This provision also describes what a PAD must contain:

• Content and objectives;
• Estimated policy, program and industrial development benefits;
• Results of consultations with concerned federal partners and industry;
• ESA position on Canadian participation in the program;
• Eligible costs;
• Proposed subscription rate;
• Potential contributions and required budgets by fiscal year; and
• Performance measurement indicators.

Our review showed that the submission to the PRAB takes the place of a PAD for optional programs. However, the format of submissions to the PRAB is preset and does not cover all the information required to prepare a PAD, as detailed in the following table:

TABLE 1 - REQUIRED INFORMATION FOR PADS

RECOMMENDATIONS

1. Review the relevance of including an item respecting "ESA position on Canadian participation in an optional program" in the TB submission regarding the renewal of the program's terms and conditions.
2. Ensure that all required information is included in PADs in accordance with the program terms and conditions approved by TB.

3.2 CONSULTATION PROCESS

The TB decision on the Canada/ESA Cooperation Agreement stipulates in item 4 that the CSA shall consult with DFAIT, DND and other departments and agencies concerned before signing any Arrangement respecting an ESA optional program. Appendix 1 of the TB decision states that the results of consultations with concerned federal partners and industry shall form part of the PAD's content.

However, very little documentation detailing the consultations held was available.

We obtained a list of main events used to promote the ESA program, or one of its components, and/or used as a mechanism for holding consultations on the relevance of maintaining the ESA program. The events on that list were the following:

TABLE 2 - PROMOTIONAL AND CONSULTATION EVENTS

Of the events on the list, the only one to result in an actual consultation process was the consultative conference held at the CSA on September 10, 2001.

After that meeting, the stakeholders who attended were asked to complete a survey of their needs. In all, 41 responses were collected. The responses were examined by the ESA program manager. However, the manager told us that the survey did not provide a better idea of the industry's needs.

The ESA program manager also told us about some of the weaknesses of this process. No true consultations had actually taken place. Comments, opinions and needs were one-way, in that there was no direct feedback from managers of ESA or CSA programs.

According to the ESA program manager, it would be appropriate at this time to implement a consultation process with a view to preparing for the ministerial-level ESA Council meeting scheduled for December 2005. The objective of this process will be to make known Canada's position on optional programs to which the CSA would like to commit in coming years.

Given that these consultations only take place every four years or so, a new means of holding consultations with industry should be considered immediately in order to gain a better understanding of industry needs.

We also noted that, during the consultative meeting held on September 10, 2001, 10 Canadian firms had the opportunity to meet with an ESA representative. These bipartisan meetings allowed these Canadian firms to forge ties with ESA. However, the CSA was not present at the meetings and therefore, could not take advantage of this opportunity to learn about the needs of Canadian industry.

With regard to consultations with departments such as DFAIT and DND, these departments were generally consulted when appropriate, and such consultations were mentioned in submissions to the PRAB as required.

RECOMMENDATION

Implement a structured and transparent process for consultations with industry with a view to supporting decisions respecting optional programs selected by Canada.

4.0 CONTRIBUTION ACCOUNTING

4.1 COMMITMENT OF FUNDS

Under the Canada/ESA Cooperation Agreement, Canada makes an annual contribution to expenditures entered under the General Budget and to optional programs.

Funds for the General Budget and optional programs are committed by service line in the following manner:

• Optional Programs - EO
• Optional Programs - Satellite Communications
• GSTP 3 (General Support Technology Program) - Space Technology
• Aurora - Space Exploration (under EO)
• General Budget - EO
• General Budget - Satellite Communications
• General Budget - Space Technology

According to information obtained from the sector's financial analyst, the General Budget is broken down among three CSA service lines: EO, Satellite Communications and Space Technology. The financial analyst also indicated that this breakdown had been arrived at several years ago and had been prorated according to the optional programs existing at that time. This breakdown had not, however, been revised since then and, therefore, no long reflects reality in terms of the current proportions of optional programs. Consequently, the General Budget is no longer broken down in a representative manner with respect to the optional programs to which the CSA contributes. This has an impact on the CSA's financial results by service line. Table 3 illustrates the scope of this problem.

For example, if we were to take the $5,451,172 allocated to ESA's General Budget for the 2003-2004 fiscal year and break it down on a pro rata basis among optional programs, a variance in the CSA's financial results would be produced, as follows:

TABLE 3 - BREAKDOWN OF THE 2003-2004 GENERAL BUDGET

Breaking down the General Budget on a pro rata basis with respect to optional programs would have provided more accurate financial data that reflects the true situation.

Furthermore, under the new Program Activity Architecture (PAA), programs will be categorized differently, effective April 1, 2005. According to information from the Assistant Manager, Planning and Financial Analysis, generic expenditures incurred under certain programs including the General Budget will be broken down according to a predetermined percentage set by senior management in accordance with the CSA's long-term objectives and vision for each of these activities.

Once again, according to the Assistant Manager, Planning and Financial Analysis, the following percentages will be used for the breakdown of ESA General Budget expenses, effective April 1, 2005:

• Earth Observation (EO): 46.39%
• Space Sciences and Exploration (SSE): 41.24%
• Satellite Communications (SC): 12.37%

Since participation in the General Budget is a prerequisite to being able to contribute to optional programs, it would be reasonable to consider breaking down the ESA General Budget in accordance with the optional programs to which the CSA contributes. Like the current breakdown method, the proposed approach that is to take effect April 1, 2005, does not allow the contribution to the General Budget to be broken down according to the relative size of optional programs to which the CSA contributes.

To illustrate the consequences of using one method versus another, we broke down the $7,144,000 contribution to the 2004-2005 General Budget among the three policy sectors according to the pre-established rates that will be in force for 2005-2006, then recalculated the breakdown, this time prorating it according to the projected contribution to optional programs for 2004-2005. The magnitude of the variances in Table 4 clearly shows the impact on financial data.

TABLE 4 - BREAKDOWN OF THE 2004-2005 GENERAL BUDGET

RECOMMENDATION

Review the rationale for treating the contribution to the General Budget as a generic expenditure related to various activities and instead consider breaking down these expenditures according to the actual contributions to optional programs, thereby obtaining more accurate financial data.

4.2 ADVANCE PAYMENT OF TRANSFER PAYMENTS

The TBS Policy on Transfer Payments states that any overpayments in the hands of a recipient at year-end or program-end must be accounted for and refunded if necessary.

Our review of payments to ESA under the Canada/ESA Cooperation Agreement showed that such payments-two payments of 40% of the annual total for the General Budget and optional programs paid in January and June respectively, plus the balance of 20% in October-were not fully used in the financial year in which they were made. Because cash flow does not always conform to the budgets initially set for optional programs, surpluses occur every year. These surpluses or portions thereof are used up in the following fiscal year. This obliges the CSA to account for these accumulated appropriations in its financial statements under an "Advance payments of transfer payments" account.

The item "Advance payments of transfer payments" in the CSA's financial statements to March 31 comes from two sources:

• Accumulated appropriations to ESA to December 31;
• Unused portion of the final payment (prorated for the remaining months in the period covered by the payment)

Table 5 below shows the amounts of advance payments of transfer payments recorded in the CSA's financial statements since 2002 and the small year-over-year variance.

TABLE 5 - STATUS OF "ADVANCE PAYMENTS OF TRANSFER PAYMENTS" ACCOUNT

The current practice of using the accumulated amount of appropriations to December 31 as reported by ESA has the effect of overvaluing the "Advance payment of transfer payments" account. The accumulated amount of appropriations as of March 31 would better reflect the actual situation, but ESA policy is to report this information to December 31.

The financial analyst responsible for the ESA program also told us that ESA's accounting system will be changed effective January 1, 2005. The new accounting system will allow ESA to make payment calls that will be very close to actual expenditures to be incurred by optional programs in the period covered by the payment call. The analyst also told us that accumulated appropriations to December 31, 2004 would be completely used up by the first payment call in January 2005.

Starting in January 2005, the amount of transfer payments to be accounted for as advance payments will be the portion of the latest payment call not yet used, prorated for the remaining months in that period. The amount entered in the CSA's financial statements would thus be more representative of reality.

RECOMMENDATIONS

Monitor accumulated appropriations to ESA to ensure that all appropriations accumulated to date are actually used before the next payment call is made, thereby bringing the amount of accumulated appropriations to zero, as planned.

4.3 EXCHANGE RATES

The TBS's Policy on Transfer Payments states that, in order to deal with possible foreign exchange fluctuations, a department should consult with the Treasury Board Secretariat where it is proposing a payment based on an assessment made by an international organization for Canada's contribution to the organization or its projects.

This provision is part of a section of the Policy concerning Treasury Board submissions for approval of program terms and conditions. Since the CSA is currently preparing a submission to the TB concerning the renewal of the terms and conditions of the Canada/ESA Cooperation Agreement, it would be an opportune moment to consult with the TBS to solicit their input on how the issue of exchange rates should be approached in the submission and eventually handled in terms of the day-to-day management of the program.

One of the recommendations of the May 1997 evaluation report prepared by the Hickling Corporation addressed the exchange rate aspect of the contribution to ESA. The following statement appeared in the report:

A contingency fund should be provided in the CSA budget for ESA to allow for exchange rate fluctuations which affect the cost of Canadian participation in ESA programs. The size of this fund should be assessed in light of the adoption of the new financial system based on the ECU.

Our discussions with the manager of the Canada/ESA Cooperation Program revealed that, to date, no mechanism had been set up to protect the CSA against fluctuations in exchange rates.

The recommendation of the May 1997 Hickling report is still valid, and the need to implement it is even more urgent, since exchange rates have been climbing steadily for several years, as the following table shows:

About three years ago, the program manager used an estimated exchange rate of approximately 1.60 for the budget estimates, whereas the actual rate was roughly 1.40. This allowed for a reasonable tolerance level to cover the risks of fluctuations in exchange rates, inflation and possible cost overruns in certain programs. Today, budget estimates are calculated using an exchange rate of 1.65, but actual exchange rates are currently ranging between approximately 1.60 and 1.65. The current situation thus leaves little or no room for covering the previously mentioned risks. This makes it even more urgent to set up a mechanism for covering the risks of exchange rate fluctuations and protecting the CSA against program cost overruns.

There are a number of different mechanisms that could be used to protect against exchange rate fluctuations, such as using foreign exchange contracts or adding the Canada/ESA Cooperation Agreement to the CSA's risk management framework. Furthermore, before an exchange rate fluctuation management mechanism is implemented, it is essential that the TBS be contacted, as stated in the TBS Policy on Transfer Payments, to obtain as much information as possible so that an informed decision can be made.

RECOMMENDATION

In consultation with the Treasury Board Secretariat, set up a mechanism to mitigate the risks of exchange rate fluctuations.

4.4 FINANCIAL CONTROL OF CONTRIBUTIONS TO OPTIONAL PROGRAMS

The TBS Policy on Transfer Payments states that departments must establish policies and procedures to ensure that effective financial and program controls are designed and implemented within departmental transfer payment programs.

Appendix 1 of the TB decision stipulates that Canada's financial contribution to an optional program may be increased up to 120% of the original approved envelope. It also states that Canada may decide to withdraw from the optional program once the approved budget exceeds 120% of the original envelope.

Our discussions with the manager of the Canada/ESA Cooperation Program and the sector's financial analyst revealed that there were no mechanisms in place to ensure compliance with these provisions.

When an Arrangement is signed for a particular optional program, Canada is aware of the amount it has committed, in euros (or AUs and ECUs, for older programs). The amount of Canada's contribution is identified in the optional program's Declaration. Payment calls are then made in euros and converted into Canadian dollars for accounting entry purposes. Contributions to which Canada commits are also generally spread out over a long period, until 2008-2009 in most cases.

Currently, the sector's financial analyst follows up on each payment, including annual payments, made in Canadian dollars for each optional program. However, there is no reconciliation of the total amount called for in euros and the amount Canada committed. It is therefore impossible to tell if optional programs have incurred cost overruns.

In order to ensure compliance with the clause that states that Canada's financial contribution to an optional program may be increased up to 120% of the original approved envelope and that Canada may withdraw if that limit is exceeded, the program manager must monitor the amounts paid out for each optional program. The manager may, however, call upon the sector's financial analyst to submit the necessary reports to facilitate this monitoring.

Our discussions with the sector's financial analyst revealed that the current analyst's predecessor had already complied such information. However, the reports were not completely up to date, and the information had not been forwarded to the program manager. The financial analyst also told us that he had recently received two documents from ESA's finance directorate that allow actual expenses in euros to be compared against amounts committed in euros. One report is a compilation of total expenditures in euros by optional program and by member state since the very beginning of optional programs in progress; the other is a compilation of actual expenditures and estimates to the optional programs' end, once again by member state. This information will allow the ESA program manager to monitor changes in optional programs and any cost overruns that might occur.

RECOMMENDATION

Monitor optional program expenditures to ensure that payments match the amounts in foreign currency to which the CSA has committed under its Arrangements for optional programs and make it possible to detect cost overruns.

5.0 RISK-BASED AUDIT FRAMEWORK (RBAF)

5.1 IMPLEMENTATION OF THE RBAF

The TBS Policy on Transfer Payments stipulates the following:

• Treasury Board submissions for program approval of terms and conditions for... contributions should include... a risk-based framework for audit of recipients of contributions;
• Departments must develop a risk-based audit framework for the audit of contributions;
• It is government policy... to manage transfer payments in a manner sensitive to risks, complexity, accountability for results and economical use of resources.

According to the TBS Risk-Based Audit Framework Guide, these requirements are met by developing a risk-based audit framework (RBAF).

In April 2003, an RBAF was drawn up for the Canada/ESA Cooperation Agreement. The RBAF was not, however, submitted to the TB at that time. The manager of the Canada/ESA Cooperation Program planned to submit the RBAF to the TB with the renewal of the program's terms and conditions in April 2005.

The RBAF primarily addressed two elements: the risk-management plan and the audit framework. Actions to implement the audit framework were carried out through this audit of the program. This audit activity was also provided for in Appendix 1 of the TB decision, "Program Terms and Conditions", under item 8, "Evaluation and Audit". However, the risk management plan has not been implemented. No action has been taken to mitigate identified risks.

The program manager told us that the RBAF has not been implemented because it has not yet been submitted to the TB for approval. Once it has been approved by the TB, the RBAF will become an integral part of the terms and conditions of the program and will have to be implemented.

Since the RBAF is a management tool that helps managers carry out their duties and honour their obligation to be accountable, there is no need to wait for the RBAF to be approved by the TB before implementing it.

RECOMMENDATIONS

1. Implement the RBAF.
2. Monitor the risk management plan on a regular basis to keep track of changes in identified risks and ensure identified risk management actions are implemented.

5.2 PROGRAM RISK MANAGEMENT PLAN

During our audit, we reviewed the program risk management plan included in the RBAF. We looked at the following aspects of each identified risk:

• The probability of the risk occurring;
• The consequences, should the risk occur;
• The measures identified to mitigate the risks;
• The tangible actions to be taken to reduce the risk to a level the CSA deems acceptable.

Our review led to the following observations:

• The measures to be implemented, as well as the tangible actions to manage the risk of exchange rate fluctuations, should be
  • reviewed, given that they refer to a TB program about which we have been unable to obtain any information, and
  • revised, in light of the recommendation made in section 4.3 of this audit report.
• One of the identified measures for mitigating the risks arising from staff turnover is the retention of appropriate administrative records. This makes the recommendation in section 2.1 of this audit report even more important.
  • The actions regarding records management identified in the RBAF should be revised.
• In the "Program Governance" section, the first identified risk refers to Canada's level of participation in an optional program that is below the critical mass required to achieve industrial returns and sustainable socio-economic benefits.
  • In order to justify the level of risk identified in the RBAF, the critical mass must be defined. If it is not, any evaluation of that risk will be arbitrary.
  • The action identified for mitigating that risk, ie, developing an evaluation methodology and procedures, has not been implemented.
• The action for mitigating the risk of rising program costs refers to ESA signing firm-price contracts. However, the CSA has no influence over the awarding of such contracts.
  • This item should instead refer to CSA's option to withdraw from any optional program when cost overruns exceed 20%.
• The "Information and Management Systems" section refers to a 2003 pilot project to evaluate whether performance measurement indicators identified in the results-based management and accountability framework (RMAF) are adequate.
  • Our audit showed that no pilot project was carried out, and no data collection system was set up.
• Having completed our audit, we find that the deemed level of risk of a paper and electronic records management system should be raised from low to medium (see section 2.1 of this audit report).

We gave the program manager an annotated version, with comments, of the risk management plan included in the RBAF.

RECOMMENDATION

Revise the RBAF's risk management plan to take the above mentioned comments into account.

6.0 RESULTS-BASED MANAGEMENT AND ACCOUNTABILITY FRAMEWORK (RMAF)

6.1 IMPLEMENTATION OF THE RMAF

The TBS Policy on Transfer Payments stipulates that "Treasury Board submissions for program approval of terms and conditions... for contributions should include... a results-based accountability framework including: performance indicators, expected results and outcomes, methods for the reporting on performance, and evaluation criteria to be used in the assessment of the effectiveness of the transfer payments. Furthermore, the TBS Evaluation Policy states that "[m]anagers have a responsibility to monitor the performance of policies, programs and initiatives - to make sound decisions and to report on performance."

In March 2003, a results-based management and accountability framework (RMAF) was drafted for the Canada/ESA Cooperation Agreement. As was the case with the RBAF, the RMAF was not submitted to TB at that time. Once again, the manager of the Canada/ESA Cooperation Program planned to submit the RMAF to the TB upon the expected renewal of the program's terms and conditions in April 2005.

As required by the PMAF and Appendix 1 of the TB decision, a program evaluation was carried out. The evaluation coincided with our audit.

The program manager told us that, since the RMAF had not yet been submitted to the TB for approval, it had not been implemented. Consequently, no data collection system had been implemented to date. Such a system is required to be able to measure the expected results of programs in terms of the performance measurement indicators set out in the RMAF.

Like the RBAF, the RMAF is a management tool that helps managers to carry out their duties and honour their obligation to be accountable. There is no need to wait for the RMAF to be approved by the TB before implementing it, especially since a pilot project to help evaluate the adequacy of the RMAF's performance measurement indicators had been planned for 2003.

RECOMMENDATIONS

1. Implement the RMAF.
2. Set up a data collection system to measure the expected results of programs in terms of the performance measurement indicators set out in the RMAF.

6.2 CONTENT OF THE RMAF

A formative evaluation of the Canada/ESA Cooperation Agreement took place at the same time as our audit.

Given that the CSA is in the process of preparing a TB submission regarding the renewal of the terms and conditions of the Canada/ESA Cooperation Agreement, the RMAF should be revised and amended, if need be, taking into account the results of the formative evaluation to be submitted at the same time as our audit report.

RECOMMENDATION

Revise the RMAF, taking into account the results of the formative evaluation of the Canada/ESA Cooperation Agreement.

APPENDIX A - AUDIT OBJECTIVES AND CRITERIA

APPENDIX B – MANAGEMENT ACTION PLAN