Guide on Grants, Contributions and Other Transfer Payments

Appendix E

Western Economic Diversification (WD)-Quality Assurance Review (QAR) Step III: G & C Project File Management Tools

Audit and Evaluation

DRAFT DOCUMENT FOR DISCUSSION ONLY

Quality Assurance Review
Step III:

G&C Project File Management Tools

Tuesday, October 24, 2000

Table of Contents

Background to the QAR Initiative

Objective and Scope of the QAR Initiative

QAR Issues and Methodology

QAR Step III

Next Steps

Appendix One - General Approach to the QAR Initiative

Appendix Two - QAR Checklist Issues

Appendix Three - Guidance for File Contents

Appendix Four - Final QAR Checklists

Background to the QAR Initiative

As a result of discussions with the Special Advisor to the Deputy Minister regarding issues raised by the Office of the Auditor General (OAG) in Chapter 27 of their December, 1998 Report; Audit and Evaluation, in consultation with all stakeholders, has undertaken a Quality Assurance Review (QAR) of grants and contributions (Gs&Cs) at Western Economic Diversification Canada (WD).

In 1994 and 1995, the OAG audited a number of contribution programs delivered by Industry Canada and by federal regional development agencies (including WD). At that time they noted opportunities for improvement in different aspects of program delivery, including concerns about due diligence in project assessment. Their recommendations included the need for:

• More persuasive assessments to support funding decisions, and documentation that reflects all the major factors supporting decisions to approve assistance;
    
• The streamlining of guidelines and criteria for assessing applications, differentiated by size and risk, and simplification of application documentation for small projects;
    
• Better co-ordination with other federal and provincial departments in view of the potential for duplication of effort and funding; and
    
• Greater attention to collecting repayable contributions.

The QAR initiative represents a proactive effort to move WD's grant and contribution efforts further along a continuum to having all of the characteristics that the OAG describes in Chapter 27 of the December, 1998 Report as, Qualities of a Well-Managed Grant or Contribution Program:

• The choice of funding instrument - grant or contribution - respects accountability to Parliament and achieves a balance among principles of cost-benefit, risk management, and reasonable treatment of program recipients.
    
• Management can explain how recipients are expected to benefit from funding and to what end.
    
• Program officers understand who and what is eligible for funding, under what conditions funding can be provided, for what purposes, and in what amounts.
    
• Potential applicants are aware of the program.
    
• Eligible projects represent value for money to both the applicant and the program.
    
• More deserving recipients are funded and at an appropriate level.
    
• Funding is used for the purposes agreed.
    
• Problems with project and program performance are resolved quickly.
    
• Management reporting demonstrates a good knowledge of program performance.
    
• Money owed to the Crown is collected promptly (in the case of repayable contributions).

The QAR Initiative has provided an opportunity for WD to be proactive and move to the forefront of government departments in providing grants and contributions that are administered in a manner which ensures due diligence and provides performance information in keeping with government policy and central agency expectations. And, in line with the Mid-Term Strategy, this initiative shows WD in a leadership position that emphasizes a move to being a Department that sets an example for others. Finally, the QAR Initiative has allowed WD to move quickly and with better knowledge and assurance than it would have with regard to the new TB Policy on Transfer Payments.

Objective and Scope of the QAR Initiative

Objective

When complete, the QAR initiative will result in a consistent approach for the grants and contributions handled by WD. Once implemented, practices suggested as a result of QAR will help to ensure that due diligence is addressed, and that the statement and assessment of clear and measurable results is provided for WD's grants and contributions portfolio. In doing so, WD will meet the expectations of the OAG and enhance the accountabilities which currently exist in the Department.

Scope

The QAR initiative is focused on existing, ongoing, and future initiatives and programs for which WD is directly responsible (e.g.: ITPP, FJST), or that WD delivers through arrangements with third parties (e.g.: CFDCs). Files from FY 1997-98 to FY 1998-99 were looked at in conducting the fieldwork related to Step Two of the QAR initiative.

The QAR effort thus far has not covered funding through the Department's O&M budget (e.g.: CBSCs), nor has it addressed grants and contributions that have recently been covered by other distinct accountability mechanisms. Both the Infrastructure Works Program (IWP) and the Manitoba flood-related Jobs and Economic Recovery Initiative (JERI) are examples of the types of programs not covered by the QAR initiative. These two initiatives have recently been covered by OAG audits or other evaluations.

QAR Issues and Methodology

The QAR Initiative consisted of four steps:

The issues considered during the Step Two fieldwork were drawn in large part from Chapter 27 of the 1998 OAG Report, many of which were re-emphasized in their November 1999 Report. The broad issues considered by the fieldwork were as follows:

• Due diligence: Demonstrated by reviewing and approving applications in a complete and appropriate manner and providing persuasive assessments based on relevant documentation to support decisions to approve assistance.
    
• Clear, attainable objectives: That allow management to explain how recipients are expected to benefit from funding and to what end.
    
• Performance against objectives is measured and reported: Ensure control, monitoring and evaluation of the funding effort and that funding is used for the purposes agreed.
    
• Funds are managed in a responsible manner: Ensure that resources are used efficiently and that payments and repayments occur in a timely manner.

QAR deals with issues at the project, not program, level. In making that distinction, QAR does not consider program-level issues such as choice of funding instrument, compliance with government or departmental priorities or overall program reporting (e.g. Annual Report, Performance Report, etc.). The QAR Step II: Best Practices Report provides a schematic to help put the QAR initiative in perspective, and place the issues covered by QAR relative to an overall program and project life cycle.

QAR Step III

Where Are We Now?

QAR Step III is now complete, and has followed from the work reported under Step II in The Best Practices Report. In that document, we recognized that best practices are not only documents; but rather the activities around ensuring that appropriate file documentation exists:

• Information gathering;
    
• Information preparation;
    
• Information entry;
    
• Information verification; and
    
• Information analysis.

We make this distinction to emphasize that although documents may be on file, if they are not supported by best practices, there is a possibility that the need to ensure due diligence, have clear objectives, measure and report on performance and manage funds responsibly has not been accomplished.

Step III has included:

• development of practical project file management tools for all levels of management and staff who handle WD grants and contributions;
    
• development of a training package in the train-the-trainer mode to share the knowledge gained;
    
• delivery of that training;
    
• redrafting of the project file management tools based on feedback during training;
    
• sharing and consultation on the re-worked tools; and
    
• a final edit of the tools (included for use as Appendix Four).

This report on Step III brings this stage of the QAR Initiative to a close in terms of A&E's developmental efforts. As noted, we have received valuable feedback from the Regions, and via this report, will clarify where the contents of the QAR checklists come from, and how they might be adjusted for the project file types to reflect the best practices and documentation identified.

A&E is now looking to the program areas to make any further adjustments to these tools to meet local and program/project considerations - bearing in mind the content of this report. In addition to this Report A&E has prepared a guidance document (Appendix Three) that is an essential part of considering the QAR Checklists.

The QAR Checklists

The checklists have been adjusted to take into account key suggestions from the latest feedback on the instruments. Each has been changed to remove the word "Form" from "Client Application Form" and substitute "Information". There is a sense that if the information is completed as suggested in the guidance document (Appendix Three), it will indeed meet the intent of TB without compromising the informal way that WD wants to interact with our clients.

For all of the QAR Checklists, including the WDP Checklist; the best practices work suggests very strongly that a T&C (Terms and Conditions) Checklist, a PAS (the Project Assessment Summary) and a PAN (Project Approval Note) should all be completed and on file. Discussions have made it clear that the PAS is seen as a document that supports the PAN; but does not replace it as an essential element of ensuring that due diligence has taken place. With that said, some discretion must be used. And that discretion should correspond directly to an on-file assessment of the risk presented by each project. It is clear that unduly complex files, and exhaustive file contents are not the best use of effort where risk is assessed to be low. Nonetheless, certain critical elements must exist in all files, and certain critical best practices must be undertaken. These include a T&C Checklist, a PAS, and a PAN.

In terms of the Loan Funds Checklist, some modification of specific wording may be in order for specific projects given regional preferences in terms of program and project-level accountability. The guidance provided by WD on our web site [http://www.wd.gc.ca/finance/programs/appsum_e.asp] is also helpful in that it clearly defines the information that is expected from each applicant. Also worth considering is the direction provided in the TB submission for LIFP that states that the application process will include the documentation of the technology review and a business plan. One specific suggestion in refining the QAR Checklist may be to replace the "Terms and Conditions Checklist" with a document (or checklist) that will help to demonstrate that basic eligibility criteria had been checked and met. In addition, some type of formal signed-off approval note is advisable. The eligibility letter to the bank is a clear file content requirement, as would be a copy of the loan approval letter from the financial institution. Undertaking these best practices will reflect the notions included in WD's TB submission suggesting that applicant screening, due diligence and loan/investment application approval is be based on an integrated and transparent process.

It is apparent that the intent of the Loan investment Funds is to bridge access to financing gaps in particular industries that represent opportunities for the diversification of the western economy. While efforts to assess program performance have focused on statistics regarding take-up and funding issues, the intent of monitoring and evaluating the effort should include whether or not the loans help achieve that broader objective. And for that reason, progress and final reports from the individual loan recipients is seen as a key component of the files.

Our QAR File Contents and Best Practices Checklist for Projects Delivered by WD's Service Partnership Network, are also finalized. Our discussions indicate that program staff view the agreements (and resulting file contents) that cover the relationship between service partners and WD as a strong point. Recent developments - most particularly Treasury Board's new Policy on Transfer Payments - have strengthened expectations in this regard. In this light, WD should continue to ensure that we meet the expectations of central agencies (e.g.: the OAG) that may be looking for comprehensive documented knowledge of individual projects funded by our service partners who deliver either WD-core, or one-time programs.

One way to meet the expectations of central agencies is for service partners to have the QAR checklist on file, and to have the necessary supporting documentation there as well. WD can contribute by undertaking activities that ensure our identified "Best Practices" are taking place on a project-by-project basis. This could be accomplished in a comprehensive manner by having critical service partner project-specific information on our files, and undertaking a planned and regular risk-based review of files. Doing so will help to ensure that due diligence is exercised, objectives are clear, performance is measured and reported, and funds are managed responsibly.

Next Steps

Attached are the three final checklists as prepared by A&E. As we have mentioned, it is now up to WD staff associated with project management to adjust the checklists to reflect wording peculiarities as they reflect operational conditions, and considering the comments above. With our role in Step 3 complete, Step 4 of the QAR Initiative is a review of files which is expected to start late in the final quarter of FY 2000-2001. In the interim, there remains the clear probability that the OAG will exercise its prerogative to audit our files as well, either as part of a government-wide G&C audit or an audit of WD alone.

A wealth of effort exists to demonstrate WD's strong approach to managing the department's grant and contribution portfolio. Through QAR, these best and promising practices have now been shared; allowing WD to build on the existing grant and contribution management efforts and promote the consistent application of the best and promising practices identified. By doing so, we are assuring that grant and contribution funds at WD are providing the maximum intended benefits to Canadians, and that scrutiny of our practices would result in consistently positive observations. Through the QAR initiative, WD is acting in a positive manner and has assumed a leadership role for all government departments that provide grants and contributions to Canadians.

Appendix One - General Approach to the QAR Initiative
  

Appendix Two - QAR Checklist Issues

Due diligence is demonstrated by reviewing and approving applications in a complete and appropriate manner. Persuasive assessments exist to support the funding decisions, and documentation is on file that reflects all the major factors supporting decisions to approve assistance.

• It is demonstrated that other methods of funding have been exhausted.
    
• An assessment has been made to ensure that funding will not have a negative impact on privately funded enterprises.
    
• An assessment has been made of the ability (e.g.: managerial, financial, etc.) of the recipient entity to ensure they are capable of delivering the project.
    
• Other federal and provincial departments have been consulted to ensure that funding is not duplicated.
    
• Recipients are funded and at an appropriate level.
    
• A comparative assessment of the value for money to both the applicant and the program has been completed (e.g.: cost-benefit, cost-effectiveness).
    
• A rigorous assessment, based on clear criteria, has been made regarding the risk inherent in the project.
    
• Alternative methods of delivery have been identified and assessed.
    
• Environmental concerns have been identified and assessed.
    
• Human rights concerns have been identified and assessed.

Clear, attainable objectives are set, allowing management to explain how recipients are expected to benefit from funding and to what end.

• Objectives are clearly stated.
    
• Objectives relate to results as well as activities.
    
• Objectives contain essential elements; stated condition, expected quantitative measure, and time frame.
    
• Objectives are plausible and realistic reflecting strength in program design.
    
• Achievement of objectives is clearly attributable to the program effort.
    
• Achievement of objectives will clearly provide incremental results.
    
• Objectives are related to the statement of overall program goals and objectives.
    
• Objectives as stated comply with overall program authorities.

Performance against objectives is measured and reported to ensure control, monitoring and evaluation of the funding effort and that funding is used for the purposes agreed.

• The terms and conditions set in the agreement are clear and unambiguous.
    
• Monitoring and evaluation requirements are clearly set out.
    
• There are measures in place that clearly reflect objectives.
    
• Monitoring occurs on a regular and timely basis, appropriate to risk.
    
• Evaluation occurs as per the agreement and in accordance with TB guidelines in a timely manner.
    
• There is a mechanism for resolving problems, and problems with performance are resolved quickly.
    
• Reporting demonstrates a good knowledge of program performance.

Funds are managed in a responsible manner.

• An ongoing assessment of the efficient use of the funds exists.
    
• Payments are based on verified evidence of appropriate expenditures.
    
• Payments are made properly in accordance with the agreement (in the case of repayable contributions, money owed to the Crown is collected promptly).
    
• Projects are processed and managed in a timely manner.

Appendix Three - Guidance for File Contents

Program Officers are expected to ensure that the Best Practices of Information Gathering, Preparation and Entry, Verification and/or Information Analysis are undertaken; and that the Core and Supporting Documents are on file.

Covered by WDP T&Cs:

1. Agri-Food Initiative [SK only]
2. Canada Foundation for Innovation Support Program
3. First Jobs in Science and Technology Program
4. International Trade Personnel Program
5. Sponsorships-Contribution Payments
6. Strategic Initiatives Program
7. Western Cooperation on S&T [AB only]
8. Western Economic Diversification Program - 807005 / 809563

Covered by Separate T&Cs [and agreements]:

1. Investment Loan Funds - 823561
2. Agriculture Value-Added Loan Program - 823272
3. Knowledge-Based Industries Loan Fund - 823182
4. Western Economic Partnership Agreements - 825994
5. Western Urban Development Agreements - 822310

Covered by Separate T&Cs for Operational Funding, and responsible for Third-Party Delivery of Programs:

1. Canada Business Service Centre Agreement - 824991
2. Community Futures Program - 826075
3. Service Delivery Network - 825101
4. Women's Enterprise Centres - 822141

The Tables below offer guidance with regard to the documents one might expect to find on file for current, on-going WD programs. The x symbol represents a required item based on established WD policy, or more importantly, Treasury Board Decision. A y denotes a document that been suggested through our QAR Best Practices exercise.

Appendix Four - Final QAR Checklists
  

Guidance Document Notes:

(i) Covered by WDP Ts&Cs - Required information is based on the content of the WDP Operations Handbook.  [Return]

1. the name of the applicant and distribution of ownership;
2. the purpose or objective of the project;
3. a statement of the work to be done;
4. the project commencement and completion dates, including critical factors in the proposed scheduling;
5. the benefits expected and, depending on the nature of the project, this may be elaborated to include projections of the net jobs to be created, increased sales, cost reductions, ancillary business/activities affected, etc.;
6. a business and financial plan, including a budget and the shares to be borne by each of the applicant, WDP, and any other contributors;
7. the ownership and responsibility for the operation and maintenance of the project after completion, if other than the applicant;
8. all other sources of funds to which this project, or a substantially similar project, has previously been submitted or is currently under consideration;
9. where applicable, environmental impacts."

(iii) WD Project Screening Information - All elements well described in WD's Operations Handbook.  [Return]

(iv) Client Project Proposal - May be included as part of the client's Business Plan.  [Return]

(v) Client Business Plan - May consist of some or all [officer's discretion and as applicable] of the following components depending on the project:

• past financial and operating performance and capabilities [required];
• current financial and operating performance and capabilities [required];
• details of ownership and incorporation [required];
• description of the purpose and objectives of the project [required];
• consideration of the operating environment;
• statement of work [required];
• marketing plan;
• location strategy;
• production plan;
• technological assessment;
• human resourcing details;
• R&D Plan; and
• Financial Plan [required]
• [Return]

(vi) WD Terms and Conditions Checklist - To be developed by program staff - will include all relevant requirements from the Terms and Conditions as accepted or demanded by TB.  [Return]

(vii) WD Project Assessment Summary - All projects covered by WDP Terms and Conditions must have a fully completed Project Assessment Summary.  [Return]

(viii) WD Letter of Offer - WDP Ts&Cs [807005] stipulates that, "The Letter of Offer will constitute the only legally binding confirmation of assistance under WDP. Letters of Offer, which may vary in contents according to the nature of the project, time to completion, etc.) will set out specific Terms and Conditions for the financial support that WD is offering to the applicant." [Return]

(ix) WD Client Contact Notes - Formalization on-file of informal contact information from phone calls, meetings, non-audit related site visits, etc.  [Return]

(x) WD/Client Signed Agreement - The parties to the agreements may vary depending on the program. For those covered by the WDP Terms and Conditions, the agreement is indeed between WD and the applicant. For the Loan Funds programs the parties to the agreement are WD and the financial institution. For SDN, WEPA and WUDA agreements may include several parties. CFDC and WEC delivered programs may involve agreements between WD and another department as well as the CFDC or WEC and the applicant.  [Return]

(xi) TB Approvals - For all projects with a WD assistance level of $10 million or more. [Section 7, WDP Ts&Cs]  [Return]

(xii) Client Progress Reports - Under WDP, required on a quarterly basis.  [Return]

(xiii) Client Final Project Report - As set out in the Letter of Offer indicating the results, actual and expected benefits, and any follow-up activity planned. [see Section 13 v) of the WDP Ts&Cs]  [Return]