Home : Reports and Publications : Audit & Evaluation : Audit of the Community Futures Program - April 2003
Findings: WD Monitoring System
In general, there are many similarities between the regions in terms of how the CFDCs are monitored, and how WD interacts with the CFDCs. For example, for the most part the interaction with the CFDCs in regards to their initiatives and programs are the responsibility of one group, while the review of financial statements and payment functions are the responsibility of another group.
The fact that there are many similarities between methods utilized by the regions to monitor the CFDCs is not surprising, since all are essentially using a process which is based on the various reporting requirements in the CF Contract. However, the manner and degree to which each of these tools is used in each region causes several differences between the regions.
The following chart summarizes the approach in each region. The major differences between regions have been shaded (although there are several more subtle differences which are too detailed to be summarized in the chart). Further commentary follows the chart.
|
Manitoba |
Saskatchewan |
Alberta |
B.C. |
|
|---|---|---|---|---|
|
Review of Operating Plans |
Review Program: No standardized program (reviewed by Client Service Officers) | Review Program: No standardized program (reviewed by Client Service Officers) | Review Process: No standardized program (reviewed by Client Service Officers) | Review Process: No standardized program (reviewed by Client Service Officers) |
| Documentation: Undocumented (however, have template document to facilitate review process) | Documentation: Undocumented | Documentation: Undocumented | Documentation: Undocumented | |
| Approval: Formal (incorporated into contract through amendment) | Approval: Informal (no contract amendment required) | Approval: Formal (incorporated into contract through amendment) | Approval: Formal (incorporated into contract through amendment) | |
|
Review of Quarterly Reports |
Review Program: No standardized program | Review Program: No standardized program | Review Program: No standardized program | Review Program: Standardized program |
| Documentation: Undocumented | Documentation: Undocumented | Documentation: Undocumented | Documentation: Documented | |
| Utilization of Reports: scanned for key items / anomalies, minimal use for monitoring or strategic decision-making. Mainly used as a tool for reporting regional results. | Utilization of Reports: scanned for key items, anomalies, minimal use for monitoring or strategic decision making.Mainly used as a tool for reporting regional results. | Utilization of Reports: screened by CSO admin. support for accuracy and completeness; also scanned by CSO. Mainly used as a tool for reporting regional results. | Utilization of Reports: on some key reporting items, used for monitoring / discussion. Otherwise, simply a reporting mechanism for the region. | |
|
Review of Financial Statements |
Review Process: No standardized program (however, a checklist is used so that each is reviewed for specific items) | Review Process: No standardized program (however, each reviewed for specific items) | Review Process: Standardized program (started in 2002) | Review Process: Standardized program |
| Documentation: Undocumented | Documentation: Undocumented | Documentation: Documented | Documentation: Documented | |
|
Approval of Payments |
Process: schedule is based on contract, each payment is recommended and approved | Process: schedule is based on contract, each payment is recommended and approved | Process: schedule is based on contract, each payment is recommended and approved | Process: schedule is based on contract, each payment is recommended and approved |
| Contingencies: payments can be withheld for major items, 4th quarter payment contingent upon F/S and certain review items on F/S review checklist | Contingencies: All payments contingent on quarterly F/S review and reconciliation | Contingencies: Payments can be withheld for major items | Contingencies: payments can be withheld for major items, 3rd and 4th quarter contingent on F/S review resolution | |
|
Site Visits |
Frequency: based on risk assessment | Frequency: not systematic (except for detailed "audit") | Frequency: not systematic | Frequency: not systematic |
| Type: board meetings, initiatives, planning retreats | Type: board meetings, assistance with initiatives. However, also have a detailed "audit" of CFDC every 3 years. | Type: board meetings (mainly the "zone" meetings), initiatives. | Type: joint initiatives, board meetings |
British Columbia
Key Monitoring Processes:
Each CFDC is subject to an extremely vigorous financial statement review program, both at the time of filing interim financial statements and the year-end audited financial statements. The review is based on a standard program, and the results of the review are thoroughly detailed on the file. Advances for the 3rd and 4th quarters are tied to the satisfactory resolution of queries raised during these reviews (or at least that the CFDC is working in good faith with WD in resolving the queries).
Also, B.C. is the only region that has developed a systematic process for reviewing the quarterly performance reports. The process utilizes Microsoft Excel macros to extract information from the quarterly reports and project them for the fiscal year, allowing comparisons to the operating plan of the individual CFDC. The client services officer (“CSO”) can then select the key criteria to be reviewed, and discuss any questions or comments with the CFDC staff. While the Excel tool will calculate variances for all numerical data, this analysis is intended to be used as a starting point for discussion with the CFDC for any variances which appear to be large, or that require further explanation. It is not intended to be a report card measuring CFDC success or failure.
Additional Monitoring Processes:
Similar to the other jurisdictions, the annual operating plans are reviewed by the CSO. This review relies on the knowledge and experience of the CSO to raise and resolve with the CFDC any concerns of the CSO. The reviewed files contain little documentation of the review process or discussion between the CFDC and WD staff. Once the CSO recommends acceptance of the operating plan, it can be approved and the CF Contract is amended to include the current year's performance targets and operating budget.
Over the past few years, a risk assessment has been done on each CFDC, which allows the CSOs to focus their effort on CFDCs that are higher risk. However, due to resource issues, site visits are not done on a systematic or methodical basis. The primary purposes for the CSOs' visits include attendance at board meetings, participation in joint initiatives or presentations, not monitoring.
(For more a more detailed overview of the system in B.C., please refer to Appendix A-4).
Alberta
Key Monitoring Processes:
As with B.C., each CFDC is subject to a financial statement review. The review is carried out by the Monitoring and Payments group based on two review programs – one for the financial statements, and one for the investment fund. All of the findings from the financial statement and investment fund reviews are documented on file. This standardized review programs are a relatively new endeavour. The fiscal years ended March 31, 2001 and March 31, 2002 are currently under review for all of the CFDCs. In the future, it is planned that each CFDC will undergo this review on an annual basis.
The Alberta region is divided into three zones. In each zone, the CFDCs have quarterly meetings involving representation from the management and/or board of each CFDC. The CSOs in Alberta attend these meetings, which gives them the opportunity to interact face?to?face. WD Staff indicated that the resources to visit each CFDC with this amount of regularity are essentially not available, and while this is not a substitute for visits to each CFDC on an individual basis, it is considered an efficient compromise. The site visits that are made to the CFDCs, like B.C., are not done on a systematic or methodical basis, and usually would be along the lines of occasionally attending board meetings where WD presence is requested or to participate in presentations or initiatives.
Additional Monitoring Processes:
Responsibility for the review of the annual operating plans is with the CSOs. Similar to the practice in B.C., the review relies on the judgement of the client services officer. The reviewed files contain little documentation of the review process or discussion between the CFDC and WD staff. The CSOs, however, meet and discuss common issues and practices to ensure consistency among the group. The approval of the operating plan is evidenced through an amendment to the CF Contract to include the current year's performance targets and operating budget.
The quarterly performance reports filed by the CFDCs also go through a screening process. The reports are typically filed with the Program Assistant, who reviews for completeness and for errors. The reports also are sent to the CSOs for their review. The extent of the review is left to the discretion of the CSO. For the most part, the quarterly reports are compiled and used as a tool for reporting on performance of the entire region, and their use as tools to monitor the performance of individual CFDCs is limited.
(For more a more detailed overview of the system in Alberta, please refer to Appendix A-5.)
Saskatchewan
Key Monitoring Processes:
Every quarter, each CFDCs files financial statements and a claim form for payment relating to that quarter's funding from WD. The statements and form are reviewed, and WD staff reconciles the funding received with expenses incurred, allowing WD staff to monitor spending and activity undertaken by the CFDC, and control the payments made. Saskatchewan is the only province to require that the CFDCs file quarterly unaudited financial statements (all other jurisdictions require only the interim unaudited financial statements for the 6 months ended September 30 each year).
Additionally, on a cycle of approximately 3 years, each CFDC is subject to a review on the premises of the CFDC. The review is broad, and encompasses an operational review of the CFDC including their polices and procedures, a detailed project file review and a financial systems review. In general, the review involves a group of two or three WD staff for a period of two to three days. This review not only gives WD Staff assurance regarding the practices of the CFDC, but also the opportunity to make constructive recommendations to the CFDC to improve its operation.
Additional Monitoring Processes:
Annual operating plans are reviewed by the CSOs. There is not a formal review program, and the client services officer is responsible for the review. The reviewed files contain little documentation of the review process or discussion between the CFDC and WD staff. Also, Saskatchewan is the only jurisdiction where the CF Contract does not include annual performance targets and operating budgets. Therefore, in the other jurisdictions the CF Contract is amended every year to included these items, which evidences the approval of the annual operating plan. In the case of Saskatchewan, the approval of the annual operating plan is not documented.
Client services officers also review quarterly reports, based on the own experience and judgement. There is no formal review program or variance analysis performed. Traditionally, it has been at the discretion of these individuals that these reports are reviewed, and they have been mainly used as a tool to report statistically on the entire region.
(For more a more detailed overview of the system in Saskatchewan, please refer to Appendix A-6.)
Manitoba
Key Monitoring Processes:
WD Staff in Manitoba have developed two important tools that they use to help monitor the CFDCs. The first is the “Accomplishments Template” which gives the CF Consultant a concise overview of the CFDC's Goals, Actions and Measurement of Success. The CFDCs will be required to submit their 2003-2004 Operating Plans using this template, which will facilitate the review process. Once the Operating Plan has been reviewed, the approval of the plan is the amendment to the CF Contract to incorporate the current year's performance targets and operating budget.
The second tool is a Financial Statement checklist. While it is not a comprehensive review program, the items on the checklist point the CF Consultant to the key areas of focus in reviewing the financial statements. The fourth quarter payment approval is also part of the checklist.
Manitoba also has implemented the risk assessment process leading to a systematic process of determining the frequency of site visits. These site visits include board meetings, planning retreats and joint initiatives with the CFDCs, and form an important part of the system for monitoring the CFDCs.
Additional Monitoring Procedures:
The quarterly performance reports are handled in a similar way to Alberta and Saskatchewan. The CF Consultants can review the reports based on their own judgement and experience, but there is not a set review program, and any review is largely undocumented. The reports are compiled, and statistics on the performance of the region as a whole is generated.
(For more a more detailed overview of the system in Manitoba, please refer to Appendix A-7).
Auditor General's Report and Recommendations
In response to comments made in the 2001 Report of the Auditor General of Canada, WD committed to implementing certain policies and practices. The Auditor General's comments in relation to the monitoring and control of the CF Program are summarized below, with the response of WD (the citations correspond to the paragraph numbers of the 2001 Report).
Auditor General's Comments:
- WD should consider conducting a program evaluation and an internal
audit of the program. (5.283)
- WD should establish procedures to assess the accuracy and validity
of information that CFDCs provide on their performance. (5.283)
- WD officials have informal ways of checking on CFDCs. In our
view these practices are insufficient. They do not provide a systematic
and objective overview of CFDC's lending activities to ensure
that they are meeting the Program's terms and conditions.
(5.290)
- Starting with years ended March 31, 2001, WD asked each CFDC to have its auditor provide an opinion on its compliance with the contribution agreement. This should provide WD with good information on the overall lending activities of each CFDC. However, we found these opinions on compliance in only half of the files we examined. (5.291)
WD Response:
- WD will initiate an evaluation, and an audit of the management
control framework. (5.283)
- WD will strengthen procedures to assess the accuracy and validity
of CFDC performance information. (5.283)
- WD implemented an effective monitoring system through a quality
assurance review process that became effective July 1, 2001. WD
is committed to implementation of this system, including improved
file documentation and taking appropriate action on issues identified
as a result of the monitoring procedures. (5.293)
- WD was in the process of following up with CFDCs that had not met the reporting standard for the reports of compliance. (5.291)
Audit and Evaluation
This audit, and the concurrent evaluation, which is also being completed by Grant Thornton, represents the fulfillment of the commitment of WD to complete an Audit and Evaluation of the CF Program. While the audit findings and recommendations are presented in this report, the Evaluation findings are the subject of a separate report.
Accuracy and Validity of the CFDC Performance Information
For the most part, assessing the accuracy and validity of the performance information reported by the CFDCs to WD was outside the scope of this audit. However, comments were raised in each of the regional offices regarding this subject, indicating WD staff also had some concerns over the information being reported on including:
- Some relevant information may not be captured
- Some irrelevant information may be captured
- Reporting may be inconsistent between the different CFDCs because of differing interpretations of what is required
- Reporting may be inconsistent over time within individual CFDCs
Again, while this subject is largely outside of the scope of this engagement, it was noted that at the time of this report, WD was organizing training across the entire region for staff of the CFDCs to improve the reporting process.
Quality Assurance Review and Improved File Documentation
In order to address concerns raised in the 2001 Report regarding the adequacy of file documentation and systematic review of CFDCs, WD commenced several initiatives. In terms of direct monitoring of the CFDCs, two of the key initiatives were the implementation of a risk assessment process for individual CFDCs (tied to a CFDC site visit program) and the implementation of a Quality Assurance Review Checklist for CFDC files.
- Risk Assessments: For each CFDC, a “WD Annual CFDC /
WEI / FEDO Project Risk Assessment” form is to be completed
based on 11 criteria including CFDC history, operations, experience
and turnover of management, record of compliance and meeting of
targets, etc. A numerical risk rating is computed. This rating
drives the degree of monitoring that is required by WD. This Risk
Assessment form is included as Appendix A-9.
Prior to the implementation of this form, a more generic “WD
Project Risk Assessment” form, which uses some of the same
criteria, was available to be used by WD Staff.
- Quality Assurance Review Checklists: These forms are designed to track each CFDC's compliance with the terms of the CF Contract and the WD Staff's review of these documents. The documents listed are the same as discussed earlier in this report under “Monitoring and Control System” (Page 11). This checklist is included as Appendix A-10.
For the 20 CFDC files selected, these processes were reviewed for the 2002-2003 fiscal year. The results of this review are summarized below. Complete results are in Appendix A-2 to this report.
| Process | Degree of Implementation | |
|---|---|---|
| Risk Assessment Process | “WD Annual CFDC Project Risk Assessment” was completed during 2002-2003 | 5 files, 25% |
| “WD Project Risk Assessment” had been completed during 2002-2003 | 8 files, 40% | |
| No risk assessment had been completed during 2002-2003 | 7 files, 35% | |
| QAR Checklist | QAR Checklist was on file for 2002-2003(in various stages of completion) | 7 files, 35% |
| QAR Checklist not on file for 2002-2003. However, a similar checklist with largely the same fields was being used. | 6 files, 30% | |
| No checklist on file for 2002-2003 | 7 files, 35% | |
It is important to note that these results tend to correspond to the different provinces, which appear to be in different stages of implementation of these practices.
British Columbia:
For the seven files reviewed in B.C., it was found that for the most part they did not contain the QAR Checklist or the CFDC Risk Assessment checklist. However, they were also equally consistent in that “WD Project Risk Assessment” forms had been completed for the current and past years, and that the files contained a checklist similar to the QAR Checklist. It appears that the staff is inconsistent in using the new forms – although these processes are in place.
Alberta:
The results for Alberta were very similar to B.C. In the current year, QAR Checklists were on file, although some did not appear to be up-to-date. While risk assessments were not done in the current year, all files reviewed did have risk assessments completed using the “WD Project Risk Assessment” form during the prior year. Staff indicated that the current year's risk assessments would be done.
Saskatchewan:
It appeared as though Staff in Saskatchewan were just implementing these procedures. One of the three files reviewed had both a QAR Checklist and a risk assessment, while the other two files had neither. Since only three files were reviewed, it is difficult to speculate for the province as a whole. Staff indicated that these processes would be in place by the end of the current year.
Manitoba:
All of the four files reviewed in Manitoba contained completed risk assessments. Also, Staff were able to provide a copy of a summary of the risk assessments for all of the CFDCs in Manitoba. The checklists reviewed also evidenced site visits for two of the four files (one considered medium risk, one considered high). It appears as though Manitoba has fully implemented the risk assessment process. However, none of the four files contained QAR checklists. Similar to Saskatchewan, Staff indicated that these would be in place at the end of the current year.
The 2001 Report also indicated that, while the compliance certificates
that would be provided by each CFDC's external auditors would
provide good information on the overall lending activities of each
CFDC, these certificates were found in only half of the files examined.
Based on the review of files completed as part of this audit, it
appears that the filing of compliance certificates has improved
dramatically. For the year ended March 31, 2001, there was only
1 exception, and none for the year ended March 31, 2002. The filing
deadline for compliance certificates for the year ended March 31,
2003 is July 31, 2003.