Aboriginal Business Canada
Audit of Third Party Delivery Arrangements
Audit and Evaluation Branch
Industry Canada
September 2003
Executive Summary
The audit of Aboriginal Business Canada’s (’ABC’s’) Third Party Delivery Arrangements was selected based on a risk
assessment conducted in 2001 that identified third party delivery arrangements as a high-risk area within ABC. Risk in this context refers to elements of ABC’s program delivery that may have a significant impact on the program’s ability to achieve its objectives. The audit considered the soundness of ABC’s risk management strategies and practices, management control frameworks and practices, and information used for decision making and reporting in third party delivery arrangements. This differs from a financial audit, which relates solely to the examination of items of a financial nature. The scope of the
audit included all identified third party delivery agreements under contract with ABC as of February 28, 2003.
The audit was conducted in accordance with the Institute of Internal Auditors’ Standards of the Professional Practice of Internal Auditing, Treasury Board’s Policy on Internal Audit and Treasury Board’s/Institute of Internal Auditors’ Alternative Service Delivery Management Guide and Alternative Service Delivery Internal Audit Guide (’Guides’).
The results of the audit should be used by ABC management to improve existing management practices, controls, processes and procedures.
Background
ABC’s third party delivery arrangements are considered to be joint delivery arrangements, which are arrangements under which both ABC and the third party delivery provider share responsibility for program delivery
through a collaborative approach. ABC has two main types of third party delivery arrangements that were the subject of this audit:
- External Delivery Organization Arrangements (XDO) — Under XDO arrangements, ABC provides operational funding to Aboriginal organizations to allow for outreach activities on behalf of ABC. External delivery officers act on behalf of ABC to deliver ABC’s program in selected areas and all projects are approved and funded by ABC. XDOs arrangements were originally designed to utilize the successful Aboriginal Capital Corporation network to address issues of geographic coverage and build capacity within these organizations.
- Alternative Service Delivery Arrangements (ASD) — Under ASD arrangements, aspects of ABC’s program delivery are transferred to a third party organization that is Aboriginal owned and operated. ASD’s are provided with operational and project funding to deliver the ABC program, and have been given the authority to approve projects up to a maximum of $50,000.
There are four stages typically associated with third party delivery arrangements: Decision Stage, Development and Approval Stage, Operational Stage, and Wind-up/Renewal Stage. As the majority of ABC’s third party delivery arrangements have been in place for several years,
ABC’s third party delivery arrangements currently lie within the Operational Stage. As such, the audit focused primarily on the two key management processes that impact the Operational Stage: management’s review of the implementation of the transition plan, and contract performance monitoring and evaluation.
Scope and Findings
The audit examined ABC’s management controls, risk management practices and information for decision making and reporting in the Decision Stage and Development and Approval Stage covering the initial decision to enter into third party delivery arrangements and the selection and approval processes for external delivery organizations and alternative service delivery organizations. There were no significant findings noted, i.e. no major deficiencies in management practices and controls over these stages were identified. While the processes over the selection of the original XDO arrangements in the early 1990’s were not overly formal or consistently documented, considerable improvement was noted during the ASD selection process in the late 1990’s during which candidates were formally assessed against established criteria, risks were considered in the decision making process, and individual assessments and approvals were formally documented on file. ASD arrangements were made with successful XDOs that demonstrated a high level of management and organizational capacity to assume the additional responsibilities associated with this type of delivery arrangement.
Significant findings from our examination of the Operational Stage of ABC’s third party delivery arrangements have been categorized under three main headings as detailed below.
Management Control Framework
A formal management control framework has not been established over third party delivery arrangements to guide the assessment and monitoring of the significant risks impacting these arrangements, the ongoing management of third party delivery arrangements, and to strengthen accountability. While agreements with both XDOs and ASDs describe expected monitoring practices and reporting requirements of the third party deliverer, a significant amount of flexibility is granted to regional ABC managers in determining the nature and extent of ongoing monitoring of ASD arrangements. This is intended to allow for managers to work with the needs of each individual organization. As a result, the frequency and form of ongoing communications with ASDs and monitoring activities is largely left to the judgement of regional ABC managers. This may result in inconsistencies in the strength of ABC’s relationships with ASDs and the frequency, nature and quality of ongoing monitoring of ASD arrangements.
Although ABC considers risk in its selection of third party deliverers and has established some controls over the selection and operation of its third party delivery arrangements, changes in risk over the life cycle of the arrangements has not been consistently considered and lack detailed supporting documentation.
Further, there may be a lack of understanding both within ABC and within the third party delivery organizations of which ABC representative has overall responsibility for the management of third party delivery arrangements, a lack of common understanding of the intended roles and results of third party deliverers, inconsistencies in the hiring and training of third party delivery officers across the country, and inconsistencies in the
frequency and form of communication between ABC managers and ASD and XDO management across the country.
We recommend that a formal management control and accountability framework be developed for third party delivery arrangements that addresses roles and responsibilities, performance expectations, training requirements, and monitoring expectations. ABC should continue to investigate and implement initiatives which encourage more frequent and ongoing dialogue to discuss opportunities, concerns, and sharing of best practices and should develop risk-based guidelines to assist in the monitoring of third party delivery
hiring practices, identifying minimum qualifications required and considering the differences between the employment pools in remote locations and larger urban centers. ABC’s right to influence the hiring of XDO officers and ASD staff involved in ABC program delivery should be entrenched in its third party delivery contribution agreements.
Performance Measurement System
ABC lacks a formal and comprehensive performance measurement system for the evaluation and monitoring of its third party delivery arrangements. This reduces management’s ability to gain a clear picture of the overall performance and results of its third party delivery programming, and may impede or reduce ABC’s ability to be proactive in responding to issues as they arise.
Opportunities exist to enhance the existing performance measurement system through: the inclusion of qualitative measures in order to provide a well-rounded evaluation of third party delivery performance; requesting input from the third party deliverers in the development of performance measures; the communication of performance measures to all staff which will form part of the accountability framework; and the development of standard benchmarks with detailed guidelines to enable comparison among the third party delivery arrangements.
Monitoring and Reporting
Although ASD agreements require quarterly reporting to
ABC, ASD organizations have not been provided with a clear definition of the expected format or content of these reports. Third party reporting has been mainly informal and inconsistent. While agreements currently define the expected performance measures that should be reported on by the ASD organization, there is no prescribed format for such reporting.
ABC should clarify its reporting expectations and communication guidelines that will form part of the management control and accountability framework, and develop related standard reporting formats to assist with the effective operation of the monitoring process. In addition, monitoring expectations for both individual agreements at the operational level and at the program level should be clearly defined, communicated and followed up. Finally, data gathering and reporting systems should be modified to capture all relevant project information and to support an early warning system to indicate when performance results may not meet expectations.
Final Report (PDF, 282KB, 26 pages)
Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.
Date Created: 2004-05-03
|