Alternative Service Delivery (ASD) Pilot
Evaluation Study Report

Audit and Evaluation Branch
Industry Canada

October 29, 2003

Executive Summary

Background

Aboriginal Business Canada, a branch of Industry Canada, works with Aboriginal entrepreneurs to promote the development, competitiveness and success of Aboriginal business in order to build a competitive, sustainable Aboriginal economy actively linked to the economies of Canada and the world. Aboriginal Business Canada has four strategic priorities:

• Expand markets and trade opportunities (including tourism).
• Encourage innovation.
• Support youth entrepreneurship.
• Strengthen Aboriginal financial and business development organizations.

One of its programs, Aboriginal Business Development (ABD) provides financial support (contributions) to existing and potential Aboriginal businesses to encourage viability and self-reliance. The program is delivered by Aboriginal Business Canada staff in nine urban areas, by third-party delivery arrangements with External Delivery Organizations (XDOs) in other areas and recently on a pilot basis by an enhanced form of XDOs called Alternate Service Delivery (ASD).

Compared to XDOs, ASDs operate in more remote areas, serve all eligible groups within their coverage areas, and make decisions related to contributions in conformance to ABD guidelines and procedures from an equity fund they maintain on behalf of the program. Other features of delivery remain the same between XDOs and ASDs, and the same Accountability Accord governs each groups' operation.

A pilot evaluation study was conducted of ASD in the third year of this pilot phase. This evaluation addressed a number of specific questions within the general evaluation issues of program rationale, program design, program and service delivery, client reach, and pilot program impacts.

This chapter summarizes the key findings of the evaluation of the Alternative Service Delivery (ASD) Pilot.

Rationale, Clarity of Intent and Appropriate Design Choice

Key findings related to the rationale, clarity of intent and appropriateness of design choices made are:

• There is strong agreement across all information sources available to the evaluation that the rationale for ASDs is to:
• Improve client service.
• Build capacity.
• Rationalize delivery.

• The evaluation found that client service, other than speed of service, was improved. Organizations that became ASDs were already operating at a high level of delivery capability. This capability may have been increased slightly through becoming an ASD. According to some, delivery efficiencies may have resulted through adding ASD to the existing multiple program responsibilities of the organizations.
• Although objectives and desired outcomes may have been clear the method to monitor achievement of goals was not. Also some improvements could be made to ease the reporting burden for ASDs as well as to facilitate more efficient tracking of ASD project details within ABD data systems through greater use of Project Summary Forms.
• ASDs felt that the number of projects and contribution levels delivered through ASDs warranted the creation of the pilots. The number of projects less than or equal to $50,000 grew by 49% in ASD coverage areas (two years pre-ASD compared to second year of ASD pilots) while it declined by 24% in other areas. Growth in the number of projects of this size in ASD coverage areas has occurred across all strategic priority areas although the small number of cases make the analysis tentative. At the same time the average dollar value of such projects in ASD areas rose from $11,994 in the pre-period to $15,370 in the second year of operation as an ASD. For non-ASD areas the average value had increased marginally from $11,014 to $11,276. As a result, contributions to projects less than or equal to $50,000 have increased by 91% in ASD areas compared to a decline by 22% in non-ASD areas. This may indicate success for the ASDs as they flow more equity support and approve more projects to Aboriginal businesses in their area. However, data limitations limit this conclusion. Another equally plausible explanation is this growth reflects the normal growth (unaffected by the creation of ASDs) in their coverage areas. Data from another year of ASD experience would help determine which interpretation is correct

Program and service delivery

A number of observations arise from the evaluation related to program and service delivery under ASDs:

• Roles and responsibilities provided to ASDs under the pilots appear appropriate.
• ASDs are likely more timely in terms of disbursements once projects are approved but are not faster at approving projects. Performance measures for non-ASD projects suggest that 85% of business plan assessments occur within 65 days. For comparison only 60% of applications to ASDs are approved within two months. ASD clients compared to non-ASD clients of ABD are more dissatisfied with speed of service. Speed of service for ASDs could be improved if approval for small value or low-complexity projects could be made without full Board approval.
• In addition to more speedy disbursements one ASD uses a third party to achieve cost-effective disbursements.
• ASDs appear to have done a good job of promoting themselves and Aboriginal Business Canada among the communities they now represent.
• A training package and procedures manual specific to ASDs would be appropriate if ASDs were extended. Mentoring opportunities for new development officers with established ASDs would be useful if the ASD approach was extended.
• ASDs may be able to make better funding decisions due to their local knowledge. The evaluation is not able to assess this assertion but consistent use of Business Performance Reviews (follow-up surveys) to all ABD clients could facilitate such an assessment.
• Given data limitations and the uncertainties of key informants the evaluation is not able to make definitive statements about the cost-effectiveness of ASDs.

Impacts of ASDs

The evaluation observed the following short-term impacts of the ASD pilot:

• ASDs are supporting projects in all Aboriginal Business Canada Strategic Priority areas and as a result their clients are able to take advantage of the diversity offered by the program criteria.
• Clients appear satisfied with the service they are receiving through ASD. Client satisfaction is similar to satisfaction levels by Aboriginal Business Canada clients more broadly. However, ASD clients appear more likely to have concerns with speed of service. Approval times for ASDs were found to exceed those for other clients of ABD.
• Clients are made aware of ASD delivery through referral by other groups (possibly the result of prior promotional activities of the ASDs), by ASD directly, or through their own initiative.
• A few clients report there have already been some spin-off benefits to them in terms of improved access to other funding or increased credibility of their organization through their involvement with ASDs. Most client involvement is recent (many projects are still underway or only recently completed) limiting the opportunity for such spin-off benefits.
• ASDs were already advanced when selected for pilots. Nevertheless, one-half of key informants report that ASD institutional capacity is increased through greater knowledge and experience in the delivery of ASD.
• ASDs were already delivering a number of other Federal programs. Again about one- half of key informants felt there would be some improvement/efficiency gained when ASD delivery was added to the ASD portfolio of programs.
• ASDs follow the same procedures as other delivery mechanisms for the ABD. As a result it supports Industry Canada's sustainability goals — that development meet the needs of the present without compromising the ability of future generations to meet their own needs.

Pilots were started in September 2000. This evaluation was conducted in early 2003. Data reflecting the ASD experience was available for only one full year (September 2001 to August 2002) due to phase-in of the pilots. As a result, the evaluation could not be definitive about longer-term impacts. Longer-term impacts could be addressed by the next summative evaluation of ABD.

Key Findings

The evaluation found no significant problems related to the delivery by ASDs. Key findings were:

• A strong rationale for ASDs exists that was agreed to by all parties.
• Client service other than speed of service is improved through ASD. The time to approve projects appears longer and time to disburse funds appears shorter for ASDs relative to other forms of ABD delivery.
• The number and value of projects increase in ASD areas compared to non-ASD areas in the pilot period.
• Projects have been approved in all Aboriginal Business Canada Strategic Priority areas suggesting clients of ASDs are able to take advantage of the diversity offered by the program criteria.
• Promotion by ASDs appears to be effective through its reach to eligible clients or to those who refer eligible clients.

However, positive findings of the evaluation are weakened due to:

• The possibility that the increase in the number and value of projects is due to the natural growth in ASD areas (not accounted for by the implementation of ASDs).
• Concern that increases in the value of projects in ASD areas might reflect an increase in the proportion of equity support to safeguard loan re-payments when a single organization is responsible for both equity and loan support decisions.
• A limited amount of experience with the pilots (only one year of post-implementation data on the number and value of projects could be used) and an absence of key data which could strengthen evaluation findings.

These positive yet tentative conclusions of the evaluation lead to a series of recommendations to:

• Continue the current ASD pilots until March 31, 2006. During that time gather needed data to support a decision related to the future of ASDs and make changes to improve on the design of the ASD delivery. Before the end of the extended period undertake a management review of ASD to determine if it should be a permanent delivery mechanism for ABD
• Release equity funds based on receipt of completed Project Summary Forms (PSF) for approved projects within the coverage area of the ASD. Record PSF information within the Contribution Management Information System (CMIS). Ensure ASDs have an adequate "float" (provide an advance if needed) to ease the transition to the new system of transfers for equity funds. (This recommendation is conditional on a timely and efficient method (direct deposit/e-claim) to transfer funds to ASDs.)
• ASDs should adopt and report on comparable performance measures as are/will be used for ABD. ASDs should conduct Businesses Performance Reviews (BPRs) on all completed projects before the end of the extended pilot period. CMIS or an equivalent system should be updated to reflect data collected through the BPRs.
• For ASDs with a proven record delivering ABD, increase their project authority limits from $50,000 to the Aboriginal Business Canada contribution limit for individual businesses (currently $75,000 but expected to rise to $100,000). For community-based projects adopt the same contribution limit. Note projects under the priority area Strengthen Aboriginal Financial and Business Development
• Organizations should continue to remain the responsibility of Aboriginal Business Canada.
• Encourage each ASD to make a delegation proposal to Aboriginal Business Canada. Such a proposal must include: a method to identify projects that do not warrant the approval of an ASD Board or Project Review Committee (PRC); an appropriate alternate mechanism for approval of such projects; and a method to inform the Board/PRC on the approval/rejection/deferral of such projects.
• Develop a framework to ensure collection of cost data to permit a cost-effectiveness analysis of ASD. Both the direct cost of third party delivery and indirect costs (net cost related to other XDO and government staff) should be determined. Two ASDs (one in each of East and West) and one or two comparable XDOs in the East and West should be used to provide sufficient data yet keep study costs manageable.

Final Report (PDF, 307KB, 41 pages)

Management Response — January 2004 (PDF, 103KB, 5 pages)

Management Response Update — May 2005 (PDF, 21KB, 5 pages)

Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.