Home : Reports and Publications : Audit & Evaluation : Evaluation of the ABSN
Evaluation Methodology
Evaluation Overview
Evaluation Purpose
The primary purpose of this evaluation was to assess the ABSN initiative and the degree of success achieved in reaching its goals and objectives. As a result, areas for improvement have been identified and recommendations made with respect to future initiative directions. This evaluation assessed the relevance, success, and effectiveness of the ABSN with particular reference to the April 1999 program submission to Treasury Board. That initial document outlined the following expected results:
- Increased awareness by Aboriginal business and business intermediaries;
- Increased utilization by Aboriginal business and business intermediaries; and
- Increased service and support capacity of partnering Aboriginal business development organizations.
Program Logic Model
An ABSN Program Logic Model was developed by the evaluators, based on the Request for Proposal documents, the April 1999 ABSN submission to the Treasury Board, and the input of regional ABSN program stakeholders. The model clearly delineates the links in the initiative process, from initiative supports and inputs, to the implementation process, to initiative outputs. It also identifies initiative outcomes including short-, intermediate-, and long-term outcomes. The model also provides a thumbnail sketch of the evaluation plan and is presented below.
|
ABSN Program Logic Model
|
|||||
Process |
Outcomes |
||||
|---|---|---|---|---|---|
Administrative Supports/Inputs |
Implementation Process |
Outputs |
Short-Term Outcomes |
Intermediate-Term Outcomes |
Long-Term Outcomes |
|
– Aboriginal community's need for tailored business information is identified – ABSN fits with WD priorities – ABSN has links to CBSC/ existing infrastructure – Management/staff structure is efficient – Partners/business intermediaries are identified – Adequate resources are provided – Other supports are provided as needed |
– A core product line of business information is provided – Networks of Aboriginal business/economic service providers are developed – Advice/ counselling/ referral mentoring services are provided – A service delivery network is developed & implemented – Marketing & awareness activities are provided |
– # Web hits – # Publications – # Contacts – # Referrals – # Other services – # Workshops & attendees – # Other events & activities – Satisfaction with services & products – Evaluations |
– Aboriginals' recognition of the need for business information is increased – Aboriginal businesses have increased awareness of ABSN services – Aboriginal businesses have increased access to ABSN services – ABSN services are used more frequently by Aboriginal businesses |
– Partners/ business intermediaries increase supports & services for Aboriginal businesses – Provincial/ national networks of Aboriginal business & economic service providers are strengthened |
– Awareness of entrepreneurship as an option for Aboriginal people is increased – Aboriginal entrepreneurs have: – Increased access to capital |
Initiative supports and inputs include the Aboriginal community’s need for tailored business information; the extent to which ABSN fits with WD’s priorities; ABSN’s links to CBSC and the existing infrastructure; the efficiency of the management and staff structure; the identification of partners and business intermediaries; the provision of adequate resources; and the provision of other supports as needed.
The implementation process includes the provision of a core product line of business information; the development of networks of Aboriginal businesses and economic service providers; the provision of advice, counseling, referrals, and mentoring services; the development and implementation of a service delivery network; and the provision of marketing and awareness activities.
Initiative outputs include the number of hits to the ABSN websites; the number of publications; the number of contacts established; the number of referrals made; the number of other services provided; the number of workshops conducted and the number of attendees at these workshops; the number of other events and activities; the level of satisfaction with the services and products; and the evaluations.
Short-term outcomes include Aboriginal people’s increased recognition of the need for business information; Aboriginal businesses’ increased awareness of ABSN services; Aboriginal businesses’ increased access to ABSN services; and Aboriginal businesses’ more frequent use of ABSN services.
Intermediate-term outcomes include increased provision of supports and services for Aboriginal businesses by partners and business intermediaries; and strengthened provincial and national networks of Aboriginal business and economic service providers.
Long-term outcomes include increased awareness of entrepreneurship as an option for Aboriginal people; and increased access to capital, increased success with business start-ups, and increased success with business growth for Aboriginal entrepreneurs.
Based upon this model, a Data Collection Matrix (DCM) was designed to guide the evaluation research activities. Input from the regional ABSN program stakeholders was used to guide development of the Matrix along with the Request for Proposal documents and the April 1999 ABSN submission to the Treasury Board. A copy of the DCM is provided in Appendix 1.
Evaluation Objectives
A number of objectives were identified for the evaluation, based on the ABSN Program Logic Model and the purpose of the evaluation. These objectives include:
- To describe the background and development of the ABSN in the four Western regions;
- To describe the implementation of the ABSN in the four Western regions;
- To determine the success of the ABSN in achieving its short-, intermediate- and long-term outcomes;
- To determine the continued relevance of the ABSN; and
- To determine the effectiveness of the ABSN in meeting its objectives compared to other alternatives.