Home : Reports and Publications : Audit & Evaluation : Canada Business Service Centres in Western Canada: Evaluation Report 2002
Canadian Business Service Centres
in Western Canada:
Evaluation Report 2002
1.0 Evaluation Purpose
Western Economic Diversification Canada, Audit and Evaluation Branch (WDAE), retained the Macleod Institute to carry out an evaluation of the Canada Business Service Centres (CBSCs) in the four western provinces. WDAE defined the purpose of the evaluation by saying it will "provide senior management with an independent examination and assessment of the CBSCs, advising on their relevance, success and effectiveness." (Terms of Reference, Appendix A)
2.0 Methodology
The Macleod Institute applied a methodology that is consistent with the National Evaluation Framework. Eight research questions were framed to focus the evaluation on three issues:
| 1. | Relevance: | Are client needs being met? |
| Are Partner needs being met? | ||
| Is the mandate relevant? | ||
| 2. | Success: | Have CBSCs produced their intended outputs? |
| Have CBSCs achieved their intended outcomes? | ||
| Were there unintended impacts? | ||
| 3. | Effectiveness / | Is there a more cost-effective way to deliver the program? |
| Alternatives: | Are there alternative ways to design and deliver the program? |
Multiple lines of evidence were used to address each evaluation
question. A summary of the indicators for each question and the
source of data for each indicator is presented in Figure 1.
FIGURE 1: Evaluation Questions, Indicators and Sources of Data
| EVALUATION QUESTIONS | INDICATORS | DATA SOURCE |
|---|---|---|
| RELEVANCE | ||
|
Are client needs being met? |
Information ~ |
Client Survey |
|
Are Partner needs being met? |
Alternate sources |
Client Survey |
|
Is the mandate relevant? |
Mission |
Client Survey |
| SUCCESS | ||
|
Have CBSCs produced their intended outputs? |
Information and
referrals |
Client Survey |
|
Have CBSCs achieved their intended outcomes? |
Mission |
Client Survey |
|
Were there unintended impacts? |
Identified impacts |
Client Survey |
| EFFECTIVENESS / ALTERNATIVES | ||
|
Is there a more |
Willingness to
Pay |
Client Survey |
|
Alternative ways to design and deliver the program? |
Delivery of Services |
Client Survey |
Five lines of evidence were used to evaluate the CBSCs: a Client Survey, a Potential Client Survey, a Staff Survey, interviews with senior officials and a review of pertinent documents.
Client Survey
Data with respect to CBSC clients were collected using a web-based survey instrument (survey questions are listed in Appendix B). A client was defined as any individual who had contacted a Centre with a request for business information. Prospective respondents were randomly selected from lists provided by each of the CBSCs in British Columbia, Alberta, Saskatchewan and Manitoba. Sampling techniques are summarized in Appendix C. E-mails were sent to prospective respondents with an invitation to respond directly on-line. If no response was received within ten days, the original e-mails were followed up by both e-mail and telephone reminders. Additional prospective respondents were invited to participate by telephone. Both on-line and telephone responses to the survey instrument were entered in the Client Survey Data Base. Statistics regarding the Client Survey are summarized in Table 1. Survey results are summarized (without attribution) in Appendix D.
Table 1: Client Survey Statistics
| BC | AB | SK | MB | Totals | |
|
Initial Sample Size* |
222 | 350 | 139 | 160 | 871 |
| Responses (#) | 66 | 69 | 63 | 78 | 282** |
| Response Rate (%) | 29.7 | 19.7 | 35.3 | 48.6 | 32.3 |
| Precision @ 90% confidence level | +/- 10.1% | +/- 10% | +/- 10.4% | +/- 9.4% | +/- 4.9% |
NOTES:
* - Sampling frames differed between CBSCs. The results
are weighted towards walk-in and telephone clients because of the
way sample populations were gathered. In BC, they are also heavily
biased in favour of recent clients. Details are provided in Appendix
C.
** - In fact, 300 responses were received, but the province
of origin could not be
established for 18. The total stratified by province is therefore
282.
Potential Client Survey
Data with respect to potential clients were collected using a web-based survey instrument (survey questions are listed in Appendix B). A potential client was defined as any individual in the small and medium-sized enterprise (SME) business sector who had never used a CBSC to obtain business information. Prospective respondents were randomly selected from lists of SMEs for each province. The lists were obtained from a market research corporation. Sampling techniques are summarized in Appendix C. E-mails were sent to prospective respondents with an invitation to respond directly on-line. If no response was received within ten days, the original e-mails were followed up by both e-mail and telephone reminders. Additional prospective respondents were invited to participate by telephone. Both on-line and telephone responses to the survey instrument were entered in the Potential Client Survey Data Base. Statistics regarding the Potential Client Survey are summarized in Table 2. Survey results are summarized (without attribution) in Appendix D.
Table 2: Potential Client Survey Statistics
| BC | AB | SK | MB | Totals | |
|
Initial Sample Size |
500 | 500 | 500 | 500 | 2,000 |
| Responses (#) | 21 | 31 | 46 | 36 | 134 |
| Response Rate (%) | 4.2 | 6.2 | 9.2 | 7.2 | 6.7 |
| Precision @ 90% confidence level | +/- 17.9% | +/- 14.8% | +/- 12.1% | +/- 13.7% | +/- 7.1% |
Staff Survey
Front- and second-line CBSC staff were polled using a web-based survey instrument (survey questions are listed in Appendix B). E-mail addresses were obtained from CBSC staff directories. E-mails were sent to prospective respondents with an invitation to respond directly on-line. If no response was received within ten days, the original e-mails were followed up by e-mail reminders. On-line responses to the survey instrument were entered in the Staff Survey Data Base. Statistics regarding the Staff Survey are summarized on the next in Table 3. Survey results are summarized (without attribution) in Appendix D.
Table 3: Staff Survey Statistics
| BC | AB | SK | MB | Totals | |
|
Initial Census Size |
14 | 18 | 10 | 17 | 59 |
| Responses (#) | 8 | 14 | 7 | 17 | 46 |
| Response Rate (%) | 57.1 | 77.8 | 70.0 | 100.0 | 78.0 |
Senior Officials Interviews
The Institute conducted personal interviews with 37 senior officials of the Managing and Operating Partners and National Secretariat, selected in consultation with WDAE. When respondents were not available for face-to-face interviews, they were interviewed by telephone. An interview guide was developed to help ensure sufficient commonality of information collected for analysis, while also allowing greater flexibility than a survey questionnaire does for exploring areas of particular interest to the evaluation. A list of interviewees and a copy of the interview guide are provided in Appendix E.
Document Review
A number of documents were reviewed, including Partnership Agreements, annual reports and operating plans, usage statistics by mode of access, previously completed evaluation reports and statistical reports provided by the four CBSCs. A list of documents is provided in Appendix F.