Home : Publications : Audit & Evaluation : Canada Business Service Centres in Western Canada: Evaluation Report 2002

Evaluation of the Canada Business Service Centres in Western Canada - July 2002

Evaluation Report Summary

Table of Contents

1. Evaluation Purpose
2. What are the Canada Business Service Centres (CBSCs)?
3. Evaluation Process
4. Highlights from the Evidence
5. Conclusions
   1. Relevance
   2. Success
   3. Effectiveness (Alternatives)
6. Recommendations

1.0 EVALUATION PURPOSE

The purpose of the evaluation was to provide senior management with an independent examination of Canada Business Service Centres (CBSCs) in the four western provinces, advising on their relevance, success and effectiveness (alternatives).

2.0 WHAT ARE THE CANADA BUSINESS SERVICE CENTRES (CBSCs)?

CBSCs are a network of offices located all across Canada. In western Canada, the Centres are known locally by names such as 'Business Services' or the 'Business Link' (see Figure 1). Most of the Centres have their main offices in major cities, but they also support a large number of regional access centres in smaller locations spread across each province.

Figure 1: An Overview of CBSCs in the Four Western Provinces

CBSC Mandate

CBSCs aim to serve as Canada's primary source of timely and accurate business-related information and referrals on federal programs, services and regulations, without charge, in all regions of the country.

CBSC Partnerships

CBSCs were designed to give Canadian small businesses (SMEs) a 'one-window' approach for business information. Behind the scenes, however, about 37 different federal, provincial and municipal departments work with the Centres to provide the information needed by clients. In addition, the regional access centres are usually operated in partnership with local public or private sector organizations.

CBSC Staff & Budget

In the four western provinces, CBSCs are staffed either by federal, provincial and municipal employees or by individuals hired directly by the Centre. Western Economic Diversification Canada and provincial economic departments finance the operation of each Centre (the Business Link also augments its funding through revenues from other sources). A National CBSC Secretariat, located in Ottawa, is responsible for maintaining uniform computer networks, information databases and products for all Centres across Canada.

CBSCs are busy offices

Between April 1, 2001 and March 31, 2002, western Canadian CBSCs dealt with over

• 1,200,000 website visits
• 118,000 walk-in clients,
• 87,000 personal telephone contacts, and
• 30,000 automated telephone calls.

The way in which clients are accessing business information is changing. More and more people are visiting the website and contacting CBSCs by e-mail. Walk-ins are still a significant mode of access, but they are trending downwards - from 200,000 in 1999/00 to 118,000 in 2001/02. Generally, across all four provinces, CBSC records show that clients are using self-serve options more frequently and that face-to-face interactions with CBSC staff have decreased in relative terms.

About one-third of the clients who use CBSCs in Manitoba, Saskatchewan and Alberta do so because they are thinking of starting a business. Another third already own a business and are looking for information on how to operate it more efficiently. The rest of the clients are professional advisors or researchers. BC is an exception to this pattern - 60% of the clients in that province use CBSCs for business start-up information.

CBSCs provide a basic level of information to their clients.

Figure 2: The Information Services Curve (adapted from the Manitoba CBSC)

CBSC Results Chain

CBSCs are expected to improve business literacy in Canada's small business (SME) sector. The way they achieve this outcome is illustrated on the next page. Activities related to information services, marketing and partnerships produce a number of products, listed in the results chain diagram as outputs. These products are intended to achieve three direct results: (1) SME access to business information; (2) SME business planning and analysis; and (3) a single portal (one-window) to multiple sources of information. The direct results are expected to lead to the overall outcome - improved business literacy.

3.0 EVALUATION PROCESS

Eight research questions were used to focus the CBSC Evaluation on three issues:

Five lines of evidence were developed in order to answer the research questions: a survey of small businesses who use CBSCs (the Client Survey); a survey of small businesses who do not use CBSCs (the Potential Client Survey); a survey of CBSC staff (the Staff Survey); interviews with 37 managers in Western Economic Diversification Canada, provincial economic development departments and the CBSC National Secretariat (Senior Officials Interviews); and documentary evidence (the Document Review).

Client Survey

Clients were randomly selected from lists provided by CBSCs in each of the four western provinces. Most surveys were completed over the Internet, although some people preferred to answer questions over the telephone. Survey statistics are provided in Appendix A.

Potential Client Survey

Small businesses were randomly selected from lists of SMEs in each province. Again, survey questions were answered either over the Internet or by telephone, depending on the individual's choice. Survey statistics are provided in Appendix A.

Staff Survey

All front-line and second-line CBSC staff were polled using a web-based survey instrument. Survey statistics are provided in Appendix A.

Senior Officials Interviews

Most of the 37 interviews were conducted in person. When respondents were not available for face-to-face interviews, they were interviewed by telephone.

Document Review

A large number of documents were reviewed, including Partnership Agreements, annual reports and operating plans, usage statistics by mode of access, previously completed evaluation reports, statistical reports provided by the four CBSCs, and reports produced by other government departments or by international benchmarking services.

4.0 HIGHLIGHTS FROM THE EVIDENCE

Profile of the SME Business Sector

According to Statistics Canada data, small and medium sized enterprises (SMEs) make up 99% of the total number of businesses in western Canada. The following figure provides a breakdown of the number of people employed by each SME in western Canada.

Figure 4: Number of People Employed by SMEs in Western Provinces (1998)
StatsCan, 1983 - 1998 Employment Dynamics (2001)

Profile of CBSC Clients

Most CBSC clients who responded to the Client Survey were well educated. They ranged in age between 25 and 55 years and a slightly larger proportion were male. Over 80% reported that English was their language of choice.

Figure 5: Profile of Respondents - Client Survey

Answers to Research Questions

Clients were asked to assess their levels of satisfaction regarding CBSCs. Almost 82% indicated they were satisfied with the information they received in the last twelve months. Women were more likely to be satisfied than men. By comparison, far fewer potential clients indicated they were satisfied with the non-CBSC sources of business information they used. Satisfaction ratings ranged between 40 and 60%, depending on what kind of information the potential clients were looking for.

Figure 6: Clients’ General Satisfaction

Reported levels of satisfaction do not, in themselves, tell the whole story. It is also important to know what kind of information clients are using and how useful clients believe this information is. Figure 7 shows that more clients used information for business operations than for any other single purpose. In addition, more clients reported that this category of information was useful.

Figure 7: Client Information - Used and Useful

Both CBSC staff and senior officials were asked a series of questions addressing various Partner needs. Satisfaction levels for each type of Partner are illustrated in Figure 8. Generally, senior officials were more likely to be satisfied with current arrangements than CBSC staff.

Figure 8: Meeting Partner Needs - Staff and Senior Official Responses

Most Partners want the public to be aware of their efforts on behalf of small business. However, when potential clients were asked whether they had ever heard of the CBSCs, two-thirds indicated they had not. Even clients often failed to recognize the trade name "Canada Business Service Centre" until it was referred to by its regional name - for example, the Business Link. CBSCs in all four provinces identified name recognition (branding) as an unresolved issue.

The CBSC mandate was stated earlier on page 1 of this summary. The mandate has five components, each of which was reviewed to determine whether it accurately reflects what CBSCs are actually doing.

1. To serve as the primary source: The word 'primary' has several meanings. In practice, CBSCs are interpreting it to mean a basic level of information. As shown in Figure 2, the Centres deliver information and services at a primary level.
   
2. Timely and accurate business-related information and referrals: The Client Survey, Staff Survey and Senior Officials Interviews all indicated that CBSCs are delivering timely and accurate information and referrals. However, the mandate fails to mention that Centres also provide some support and advisory services.
   
3. Federal programs, services and regulations: The mandate refers only to federal programs, services and regulations. In fact, CBSCs also distribute provincial information.
   
4. Without charge: This component of the mandate does not accurately reflect what CBSCs are actually doing. For example, clients pay a fee for on-line business registries and some Centres charge a fee for customized business information reports.
   
5. In all regions of the country: All CBSCs have established a network of regional access centres.

Table 1 summarizes the products and services offered by CBSCs in each of the four western provinces.

Table 1: Core CBSC Products and Services (1999 - 2002)

	1999/2000	2000/2001	2001/2002	CBSCs
Business Information System (BIS)				All
Key government publications				All
Government programs and services				All
Other library collections/databases				All
Interactive Business/Export Planners				All
On-line Small Business Workshop				All
Performance Plus (benchmarking)				All
Workshops, seminars etc.				All
Info Guides, Pathfinders etc.				All
Team Canada Inc.				All
On-line business registrations				BC,SK
Advisory services				AB,MB
Talk-to-us!				All
Business Start-up Assistant				All

CBSCs also produce a number of marketing materials intended to catch the attention of potential customers. Marketing tools include brochures, media releases, promotional ads, telephone directory listings and trade show material.

To assess the effectiveness of these products, both clients and potential clients were asked how they first learned of CBSCs. Most people heard about CBSCs by word of mouth or were referred to the Centres by a government agency, rather than through the Centres' marketing materials.

Figure 9: How Small Businesses Heard about CBSCs

A key question of the evaluation addressed whether CBSCs produced the results they were designed to achieve (as illustrated in the Results Chain - see Figure 3 at page 3 above). Three direct results (outcomes) were examined in detail.

Access to Timely, Accurate and Relevant Business Information and Referrals

Senior officials and staff believe this outcome is fundamental to their mission. Clients also agree, as seen in the following figure where clients indicated that CBSCs helped improve their access to relevant business information.

Figure 10: Clients’ Perception of Outcomes

SME Participation in Business Planning and Analytical Practices

Many clients reported that the planning and analytical tools and services provided by CBSCs have been helpful in their businesses. One person said the most significant impact on his or her business was the "potential problem areas / risks" that CBSC staff had helped identify. Another reported that staff had "got us thinking about all possible angles instead of straight ahead." A third mentioned that she received "excellent information to help me make informed decisions," which will "help me serve my clients better." Not all responses were positive, however. One individual sent the message that "I’m still searching and in danger of losing $600, partly from not having info."

Senior officials also recognized that SMEs are participating as intended. "Building more diagnostic and interactive tools helps bring clients along", was the way one manager put it.

However, CBSCs are being pushed beyond their mandate to provide primary level services, both by clients and by the Mission Statement they have adopted. As one executive said, "a web-focus brings in a more sophisticated client, increasing the level of service demand, and the need to train our staff." The Mission Statement tends to emphasize results such as " enable clients to make well-informed business decisions in a global economy" and encourages CBSC staff and management to take on responsibility for outcomes that are beyond their control.

Single Portal to Multiple Sources of Information

CBSCs judge themselves on the one-window approach by measuring increased awareness. As mentioned earlier, only one-third of the potential clients who were surveyed had ever heard of CBSCs. This result needs to be put in context. A much larger survey (over 1,000 SMEs) reported that only one out of every two small businesses in Canada knew the federal government provides business-related information on the Internet. However, one out of three had heard about CBSCs, which demonstrates that the Centres are one of the better known sources of information.

Another way of judging whether the single portal has been successful is to determine whether it provides one convenient service rather than sending clients to several different locations in order to complete their transactions. Looked at in this way, it is fair to say that CBSCs have achieved success in terms of simple transactions like business registrations and interactive tools like the Business Planner. More complex transactions usually require that multiple services across several government departments be coordinated (often called 'bundled' service delivery). So far, CBSC partners do not seem to have developed close enough working relationships to bundle their services effectively.

Three other criteria were used to evaluate the CBSCs' one-window approach. The first one is whether the Centres systematically track client information as a way of staying on top of customer needs. This kind of data is not, however, collected. The second and third measures looked at whether services were organized around client needs and whether a individual services (rather than a 'one-size-fits-all' approach) was applied. CBSCs performed well on both counts.

Three main unintended impacts emerged during the course of the evaluation:

1. Competition: The strength and success of the CBSC one-window model has given rise to competition where least expected. Other governments and agencies are adopting a similar approach to deliver their products and services. The federal government’s Government On-Line initiative and the Government of Alberta’s ServiceAlberta website are examples of competition at both the federal and provincial levels.
   
2. Second generation products and services: Clients are requesting second-generation products and services from CBSCs. Initial success experienced by clients has created demand for seminars and educational sessions. The increase in sophisticated clients is putting a strain on CBSC resources.
   
3. Loss of visibility: Loss of visibility is an unintended impact. As a result of numerous partnerships, CBSCs have found it more difficult to have an identity. All Centres are exploring regional alternatives to help build their local identity, explore locally successful program and funding options, and build more partnerships with the private and not-for-profit sectors.

Willingness to Pay

Potential clients were much more likely than clients to indicate that they would be willing to pay for products and services. Figure 11 indicates that there is some potential to recover costs or modest service fees. More than 50% of the clients said they would at least pay direct costs, except in the areas of information about government funding, networking and self-serve computers.

CBSC staff were asked which products and services they would like to charge money for; the column on the right of Figure 11 shows that the majority think clients should pay for counselling, educational seminars, taxation, accounting and legal services, and networking opportunities.

Figure 11: Willingness to Pay - Clients and Potential Clients

The marketplace is in transition. Client profiles are shifting toward established businesses and away from start-up companies. Both clients and potential clients are also rapidly moving away from old technologies such as fax and telephone, and the Internet presents new opportunities as well as new challenges. The second major change is competition. Other public and private sector service providers are competing with one another and with CBSCs to deliver the same kind of services and products that the Centres have offered over the past decade or so. Although leaders in their field, CBSCs are now feeling pressure from rival organizations. Many of these rivals have more resources at their disposal, and some can choose to dominate the field because they fund CBSCs.

Senior officials and staff were asked to make suggestions about improving the CBSC program. Many commented on the need to concentrate on web-based solutions. Virtually everyone agreed that state-of-the-art technology (and skilled technical support staff) will be needed to respond to changing client demands. Better use of current information technologies was also mentioned. For example, one staff member made the simple, but potentially effective, suggestion that CBSCs "use a centralized computer/Internet kiosk for clients to download or print documents (rather than relying on printed brochures and books, which require frequent updates)."

However, no one offered specific suggestions at a strategic level.

5.0 CONCLUSIONS

Relevance: Meeting Client Needs

CBSCs are meeting most client needs. Clients reported that they use new business start-up information and consider CBSC products and services to be helpful. However, CBSCs have not yet fully responded to small business needs for assistance with operational issues. The Centres offer fewer products and services in this area compared to start-up products.

Relevance: Meeting Partner Needs

Overall, basic Partner needs are being met. The referral service is highly rated by clients and Partners also responded favourably when asked about the CBSCs.

Relevance: Mandate

The CBSC mandate is relevant, but it does not fully describe what the Centres are doing.

Clients and potential clients reported a need for management skills as well as business-related information. The Business Link has described the situation well by saying "what you know and what you do with it makes all the difference in the world." CBSCs are responding to both needs, although their mandate makes no mention of management skills as a component of their services. In addition, the mandate fails to mention the small business sector which forms CBSCs' target market.

The mandate also incorrectly describes the kind of information being offered by CBSCs. The Centres deal with far more than just "federal programs, services and regulations". Apart from materials provided by provincial, municipal and private sector Partners, CBSCs have developed and promoted general business data and tools such as the Interactive Business Planner. Furthermore, the mandate incorrectly states that CBSC services and products are delivered without charge. Business registrations typically require payment of a processing fee, and some Centres are in fact charging fees for services (customized reports are an example).

Success: Intended Outputs (Products)

CBSCs are producing their intended outputs, although shifting market demands and new technologies are presenting challenges. The change in emphasis from start-up to operational issues should encourage CBSCs to put a greater amount of effort into products and services directed at ongoing small business needs.

Success: Intended Outcomes (Results)

CBSCs identified access to information and knowledge as their core service. Clients generally agreed that the Centres have achieved this outcome. Although not everyone was satisfied with the time it took to deliver services or products, over 85% of the clients responded that CBSCs helped improve their knowledge of government programs.

In addition, CBSCs have recognized the strategic importance of management skills in the small business sector, and have succeeded in engaging many clients in business planning and analytical practices.

CBSCs were pioneers in developing a one-window approach to service delivery. They have largely succeeded in this area. However, each CBSC has its own data collection system, resulting in a range of data sources and quality across the Centres. Most of the data were activity-based and tracked the number of clients according to what products they used or whether they used telephones as opposed to email. Very little results-based information was available. CBSCs could do more in the way of applying modern customer relationship management techniques to develop information about market potential and customer needs.

The Centres could also do more to explore opportunities around 'bundled' services, in effect becoming service brokers. This approach represents the next level of service in delivering what is often called a 'seamless government' - government services are packaged according to the client's individual requirements rather than offered separately by different departments.

Success: Unintended Impacts

Competition, more sophisticated clients, and loss of visibility were identified as important impacts which had not been intended when the CBSC program was designed.

Effectiveness: Alternatives

Full cost recovery or modest service fees may be an option for some CBSC products and services - a willingness to pay was seen most clearly in the case of potential clients.

Competition, together with changes in market demand, have put a fair amount of pressure on CBSCs. The Centres have succeeded in creating a culture of continuous improvement - the number of new services and business tools introduced over the past three years is proof of their ability to adapt to changing circumstances. However, there was little evidence that CBSCs in the four western provinces have developed a collective strategy for dealing with their current challenges.

6.0 RECOMMENDATIONS

Meet the market head on

• Focus on business operational issues, with the objective of creating a balanced portfolio of information and knowledge products and services that meet the needs of existing businesses as well as those of start-up businesses in the SME sector.
  
• Develop a customer relationship management (CRM) strategy that enhances the CBSCs’ ability to target basic small business information and knowledge needs, while respecting privacy rights. This strategy should also be designed to move beyond activity-based data in order to provide Managing Partners with meaningful information about the SME sector.
  
• Conduct market research to develop a ‘best of breed’ profile of potential clients in the small business sector.
  
• Test the market potential for cost recovery and modest service charges.
  
• If a communications strategy is adopted and implemented, focus on key messages about reliable, impartial and important value-added business data. Be specific about how small business owner/operators will benefit.
  
• Explore the next generation of the one-window approach, extending single access to provide bundled services in addition to bundled information packages.

Meet the competition head on

• Identify best e-government practices and demonstrate that CBSCs in western Canada meet or exceed requirements to satisfy the criteria for success. As a first step, report annual CBSC results using performance measures outlined in international benchmarking reports.
  
• Engage private sector advisory committees in designing web-based services and products.
  
• Explore the feasibility of becoming Government On-line’s lead agency for developing policy and standards related to e-government SME programs. In particular, capitalize on the Centres’ competitive advantages in this field. For example, CBSCs enjoy a dual presence in the SME market - ‘clicks and mortar’ - which could pay dividends if a CRM strategy and potential market research were deployed to best effect.
  
• Continue to explore ways of increasing the profile of CBSCs and Partners. BusinessGateway.gc.ca provides a precedent - it identifies provincial partners by displaying their flag as a logo. Using logos both on-line and off may offer a simple but effective answer to CBSC network members who need recognition.

Look after the fundamentals

• Refine the mandate to reflect what CBSCs in western Canada actually deliver, and align the mission with the mandate. Specifically, focus the mission statement on the three outcomes within a Centre’s control - SME access to business information and knowledge, SME use of planning and analytical practices (management skills), both supported by a single portal with multiple sources of information and knowledge.
  
• Develop a strategic plan.
  
• Continue to provide technical support staff to enhance each CBSC’s capacity to exploit both current and future information technologies, and capitalize on the technology’s flexibility to meet differing regional needs.
  
• Review the survey results with respect to questions. Specifically, Western Economic Diversification Canada and the Treasury Board Secretariat may wish to consider linking potential market research and customer related management strategies with future applications of the Common Measurement Tools (a nationally accepted evaluation protocol).
  
  

APPENDIX A:

Figure A1: Evaluation Questions, Indicators and Sources of Data ¹

EVALUATION QUESTIONS	INDICATORS	DATA SOURCE
Relevance
Are client needs being met?	Information ~ Satisfaction Importance Usefulness	Client Survey Potential Client Survey Staff Survey
Are Partner needs being met?	Alternate sources Accuracy of referrals Service to Partners	Client Survey Potential Client Survey Interviews
Is the mandate relevant?	Mission Overall Outcomes	Client Survey Potential Client Survey Interviews Document Review
Success
Have CBSCs produced their intended outputs?	Information and referrals Partnerships	Client Survey Staff Survey Document Review
Have CBSCs achieved their intended outcomes?	Mission Overall Outcomes	Client Survey Staff Survey Interviews Document Review
Were there unintended impacts?	Identified impacts	Client Survey Staff Survey Interviews
CE/Alternatives
Is there a more cost-effective way to deliver the program?	Willingness to Pay Cost and Benefits	Client Survey Potential Client Survey Staff Survey Interviews
Alternative ways to design and deliver the program?	Delivery of Services Design/Delivery Options	Client Survey Potential Client Survey Staff Survey Interviews Document Review

¹ Questions asked in the Client Survey, Potential Client Survey, Staff Survey and Senior Officials Interviews, a list of all documents reviewed for the evaluation, and a more detailed discussion of evaluation methods are contained in appendices to the full Evaluation Report.

Notes:
¹ Sampling frames differed between CBSCs. The results are weighted towards walk-in and telephone clients because of the way sample populations were gathered. In BC, they are also heavily biased in favour of recent clients.

² In fact, 300 responses were received, but the province of origin could not be established for 18. The total stratified by province is therefore 282.