2.0 Evaluation Design

2.1 Scope and Objectives

The overall objective of this evaluation was to assess the relevance, service delivery, results and effectiveness of reporting and performance monitoring of the WEI organizations. In order to best utilize consultant resources, during the data collection for the WEI evaluation, a few questions were also asked with appropriate key informants to gather information for the evaluations of the WCBSN (Western Canada Business Service Network), CEDO (Community Economic Development Opportunity) and CEDIP (Community Economic Development Internship Program) that were being conducted at the same time. The perspectives given by the WEI respondents pertaining to these evaluations are found in the individual evaluation reports.

The scope of this current evaluation focused on the perspectives of key stakeholders and clients currently involved with the WEI organizations. Key stakeholders included WD personnel, Executive Directors and Board members of WEI organizations and other relevant provincial agencies/organizations.

2.2 Evaluation Issues and Questions

The following matrix (Table 2) identifies the evaluation issues, questions and data sources that formed the basis for the evaluation. The findings of the evaluation will be presented under these headings. The evaluation issues and questions were approved by the Evaluation Steering Committee prior to the start of the evaluation research.

Table 2: Matrix of Evaluation Issues, Questions and Data Sources
Evaluation Issues and Questions	Data Sources
Evaluation Issues and Questions	Document review	KI interview	Focus Groups	Case studies
Relevance
Is there a need for the Women’s Enterprise Initiative?		[	[		[
Are the objectives and activities of the WEI organizations consistent with WD’s mandate?	[	[			[

Service Delivery of WEI Organizations
Is the governance structure of the WEI organizations appropriate?	[	[			[
Is the WEI appropriately managed by WD?		[
Are the WEI organizations well marketed? Are they well known?	[	[	[		[
Are WEI organizations clients satisfied with the services provided by the WEI organizations?		[	[		[
Are WEI organizations capital assets properly managed? Are the loan loss rates at an appropriate level?	[	[	[		[
Are the WEI organizations loan recovery practices appropriate?		[	[

Results and impacts
Do WEI organizations help identify and fill service gaps facing women entrepreneurs?	[	[	[		[
Are WEI organizations effective in providing services i.e. small business information; referrals to government programs; information sessions, workshops and exhibits to strengthen the skills of women entrepreneurs and to allow them to explore business opportunities; and business management skill development, including marketing and bookkeeping? Do these services result in more effective businesses?		[	[		[
Do WEI organizations help improve access to financing, business counseling, mentoring, networking and loan aftercare?	[	[	[		[
Are WEI organizations delivering an appropriate amount of loans to women entrepreneurs? Are they increasing the availability of loans to women entrepreneurs?	[	[	[		[
Are WEI organizations effective in supporting entrepreneurs to decrease the risk-level of loans?	[	[	[		[
What are the economic impacts of the WEI organizations on women entrepreneurs, the communities, and western Canada’s economy overall? (E.g., businesses created, jobs created, etc.)	[	[	[		[
Are the WEI organizations’ network activities resulting in more effective businesses?		[	[		[

Reporting
Are the WEI organizations reports allow WD to monitor the performance of WD’s investment in WEI organizations?	[	[
Are the deliverables/reports/information prepared relevant to WD’s needs? Do the WEI organizations have systems to measure their performance?	[	[

Cost-Effectiveness
Are WEI organizations cost effective? Are the results of WEI organizations significant considering WD’s investment?	[	[

Alternatives
Are there alternative models to be considered to improve service delivery and impacts of WEI organizations?	[	[	[		[

2.3 Approach and Methodology

The evaluation was designed to evaluate the overall WEI, rather than segment and critique the individual WEI organizations in each province. The evaluation relied on multiple sources of evidence that included key informant interviews, WEI organization client focus groups and thematic mini case studies.

Interviews were held with WD staff responsible for the WEI organization files, WEI organization Executive Directors, Board Members and staff and other agencies that were familiar with the WEI organization in each province. Two focus groups attended by approximately 14 to 20 WEI organization clients were held in each province (the total number of focus group participants – 112). The key informant interviews and the focus group sessions explored issues of relevance, service delivery, impact and results, reporting and cost-effectiveness.

The focus groups were reflective of the client base served by the WEI organizations across the province. The groups were comprised of loans and services clients in the retail, services and manufacturing sectors and included:

New business start-ups;
Expansion oriented individuals;
Non-growth oriented individuals;
Individuals who had successfully obtained a loan (s) through the WEI organization; and
Individuals who had not been successful in obtaining a loan through the WEI organization.

The groups were held in a neutral environment and the discussion was facilitated by a member of the consulting team. Discussions lasted approximately two hours and sessions were recorded by professional notetakers contracted for that purpose. The focus group perspectives provided an important source of information for the evaluation and were utilized in the place of a client survey. All the evaluation issues (relevance of the WEI, service delivery, results and success, cost-effectiveness and reporting) were covered during the focus group sessions. Although the focus group information is not as easily quantifiable as survey data, the analysis of the information provides a rich source of WEI organization clientele’s personal experiences and insights as to the impacts of the WEI.

As an additional line of evidence, mini case study themes were selected through consultation with WD staff and were chosen based on implications for the future functioning and success of the WEI organizations. The four thematic areas examined in the case study material are:

Mentoring
Networking
Technology
Partnerships

The case study findings have been integrated within the broader analysis of the evaluation findings, and are presented as italicized text to highlight chosen themes.

<< previous | next >>