3.0 Evaluation Findings (cont'd)

3.4 Evaluation Issue – Reporting and Cost-Effectiveness

Evaluation Issue: Reporting and Cost-Effectiveness
This issues covers:

Are the WEI organizations reports to WD relevant and do they allow WD to monitor the performance of WD’s investment in WEI organizations?
Do the WEI organizations have systems to measure their performance?
Are WEI organizations cost effective? Are the results of WEI organizations significant considering WD’s investment?
Are there alternative models to be considered to improve service delivery and impacts of WEI organizations?

Findings

In our examination of reporting and cost effectiveness, the evaluation found that:

There are significant shortcomings in the data available from WD that has been provided through the WEI organization quarterly reports. The data is incorrect and unreliable. It is not utilized on an ongoing basis to track WEI organization performance.
WEI organizations would like to know the value of the information that they are asked to collect and how the information is actually utilized;
Some of the WEI organizations indicated they would like to be able to collect results-based information (e.g. what has been the impact of their services/loans) but need the resources and appropriate measures to do so;
The WEI organizations were considered to be cost-effective and produced significant results in relation to the WD investment;
Alternatives for improving service delivery and impacts would focus on the further development of partnership strategies with WCBSN partners as well as other organizations to bring greater breadth to the provision of services; and
Interviews with WD indicated a strong focus on partnership development to enhance WEI program delivery and increase effectiveness. Current reporting templates do not capture essential information about the types and important aspects of the WEI organizations existing partnerships with community, national and international organizations.

Reporting

The evaluation found that the data provided by WD from the WEI organization files is not reliable or consistent, even though the data is reported to have been confirmed by the WEI organizations themselves (see notes on Program Performance figures page 23). This points to a significant requirement for both WD and WEI to ensure that the numbers collected are correct. Due to questions about the reported figures, it is difficult for researchers or evaluators to analyze or compare data with confidence. It also appears from the numbers that data collection definitions are variously understood, resulting in figures being placed in different categories depending on the WEI organization understanding. Data collection categories and definitions should be clearly identified by WD.

It is also apparent from interviews with WD personnel responsible for monitoring the WEI organizations that the data is not utilized in any systematic way to provide file managers with a ready analysis of significant information concerning the WEI organization performance. For example, information collected from WEI key informant interviews indicates that organization specific trend analysis concerning clients is occurring, but there appears to be no formal system through which to report this information to WD.

In line with the above findings, interviews with Board Members and staff of the WEI organizations indicated that they would like to know the value of the information they are asked to collect, and the ways in which the information is actually used. They receive very little feedback on the information they provide and so question whether it is useful.

As indicated previously in the report, interviews with some of the WEI organization representatives indicated that they would like to be in a position to better track the impact of their work – not only in the area of business outcomes for loan clients, but also in the area of the client supports they provide on an ongoing basis. To track these impacts requires additional human and financial resources, but may provide the robust analysis required to further explore the success of the WE Initiative. Measuring impacts would also provide WEI organization management with further information regarding their loan approval and management process.

Reporting on Partnerships

The WEI organizations are also engaged in a multitude of partnerships with a wide range of partners. The evaluation was provided with anecdotal information concerning these partnerships, but at present there is no formal collection of information concerning the range or the value of these. Interviews with WEI organization staff and other organizations indicated that the partnerships result in leveraging both funds and service delivery support and collaboration. There is also no formal statement of WD expectations surrounding partnership development and the resource allocations that may be required to maintain these partnerships. These issues should be addressed in future Contribution Agreements if partnership development is to be measured.

Performance Measurement

The first series of Contribution Agreements allocated specific amounts for annual studies. The WEI organizations used some of these resources to measure their performance. The present Contribution Agreement does not have this provision. Other funds have been used by some of the WEI organizations to conduct studies on client satisfaction, client needs and new target group needs. It was reported that these studies had been useful to the organization, particularly in providing specific information for future programming purposes. For example, WES contracted consultants to conduct a needs analysis for online learning for rural Saskatchewan businesswomen³². Another example was given by WEC: it conducted a brief assessment of immigrant women’s knowledge of the organization and the needs of this group. In some cases these reports have been shared among the four provincial associations, in some cases these studies have been used for internal purposes as well as for sharing with other organizations, including other WEIs.

Cost Effectiveness

The evaluation methodology was not intended to conduct a formal cost/benefit analysis of the WEI organizations, but rather was designed to rely on primarily qualitative data from a wide range of knowledgeable informants. The evaluation found that stakeholders consider the WEI organizations to be cost-effective, in that they were using their limited resources responsibly and continuing to focus on their mandated activities. The results of the WEI organizations efforts were considered significant by most stakeholders, recognizing the limited support options available for women entrepreneurs in the retail and services sectors.

The loan loss ratio figures into WEI organization cost effectiveness, in that it would be considered not cost effective to continue to incur high losses on an annual basis. Because appropriate loan loss ratio levels are not defined at the present time within the WE Initiative, these cannot be factored in to a determination of present cost effectiveness. As discussed previously in this report, the issue of loan loss ratios should be discussed with WE Initiative stakeholders and guidelines/policies put in place to allow sufficient levels of reporting and management of these important elements.

Alternatives

The question of alternatives in evaluation terminology is whether the same services could be more effectively and efficiently delivered in a cost effective manner by another mechanism. The evaluation determined that the WEI organizations due to their gender specific expertise and focus are the appropriate mechanism. At the same time, to continue to increase their effectiveness and efficiency, the WEI organizations need to continue to build partnerships at all levels (community, provincial, regional and national) to increase partners’ capacity to serve women entrepreneurs. As identified in Section 3.2, this will include strategies to fill client service gaps.

Summary

Reporting designed to identify results and manage to increase impacts is increasingly the focus of government funded programs. The current methods of collecting information by WD appear to have resulted in unreliable and confusing information. Due to the WD focus on partnership development, a reliable means of reporting on this aspect of WEI activity should be found and incorporated into future agreements with WD. The WEI organizations have expressed an interest in collecting data that is results focussed and useful to effective management of their organizations.

Stakeholders consider the WEI organizations to be cost effective, using their limited resources to maximum benefit. Adverse loan loss ratios could have a negative impact on this perception, but at the present time loan loss ratio levels considered to be appropriate by WD have not been defined. The future expansion of partnerships will support the organizations to address service gaps and meet client needs.

³² March 31, 2002. Conducted by Paschall & Associates, Inc.

<< previous | next >>