Home : Reports and Publications : Audit & Evaluation : Program Evaluation Study: EDP and UEDI
Design and delivery
The chapter summarizes the major findings of our review regarding:
- The delivery components of the program;
- The broader delivery models and practices; and
- Respondent recommendations for improvement.
Review of the program components
The major components of the programs include promotion of the program, the provision and management of loans, pre-care and aftercare services, and reporting. Our major findings regarding these components of the programs are as follows:
Promotion and Awareness
1. On average, the delivery agencies rate themselves as somewhat effective in creating awareness of the programs.
The mechanisms that have been most commonly used to create awareness of the programs included media (advertising and encouraging news stories), networking with partners and other groups serving a similar target market, brochures, websites, and presentations to key groups. Some organizations noted that they do not aggressively promote the program because resources are simply not available to meet the demand that could be generated. In Manitoba, a 1-800 number has been established which has generated referrals to both CFDCs and the organization delivering UEDI in Winnipeg.
The delivery agencies were asked to rate their effectiveness of their efforts in creating awareness of the program on a scale of 1 to 5, where 1 is not at all effective, 3 is not at all effective and 5 is very effective. On average, delivery agencies rated their effectiveness as 3.3, while representatives of WD and the provincial associations rated the effectiveness of efforts to create awareness of the programs as 3.1.
Many of the organizations noted that, while some progress has been made, both awareness levels and the numbers of clients served remain relatively low. Approaches such as setting up a common 1-800 number, networking with other organizations that share a similar target market, and the placement of news stories were identified as specific strategies that have been effective. Some of the constraints to increasing awareness include the limited amounts of funding available for promotion, entrepreneurs with disabilities are a difficult group to reach through advertising, and the name by which the programming is known (as well as the names of the delivery agencies) varies by region which has made it difficult to implement joint promotions.
2. Most clients found out about the program through referrals or seeing ads or newspaper stories.
Of the 148 clients surveyed, 24% indicated that a government representative had referred them, 16% had seen an advertisement or news story, and 11% were referred by another organization or individual. Many of the respondents could not remember how they first became aware of the program.
3. Approximately one-half of the clients (48%) indicated that they were aware that Western Economic Diversification Canada funds the program, even before participating in the survey.
4. Awareness of the programs is relatively low amongst the stakeholder organizations, indicating that the programs have a relatively low profile even amongst other organizations that serve people with disabilities.
Only 61% of the representatives from stakeholder organizations indicated that they were at all familiar with either the EDP or the UEDI. On a scale of 1 to 5, where 1 is not at all aware, 3 is somewhat aware and 5 is very aware, the average response was 2.4 and only 10 of the 75 indicated that they are very familiar with the program.
Of the 46 stakeholders who indicated that they were aware of one or both programs, 14 of the stakeholders indicated that they have interacted directly with the program before, 11 have been directly involved in the delivery of the program or related services (e.g. as a member of an advisory committee, as a contractor to the program, or through the delivery of joint or related activities), 10 have referred clients to the program, 5 had heard about the program through other people, 4 attended an information session on the program, and 3 have read literature on the program.
Loan Component
5. The processes used in reviewing loan applications and administering the loan portfolio can vary significantly across delivery agencies.
CFDCs use the same process to manage the EDP loan portfolio as is used for their General Loan Fund. The loan applications are adjudicated by a loans committee, which consists of volunteers from the local business community. The criteria for EDP clients are generally similar to that for general loan clients although certain criteria may be applied less stringently.
The UEDI delivery agencies have each developed standard processes for reviewing loan applications and administering the loan portfolios. Some of the ways in which the systems vary include:
- The processes involved in preparing business plans and funding applications as well as in screening applications;
- The composition of the loan committee and the relationship between the committee and the Board (e.g. who formally approves the loans);
- Whether other organizations or contractors are involved in assisting the client develop a business plan and loan application;
- Whether other organizations (e.g. Edmonton Loan Fund or the Assiniboine Credit Union) are involved in administering the loan portfolio; and
- Whether the client has an opportunity to personally present his or her business plan to the loan committee.
6. The delivery agencies rated the loan component of the program as being somewhat successful.
When asked to rate how successful the loan component has been on a scale of 1 to 5, where 1 is not at all successful and 5 is very successful, the delivery agencies provided an average rating of 2.9.
While some agencies noted that the loan component has been effective in supporting the development of successful businesses, others noted that there have been relatively few loan clients and default rates have been high.
7. The delivery agencies estimated that over one-quarter of the loans that they have advanced under the program have been defaulted upon or written off.
On a simple average basis, the delivery agencies estimated that 29% of their loans have been defaulted upon or written off. By multiplying the default percentages by the value of the loans issued by each organization surveyed, we can develop a weighted average default rate. As indicated below, on average, the CFDCs reported a default or write-off rate of 25% while the UEDI agencies reported a default rate of 49%. Overall, the weighted average was 33%.
Some of the factors that have contributed to a high default rate are that businesses operated by entrepreneurs with disabilities tend to involve a higher level of business risk (e.g. the business may be undercapitalized, there tends to be less security available, the entrepreneurs may have less experience, the nature of the disability can add an additional level of risk) and, at least in the initial years, the CFDCs and credit unions tended to have limited experience in working with entrepreneurs with disabilities while other UEDI agencies tended to have limited experience in preparing and reviewing loan applications as well as in managing loan portfolios. Several representatives also noted that, unlike the general loan fund, EDP loan funds are conditionally repayable to WD and, as a result CFDCs may have focused more on making capital available to entrepreneurs with disabilities and placed less of an emphasis on ensuring that the loan pool survives intact.
8. The delivery agencies suggested that a reasonable loan default rate for programming such as this would be about 20%.
When asked their opinion regarding what would be a reasonable default rate for loans to entrepreneurs with disabilities, the responses ranged from 0% to 50%. On average, the respondents identified a reasonable target default rate to be 20%, which is lower than the simple average of 29% and the weighted average of 33% that they have experienced to date. The following chart compares the distribution in responses regarding actual default rates to those regarding reasonable default rates.
Some of the representatives noted that a default rate lower than 20% may indicate that the loan criteria are too stringent while rates above 20% could mean that the negative impacts of the program (e.g. on the financial situation of the clients as well as their self-confidence, independence and health) may begin to outweigh the positive impacts.
Precare and Aftercare
9. Many entrepreneurs with disabilities require more intensive precare and aftercare support than may be required of clients associated with the General Loan Fund.
Entrepreneurs with disabilities often face more significant barriers related to the establishment and development of their businesses that need to be addressed in order to improve the likelihood of success. For example, it was noted that potential entrepreneurs often have limited business and work experience, under-developed skills and self-confidence, and more limited access to information and other resources. In addition, the disability itself can serve as constraint in terms of imposing additional costs, physical limitations, and the perception of the marketplace others in the business community.
10. The availability of precare and especially aftercare services is very limited in most communities.
In addition to providing loans, the delivery agencies may provide other services to both loan clients and other clients. Examples of these services include business counseling, assistance in the preparation of business plans, pathfinding to other resources, and access to training. Generally speaking, these services represent pre-care services in that they are most commonly provided before a client receives a loan and/or starts-up a business. Most services supported through the programs focus on enabling clients to prepare or finalize business plans in advance of an anticipated application for loan funding.
Representatives associated with both the EDP and UEDI programs indicated that few resources have been available to provide aftercare services to clients once they have received a loan or started-up a businesses without receiving a loan. The delivery agencies generally stay in periodic contact (e.g. monthly) with loan clients and usually request that the clients provide them with monthly financial statements (although this is certainly not always done). A few agencies are able to stay in more regular contact with loan clients, particularly clients whose businesses may be struggling.
With their core funding, the UEDI delivery agencies have been able to provide more extensive pre-care and aftercare services to entrepreneurs with disabilities than have the CFDCs. A few CFDCs and UEDI delivery agencies have been successful in accessing provincial government funding or HRSDC funding to expand the pre-care and aftercare services beyond that which could be supported with the funding provided by WD.
Reporting
11. The CFDCs prepare quarterly reports for submission to WD.
These reports are designed to inform WD about the loan activities of the organizations and the other services provided. The CFDCs report their activities on standard forms, which collect data on the numbers of information services, technical and advisory services, and training and skills development services provided to clients with disabilities as well as data on EDP loans. This data is forwarded to headquarters in Edmonton, where the loan data is entered into a Lotus Notes database. However, the data on services provided to clients with disabilities has not been broken out from the data entered on services provided to all clients since CFDCs stopped receiving any direct funding for the delivery of services to entrepreneurs with disabilities (2001).
12. The data reported to WD by the UEDI delivery agencies is not standardized, but rather varies from organization to organization both in terms of what data is reported and the format in which it is reported.
For example, the data provided on loans may include a listing of the approval date or date of the first advance, the client and business name, the original principal, the value currently outstanding, and the status of the loans. Some agencies go so far as to provide a narrative description of each of the loan clients. In terms of other types of activities, examples of indicators on which one or more of the agencies report include the number of inquiries, contacts made, new clients, attendance at orientation or information sessions, the number of people participating in training, number of referrals made, the number of business plans developed or in development, the number of businesses started-up, and the number of jobs created or maintained.
13. Clients service standards have not been established with respect to the EDP or UEDI.
However, target clients have had some opportunity to provide feedback as well as input into the design and delivery of the program through mechanisms such as:
- A few advisory committees established at the provincial or local level which include representatives of the target groups; and
- Occasional surveys or focus groups with clients.
There was also an evaluation of the UEDI delivered by two of the agencies as well as an early evaluation of the EDP that provided an opportunity for clients to provide feedback.
Design of the program
Our major findings regarding the design of the program include:
1. The representatives of the delivery agencies were relatively evenly divided as to whether the programs are well designed given the perceived objectives.
Of the 44 delivery agencies surveyed, 20 indicated that the EDP or UEDI program is well designed, 21 indicated that it is not well designed given the objectives, and 3 did not express an opinion. Some of the core strengths of the program design that were identified include:
- The design of the program reflects the need to provide both capital and other forms of assistance to entrepreneurs with disabilities.
- The structure enables the program to be delivered locally through organizations that either have extensive experience in working with entrepreneurs or in working with clients with disabilities.
- The EDP was able to build on the systems developed to deliver services related to the general loan fund.
- Over time, the UEDI delivery agencies have been able develop and enhance their services and systems.
By far the most common concern expressed about the design of the program is that more precare and especially more aftercare assistance is required to improve the success rates of the businesses. Given that the impact of failure can be very significant on the health, well-being and financial resources of individuals, several representatives noted that the program has a moral obligation to better prepare entrepreneurs to meet the challenges that they will face. Other concerns were expressed about the size of the available loan pools (including the need to replenish some funds), the need for more structure with respect to the focus and objectives of the programs, and the impact that uncertainty regarding the future of the programs is having on program delivery.
2. Budget reductions and restructuring has lowered the combined cost of the programs but, in the opinion of some of the representatives surveyed, has not necessarily improved the cost-effectiveness.
Some of the strategies that have been used to encourage the programs to become more cost-efficient have included:
- Reducing the overall budgets for both the EDP and UEDI over time;
- Providing funding to the provincial associations rather than providing EDP operational funding directly to the CFDCs. In the initial years of the program, some CFDCs received some operational funding even though they were not very active in delivering the program.
- Working through existing organizations who already have delivery systems in place. Many of the CFDCs and UEDI delivery agencies noted that they, in effect, subsidize delivery of the programs (i.e. the total cost of delivering the program, including operating and overhead costs, is greater than the funds they receive).
However, a common theme in our interviews is that keeping costs low does not necessarily equate to cost–effectiveness. Several representatives noted that a lack of funding has greatly restricted the level of pre-care and aftercare services provided, resulting in higher loan losses that, in turn, result in higher costs for the delivery organization and for the clients. Other agencies noted that, in the absence of adequate funding, there is no incentive for them to actively promote the program. In fact, given the higher direct costs (more intensive precare and aftercare services are required) and indirect costs (e.g. higher loan losses) associated with serving the target group, there is actually a disincentive to marketing the program. This may help to explain why over 30% of the CFDCs (28 of 90) have not made any EDP loans over the past three fiscal years.
3. The CFDCs are generally not very supportive of the transition which occurred in 2001 when WD changed from providing EDP operating funding for individual CFDCs to providing funding through the provincial associations.
The provincial funding has been used to fund projects and special initiatives. Numerous CFDCs have taken advantage of the funding and, in each province, some the CFDC managers we surveyed identified some specific projects, initiatives, training or networking opportunities which they thought had been useful to them. The provincial associations were generally commended for their efforts in:
- Creating awareness of the funding;
- Working with the CFDCs in generating applications; and
- Creating committees to review and decide on funding.
In addition, it was recognized that there may be certain activities or initiatives which are best undertaken at the provincial or even the pan-western level. The major reservations or concerns expressed about the provincial model are that:
- In the opinion of many, the funding would be more effectively used at the local level;
- While some projects have been successful, others are considered to be a very low priority. Even some of the CFDCs that had received funding from the association indicated that there would have been better uses for the funding that they received. Several CFDCs specifically noted that they apply for funding because it is there, not necessarily because they support the provincial model; and
- The transition has resulted in some tensions between the CFDCs and the provincial associations.