The Eastern Ontario Development Fund (EODF) is a new initiative, launched in October 2004, to promote socioeconomic development in rural Eastern Ontario by building local capacity for development and creating the necessary conditions for increased business and employment opportunities. This program is an initial response to the Government's commitment in Budget 2004 to review the role played by Community Futures Development Corporations (CFDC) in regions without a regional development agency, such as Eastern Ontario, and examine the possibility of broadening and deepening their mandate.

This study is a formative evaluation of the EODF, which assesses the design, delivery and implementation of the program. The evaluation results are to provide evidence critical to FedNor program planning and design in light of the current March 31, 2006 sunset for the program.

For more details, please refer to Section 1.0 of this report.

Methodology

The evaluation addressed a series of research questions categorized according to the following issue categories:

relevance;
design and delivery;
success;
monitoring and accountability;
alternatives and cost-effectiveness; and,
lessons learned.

The evaluation used a range of methodologies to ensure that multiple lines of evidence were available for each issue. The methodologies were as follows:

document and database review;
27 key informant interviews with Industry Canada / FedNor management and staff, representatives of the CFDCs in Eastern Ontario and other stakeholders;
a telephone survey with 250 EODF funding recipients; and,
five case studies which provided an in-depth analysis of the implementation of a specific program element in a particular CFDC or of a particular project.

While there are some strengths and limitations to the methodology, overall, the approaches and sample sizes used for this study resulted in a strong and reliable formative evaluation.

For more details, please refer to Section 2.0 of this report.

Findings – Relevance

A grassroots approach was used to initially design the EODF programming. As such, it is not surprising that the evidence gathered during this evaluation reveals that the EODF programming is relevant to the needs of the community. This is evident in the documents and all interviewees agreed that there was a need for this type of programming and that the way in which the EODF was designed was relevant to this need.

The program is designed around five priorities: business development, access to capital, skills development, retention and attraction of youth, and technological enhancements. Given that the program was initially designed by the 15 Eastern Ontario CFDCs with FedNor involvement, it is not unexpected that the FedNor management and staff, CFDC representatives and stakeholders who were interviewed all agreed that the priorities were relevant. It was noted that there was a greater need for some of the priorities in certain communities and that the need could change over time. However, all five priorities are relevant, overall, and are general enough to be adaptable to the evolving economic development needs of communities across Eastern Ontario. This is evident in the early program data that shows that as of March 31, 2005, 48% of the applications received were for business development (this included three funding streams which explains the relatively high proportion of applications), 25% were for skills development, 15% for youth internships and 12% for access to capital. The recipients surveyed were also highly satisfied that the EODF was able to address their needs and, regardless of their own particular needs, that the five EODF priorities were extremely important (mean importance of 9.4 for business development – 10 meaning extremely important; 9.3 for access to capital; 9.2 for skills development; 8.9 for retention and attraction of youth; and, 8.8 for technological enhancement).

Documents show that proper care was taken to ensure that the EODF does not overlap or duplicate other government programs. The interviewees all agreed that the EODF complements existing programming, which was noted as being very limited in Eastern Ontario. In fact, only 16% of surveyed recipients indicated that they were aware of programs or services of the federal or provincial government which are comparable or similar to the EODF.

For more details, please refer to Section 3.1 of this report.

Findings – Design and Delivery

The EODF service delivery model involves a blend of CFDC and FedNor decision-making. Projects that are funded through secondary recipients (i.e., Youth, Skills Development, Business Development and Access to Capital) are delivered by the CFDCs at various funding levels. The exceptions are Community Capacity and Regional projects which are approved by FedNor / Industry Canada (IC), regardless of the funding level. The CFDCs are then responsible for the negotiation and ongoing monitoring of all local projects, although FedNor is closely involved in monitoring the larger local projects. If a project is regional, the application is reviewed by a committee comprised of representatives from each of the 15 CFDCs. Each CFDC gets one vote on whether the project should be recommended for funding. If the majority of CFDCs agree on a regional project, the application goes to FedNor for review and approval, regardless of the size of the project. If approved, one CFDC is assigned as the lead for the project; as such, this CFDC is responsible for the negotiation and ongoing monitoring of the project as well as for ensuring proper sharing of progress and results with the other CFDCs.

While there were some concerns noted with parts of the service delivery model, in general, all interviewees believed that, when weighing the pros and cons, the model was sound and appropriate. The comments made by interviewees were supported by those made in the survey. Overall, the surveyed recipients were very satisfied (mean of 8.9 our of 10) with the EODF.

While the EODF is a relatively new initiative, all sources (documents, interviews, survey) point to the fact that it was widely promoted and that it is well known. While there is considerable variation among the 15 CFDCs in the use of their website to promote the program, 13 of the 15 have information on the EODF on their website. In addition to the website, interviewees noted announcements, press releases, newspaper and radio advertisements and articles / interviews, meetings, mailings, word of mouth, and several other means to promote the program. Interviewees generally believed that, even though some people were not aware of the EODF, most of those who were potential recipients were well aware of the program. The surveyed recipients were also highly likely to indicate that they believed that the EODF was well known (62% responded very or fairly well known).

Interviewees generally believed that the selection process for local projects was effective. The actual process may differ from one CFDC to another, however, the process was deemed to work for the individual needs of the communities. However, several interviewees had concerns with the project selection process for regional projects. Concerns were raised because of the belief that some of the regional projects were not truly regional, that some could have been done with less EODF funding, and / or that some of the regional projects were not that good in terms of their potential impacts.

The project selection criteria (i.e., consistency with objectives, quality of supporting plans, project feasibility, incrementality, job creation and other lasting community economic benefits, leveraging, and track record of partners) were deemed to be clear, appropriate and applied, for the most part by those interviewed. Recipients were highly or fairly satisfied with the clarity of the eligibility criteria (8.4 out of 10), the fairness of the selection process (8.9 out of 10), their understanding of the decision-making process (7.7 out of 10), and the speed of decision-making (8.4 out of 10).

According to interviewees, the program design, promotional efforts, and selection process have resulted in projects that are relevant.

FedNor interviewees were, in general, satisfied that they had provided adequate support to the projects and that this support was important. CFDC representatives were also very satisfied with the support provided by FedNor. They noted that the support was appropriate and sufficient. Several commented on the outstanding job that FedNor had done with respect to this initiative, particularly in light of the limited time available for implementation and the limited FedNor human resources dedicated to the EODF.

For more details, please refer to Section 3.2 of this report.

Findings – Success

While the EODF is in its very early stages and many projects are not yet completed, there is already evidence that it is successful in achieving its intended outcomes. The case studies completed during the study were all indicative of the wide range of actual and potential impacts of the program and its projects. Additionally, the table below highlights the early results of the projects, based on the survey of recipients.

EODF Project Results
Result	Has occurred	Will occur	Will / may not occur / did not expect
Development of a new business	13%	12%	75%
Retention of an existing business	43%	8%	49%
Improved competitiveness	55%	20%	25%
Attraction of people to your organization or region	52%	19%	29%
Retention of people within your organization or region	58%	11%	31%
Development of people within your organization	65%	9%	26%
Development of business skills	58%	8%	34%
Development of trade skills	42%	6%	52%
Increased use of technology	44%	5%	51%
Development of an innovation	32%	10%	58%

What is particularly important about these results is that they are highly attributable to the EODF because the program is incremental. That is, 34% of all surveyed recipients indicated that they could not have undertaken the project and that this would have had a major negative impact; another 41% would have been negatively affected in a major way in that the project would have been delayed, would have taken more time, would have been of lower scope or quality, or for some other reason.

For more details, please refer to Section 3.3 of this report.

Findings – Monitoring and Accountability

Even though the EODF is in its early days, a lot of progress has been made in putting in place a performance monitoring and reporting system. What is currently in place includes:

project recipients are required to report to the CFDCs;
CFDCs compile the individual project results and provide reports to FedNor for each large or regional project and for each of their contribution agreements (one for each program priority); and,
FedNor inputs this information into a database for further analysis and reporting.

The document review and the interviews revealed that recipients and CFDCs have been fairly diligent in providing timely, complete and accurate report, to the best of their capability. However, the document review and interviews identified some challenges with reporting requirements. The principal challenges are:

the reporting requirements apply to all projects regardless of the size of EODF funding, the objectives of the projects, their intended impacts, etc. – therefore, the recipient of a very small project with limited albeit important objectives must report according to the same requirements as the recipients of larger projects with complex objectives;
the reporting requirements, while similar, do not match the program structure;
CFDCs roll up the results of several projects into one report for each of the contribution agreements they have with FedNor for this initiative; unfortunately, the report is structured for reporting with respect to only one project as opposed to several projects within one contribution agreement;
there is confusion with reporting on FedNor target groups in terms of what to report and why it needs to be reported (i.e., it is not necessarily important for some communities to target certain groups, yet they still need to report on the reach of those target groups); and,
the key indicators identified in the Results-based Management and Accountability Framework (RMAF) for the EODF are not all covered through the reporting requirements or do not directly meet the requirements.

For more details, please refer to Section 3.4 of this report.

Findings – Alternatives and Cost-Effectiveness

There were three service delivery approaches presented by the interviewees. The first is the status quo (blend of CFDC and FedNor decision-making), the second was to give more (or full) control to the CFDCs for making decisions on projects in their communities, and the third was to give more control to FedNor so that the program is administered the same as the Northern Ontario Development Fund, where FedNor deals with more than just CFDCs as primary recipients. In general, interviewees agreed that the current approach was the best alternative as demonstrated by the results of the program, to date.

In terms of cost-effectiveness, the evidence shows that the program is cost-effective in the following key ways:

The FedNor operational component of this initiative represents less than 4% of the total budget for this initiative. The CFDC operational / administrative component represents approximately 6% or 7% of the budget. The total operational is therefore approximately 10% of the total budget.
A lot of the work done at the CFDC level is done by the volunteer Boards of Directors. Across the 15 CFDCs, this could represent as much as 180 person-days of work, in total, for each application deadline.
The program funding encouraged the financial contributions of other partners on the project. As of March 31, 2005, for close to $9 million in approved EODF funding, the program leveraged an additional $28.6 million. The program therefore funded less than 25% of the total project costs, yet was highly incremental to the organizations' ability to undertake the projects.
As of March 31, 2005, 643 projects were approved for close to $9 million in approved funds. This represents average projects of less than $14,000 in EODF funding. As reported previously, these projects are already reporting significant economic development outcomes and impacts (see Success summary).

For more details, please refer to Section 3.5 of this report.

Findings – Lessons Learned

The key lessons learned are based on evidence presented throughout this report as well as on suggestions made by interviewees and recipients. These are summarized in the conclusions and recommendations table in the next section.

For more details, please refer to Section 3.6 of this report.

Conclusions and Recommendations

Conclusions

Recommendations

Relevance

Conclusion 1

The EODF programming is relevant to the economic development needs of the communities in Eastern Ontario. To a large extent, this is due to the fact that the program was designed by the 15 Eastern Ontario CFDCs based on their familiarity with the socioeconomic challenges faced by the communities they serve.

Recommendation 1

The EODF should be renewed.

Note: the balance of the recommendations are based on the assumption that the EODF will be renewed.

Conclusion 2

The five program priorities (business development, access to capital, skills development, retention and attraction of youth, and technological enhancements) are relevant, again due to the fact that the program was designed by the 15 CFDCs. While the specific need for funds across these priorities may have differed across CFDCs, these are legitimate and critical priorities to help Eastern Ontario address its economic development gaps. Additionally, while none of the projects were classified under the technological enhancement priority, this is because of the way some projects were classified and not because it is not an important priority.

Recommendation 2

The five program priorities should continue to form the basis of EODF programming. These encompass current priority needs and are broad enough to address evolving economic development needs.

Conclusion 3

The EODF does not duplicate or overlap with other government programming. The EODF is unique in Eastern Ontario.

See Recommendation 1

Design and Delivery

Conclusion 4

The service delivery model for the EODF is appropriate given the program objectives. While the CFDCs need to ensure that they involve other important economic development groups, the network of CFDCs is an excellent delivery mechanism. This is due to the fact that CFDCs have an existing relationship with FedNor, are recognized players in their communities at the economic development table, and provide complementary services through their existing programming. Additionally, CFDCs are unique in their regional rather than local perspectives.

Recommendation 3

The EODF should continue to use the existing delivery model which involves FedNor contribution agreements with each of the 15 CFDCs who, in turn, provide funding to secondary recipients.

Conclusion 5

Even though the program was implemented within a very short time frame and is in its infancy, the CFDCs used a wide range of effective means to promote it to ensure that it is well known. While there is still an ongoing need to continue to promote the initiative and make more organizations aware of its availability, the CFDCs received more applications than they could approve within their budgets.

Recommendation 4

The CFDCs should continue to use a multitude of promotional tools to ensure that their communities are appropriately aware of the EODF. However, promotional strategies should consider the limited availability of EODF funds.

Recommendation 5

While the promotional efforts of the CFDCs have been fruitful, a process should be implemented to ensure that promotional materials are shared and to thus avoid duplication of efforts and build on best practices.

Conclusion 6

The project selection process for small local initiatives is effective. While the exact process differs from one CFDC to the next, each was able to receive and review a large number of applications within a very short time frame and approve projects that were successful.

Recommendation 6

The current project selection process for small local initiatives does not need to change. However, since CFDCs have implemented different processes for the review and selection of projects, the different processes should be shared among CFDCs to help in the sharing of best practices and lessons learned.

Conclusion 7

The project selection process for larger local initiatives where FedNor needed to be involved in the approval process is also effective. The initial review and recommendations of the CFDCs was effective in that few recommended projects were not approved by FedNor. Additionally, FedNor was very effective in reviewing and approving these applications.

See Recommendation 6, but for larger local initiatives.

Conclusion 8

The project selection process for regional projects is not as effective as it could be. The definition of regional projects is unclear. However, the process involving all CFDCs in the review and recommendation of projects is appropriate, as are the role of FedNor in approving the applications and the consultations with other regional economic development stakeholders.

Recommendation 7

The project selection process for regional projects needs to be clearly defined and agreed upon by FedNor and the CFDCs. This should include a clear definition of the scope of regional projects. It could also include a series of outcomes that are regional in nature. The improved project selection process should, however, continue to involve consultations with other regional economic development stakeholders with respect to needs for regional EODF projects. In refining the project selection process, FedNor and the CFDCs will need to clarify the following:

definition and scope of regional projects.
how regional projects should be approved.
how project leads should be identified (one CFDC, several CFDCs, others).
how regional projects should be monitored;
etc.

In refining the project selection process, care should be taken to ensure that the factors that have made this initiative successful (such as its grassroots aspect) are preserved or built upon, and that other recommendations are taken into account.

Conclusion 9

The project selection criteria are clear, appropriate and applied. The relevance of the projects selected (as demonstrated by the appropriateness of the early results) are indicative of the fact that the criteria are applied and appropriate given the program objectives.

Recommendation 8

The current project selection criteria should not change.

Conclusion 10

The role of FedNor in support of the projects is important given its experience with other programming and with similar types of projects in Northern Ontario. This support is appropriate and sufficient.

Recommendation 9

Notwithstanding other recommendations, FedNor should continue to provide the same level and type of support to CFDCs and to ensure that adequate resources are allocated to provide this support.

Success

Conclusion 11

The EODF has been successful, even in its infancy, in achieving its intended immediate and intermediate outcomes. Even though many projects were still ongoing at the time of the evaluation, all projects have resulted in at least one of the program's intended outcomes.

See Recommendations 1 to 9

Conclusion 12

The EODF is helping organizations undertake projects that they could not otherwise undertake or would have to complete differently (lower quality, smaller scope, delayed, etc.). The program is therefore highly incremental. As such, the success of the projects is highly attributable to the program.

See Recommendations 1 to 9

Monitoring and Accountability

Conclusion 13

FedNor has made good progress in implementing the performance measurement strategy outlined in its Results-based Management and Accountability Framework (RMAF). In addition, the CFDCs have been very diligent in respecting their commitments regarding progress reporting. As such, they have taken the steps required to ensure that they have the information they need to meet these commitments. However, some of the reporting requirements are not clear. Additionally, it is not evident that all RMAF commitments are met through the existing reporting structure. Performance measurement is still a work-in-progress.

Recommendation 10

FedNor should continue to improve its performance measurement system with respect to the EODF. This should include updating the RMAF in consultation with the 15 CFDCs to ensure that they are in agreement with the commitments outlined in this document. It should also include revising the reporting structure to ensure that it is consistent with the program's structure as well as the RMAF indicators. The revised reporting requirements should include clear definitions and instructions.

Recommendation 11

Once the performance reporting requirements are finalized, FedNor should give due consideration to providing CFDCs with more effective tools for reporting. This could include the grants module of The Exceptional Assistant (TEA).

Alternatives and Cost-Effectiveness

Conclusion 14

There are no alternative service delivery approaches for the EODF that would produce the same results at a lower cost. Currently, the administrative portion of this initiative is reasonable, the CFDCs rely on the large contribution of its volunteer Boards of Directors, a significant proportion of the projects costs are leveraged through other sources, and the projects are contributing to the achievement of the intended outcomes. The program is therefore cost-effective.

See Recommendations 1 and 3 and:

Recommendation 12

FedNor and the CFDCs should monitor the appropriateness of the operational budgets for the EODF. It is important to ensure that, with the continuation of the program, the burden on FedNor and CFDC staff is not unreasonable. It is also important to ensure that the program requirements do not conflict with other FedNor and CFDC staff priorities.

Recommendation 13

CFDCs should continue to require that, where possible, additional sources of funding be sought by the secondary recipients.

Lessons Learned

Conclusion 15

The relationships among all players involved in this initiative was key to the success of the EODF. These included the relationship: 1) across the 15 CFDCs who worked extremely well in reaching consensus on the design and delivery of this initiative; 2) between the CFDCs and FedNor where the pre-existing relationship helped to contribute to the smooth implementation of the initiative and the commitment of both parties to the initiative helped make it a success; 3) within the CFDCs among staff and with the Boards where staff demonstrated a high commitment to make this initiative work and the Board members worked with staff to ensure that the project applications were reviewed and approved in a timely fashion; and 4) between the CFDCs and the community where the fact that the CFDC was a recognized credible economic development organization which knew the community and vice versa made it easier to roll out the program within a very short time frame.

See all previous recommendations

Conclusion 16

Flexibility was also key to the success of this initiative, yet ongoing and increased flexibility may help ensure the continued success of the initiative. The program was set up by the CFDCs to provide each CFDC the same amount for each of the program elements. However, enough flexibility was incorporated into the guidelines so that the CFDCs were able to redistribute some of their budgets across components in order to better meet the needs of their communities. While some CFDCs would have liked more flexibility right from the start, the combination of guidelines and flexibility within those guidelines helped ensure that the intended program results were achieved while ensuring that community needs were considered. Nevertheless, increased future flexibility, within set parameters, may contribute to a more effective decision-making process and ensure that the needs of the communities are better met. Additionally, more flexibility regarding bureaucratic requirements for very small projects could help make the program more cost-effective.

Recommendation 14

FedNor should consider the feasibility of putting in place a process to have only one contribution agreement with each of the 15 CFDCs. This agreement would include all elements of the program for local initiatives (i.e., excluding regional projects and community capacity building, but including business planning, access to capital, skills development, retention and attraction of youth, and technological enhancements) with guidelines as to the minimum and maximum proportion of funds to be allocated to each of these elements. This would provide the flexibility required at the local level while ensuring that all priorities are addressed.

Recommendation 15

FedNor should consider implementing a risk-based approach for its monitoring and reporting strategy. This should be integrated in the considerations for Recommendation 10.

Conclusion 17

In order to ensure good ongoing performance reports, it is key to make sure that the performance measurement system is reflective of the program and that performance reporting requirements are clearly defined. The RMAF for the EODF was developed by FedNor. Because of extremely tight deadlines, CFDCs were not directly involved in the development of this RMAF. While FedNor did attempt to consult with some CFDC representatives, these consultations were limited and most CFDCs were not familiar with the document. This presented challenges in that CFDCs did not fully understand (or agree with) the program logic model and performance indicators. As such, they did not understand some of the reporting requirements. Additionally, some of the reporting requirements were not clear and were therefore interpreted differently from one CFDC to the next. Finally, the key indicators identified in the RMAF were not fully integrated into the performance reporting requirements.

See Recommendation 10

Final Report (PDF - 385 KB - 68 pages)
Management Response (PDF - 18 KB - 5 pages)

Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.