Summative Evaluation of the Northern Ontario Economic Development Fund (NODF)

Conclusions

Recommendations

Relevance

Conclusion 1

The economy of Northern Ontario continues to struggle with economic development issues. NODF strategic priorities are relevant and adapted to these economic development issues. Additionally, the projects are relevant in their objectives and meet the criteria established by FedNor.

The key aspects of NODF which make the program particularly relevant are:

• its flexibility;
• its broad criteria and broad range of priorities;
• the range of project size it funds;
• its broad reach;
• the fact that it is customized to needs;
• its partnership approach;
• its geographic delivery; and,
• its ever evolving strategic outlook.

Recommendation 1

FedNor should continue to adapt the NODF to the changing economic development needs of Northern Ontario. In particular, it should continue to develop strategic approaches that address pan-Northern and / or sectoral needs.

Staff Support

Conclusion 2

The role of FedNor staff, in particular of program officers, is critical to the successful planning, development and implementation of projects. There are several key aspects with respect to staff support that contribute to this success:

•  the extensive advice and counselling provided by the officers in the development of proposals;
• the due diligence exercised by program officers in ensuring, not only that NODF funding does not duplicate or overlap other sources of funding, but also that the client is aware and has access to all other potential sources of funding;
• the local presence of the program officers;
• the on-going relationship of staff with some clients; and,
• the ability to provide services in the official language of choice.

However, there are concerns with some aspects of the support provided in terms of project implementation, monitoring and follow-up. These concerns include:

• some inconsistency among program officers in applying the criteria as well as in the level and type of support provided during the project; and,
• workload demands that do not provide program officers with the time to provide an adequate level of monitoring and follow-up that they (and others) believe is required.

Recommendation 2

FedNor should ensure that program officers are aware of the extent of their roles and responsibilities, and that they recognize the areas where flexibility is required versus the areas where consistency is needed. This could be done through more frequent communications via various policy bulletins, training, meetings and other forms of communications amongst program officers as well as between officers and management.

Recommendation 3

FedNor should assess the workload distribution of program officers in terms of the time they spend on NODF project planning activities, project implementation and monitoring activities, and project follow-up activities. This assessment should also examine the time spent on other non-NODF activities and tasks. Based on the result of this assessment, FedNor should make the appropriate adjustments, as required, to ensure an appropriate balance in the time spent on these tasks.

Conclusion 3

All priorities outlined in the Service Improvement Initiative plan have been successfully addressed except with respect to the turnaround time on payments. However, the turnaround time on payments is no longer assessed as being a service improvement priority, from the clients' perspective. This is due to the fact that clients are slightly more satisfied with FedNor's performance on this and they do not believe this to be as important a service feature as it was in earlier years.

Recommendation 4

FedNor should continue to monitor various aspects of its services and to adapt its service improvement priorities as required. The recommendations included in the 2004/05 Service Improvement Initiative report are still relevant in that, not much improvement can be achieved in terms of client satisfaction. Nevertheless, client expectations could be better managed through the establishment of published service standards.

Success

Conclusion 4

A number of NODF projects have contributed to the development and retention of new and existing businesses. Evidence suggests that NODF projects have resulted in the development of new businesses, jobs and growth in existing businesses through retained and new employees as well as increased revenues.  In addition, Northern Ontario CFDCs have indirectly contributed to NODF success in this regard through the impacts of the NODF capitalization assistance. The reported impacts from this assistance include close to 500 new, more than 500 maintained and close to 350 expanded business. Northern CFDC support, resulting at least partially from NODF assistance, have also contributed to the creation of more than 2,100 job and the maintenance of almost 4,300 jobs. The program is therefore contributing to the development and retention of new and existing businesses, both directly and indirectly.

Recommendation 5

FedNor should continue to address the economic development needs of Northern Ontario through its broad range of strategic priorities as these are contributing to the wide ranging impacts of the NODF.

Conclusion 5

The NODF's contribution to the competitiveness of Northern Ontario firms is, in most cases, indirect through the Community Futures top-up and the BDC investment fund. Nevertheless, the NODF also contributes directly to this objective through its trade networks and missions as well as through projects involving the development of new or improved products, services or technologies.

Conclusion 6

The NODF has been successful in attracting, retaining and developing human capital in Northern Ontario. With respect to special target groups, the program has made a significant impact on youth, women, Aboriginal people and Francophones even though the projects did not necessarily target these groups.

Conclusion 7

The NODF has made a significant contribution to economic development in Northern Ontario through community level initiatives as well as, to some extent, sectoral initiatives. FedNor's ability to encourage and build partnerships is a major contributor to the program's success in this area.

Recommendation 6

FedNor should keep the requirement for high involvement of other partners and communities in the NODF funded projects as these are also important to the success of the projects and program.

Recommendation 7

FedNor should also enhance its support of sectoral initiatives as these are important to the economic development of Northern Ontario communities.

Conclusion 8

Based on the evidence available for this evaluation, it is difficult to conclude on the extent to which the NODF has helped develop business and trade skills. However, the survey results and case studies indicate that the NODF has made a contribution in this area.

Conclusion 9

Based on the activities in support of developing external markets, the NODF has made progress in this regard. Progress has been made as a result of project-specific activities such as trade missions. Progress has also been made as a result of support provided by the trade advisors as well as through other non project-specific activities and support.

Conclusion 10

As a result of the projects funded under the Telecommunications and ICT as well as Innovation and Technology strategic priorities, the NODF has contributed to significant investments in this area. These investments have reached a large proportion of communities and businesses in Northern Ontario resulting in increased use of technology and the development of innovations. This has, in turn, contributed to socio-economic benefits in recipient organizations and communities.

Conclusion 11

The NODF has been essential in enabling a large number of organizations to undertake the funded projects. As such, the project impacts are highly attributable to the program. In addition, the design features have contributed to high leveraging of funds from other sources. Over the period of the evaluation, $122 million of NODF funding has resulted in a total of $400 million in investments in the projects. FedNor is therefore recognized as a catalyst for economic development.

Recommendation 8

FedNor should continue its practice of appropriately balancing its requirements that NODF funding be incremental to the recipient organization's ability to undertake the projects as well as requiring an appropriate level of leveraging on projects.

Monitoring and Accountability

Conclusion 12

FedNor has improved its capacity in terms of planning for results. For example, the recent Results-based Management and Accountability Framework (RMAF) is providing management with a better tool for measuring performance. The planning process is more closely linked to the NODF's performance management requirements. FedNor has also invested more resources into this, as evidenced by staff which are dedicated to performance monitoring. Additionally, FedNor has undertaken special studies to review specific aspects of the program (for example, the Youth Internship Program and post evaluations of trade missions). With respect to projects, officers are meeting their requirements in terms of assessing project risk and completing the project outcomes scoresheets. Additionally, with respect to the program as a whole, there is evidence of a lot of analysis and reporting of the data that is currently available.

However, the RMAF does not fully reflect the present design and delivery of the NODF. The existing tools and systems have not been updated to reflect the performance measurement requirements outlined in the recent RMAF. Additionally, it is unclear if the information collected from the project outcomes scoresheets is valid and reliable. It is also unclear if program officers clearly understand how to complete the scoresheets. The departmental systems for capturing and analyzing project information remains poorly aligned with NODF requirements. Finally, the departmental system is not aligned with the project outcomes scoresheets. Therefore, it is difficult to clearly determine the extent to which projects are achieving their intended outcomes.

Recommendation 9

FedNor should further refine the NODF RMAF to more accurately reflect the program priorities. The performance reporting systems and tools should also be refined to meet the requirements outlined in the RMAF. In brief, the NODF RMAF would benefit from the following enhancements:

 1.  A closer integration of the logic model with the expected results, which are priority driven.

 2.  An integrated performance measurement and evaluation strategy which:

• is directly linked to the logic model;
• provides assurance that every aspect of the logic model is measured at some point in time (not necessarily ongoing);
• recognizes the variable level of risk associated with the projects and clients;
• provides a more direct link between the evaluation success issues and the program's logic model; and,
• identifies the tools that are in place or needed to meet the performance measurement and evaluation strategy.

The revised RMAF should feed directly into the revisions of existing tools and systems as well as the development of new tools and systems as required. For example, the outcomes captured on NODF projects in the departmental system should match the outcomes in the RMAF. This is not presently the case. For example, the database captures community capacity, connectedness, e-commerce, export, knowledge-based, tourism and trade outcomes. These are currently not directly linked with neither the program priorities nor the outcomes identified in the program logic model. Similarly, what is measured in the project outcomes scoresheet is not reflective of the logic model outcomes nor of the indicators in the performance measurement plan.

Specific instructions on how to specifically complete the project outcomes scoresheets as well as on what is required from the program officer to ensure that the ratings in the scoresheet are reliable must be developed and a means for ensuring data quality and integrity needs to be implemented.

Conclusion 13

The Industry Canada departmental systems (i.e. the Grants and Contribution Reporting System – GCRS which draws upon the information in the Contribution Management Information System – CMIS) to capture information on projects is inadequate to meet the monitoring and accountability requirements of FedNor with respect to the NODF. It currently does not provide the flexibility required to capture the intended and actual results of projects as outlined in the program's RMAF. Additionally, since RMAFs are intended to be “living” documents, program results can change over time, particularly for a program such as the NODF. The Industry Canada departmental system is not currently flexible enough to provide FedNor with the ability to adjust the fields in the database as the program evolves to better reflect the changing needs of Northern Ontario communities.

Recommendation 10

FedNor should consult with Audit and Evaluation Branch (AEB) and the Information Management Branch (IMB) which is responsible for GCRS and CMIS to ensure that the departmental systems provide FedNor management with the flexibility it requires to appropriately capture the performance information it needs to better meet its monitoring and accountability requirements. If the required flexibility cannot be incorporated into the existing / new system, then FedNor should discuss other options with AEB and IMB regarding linking the FedNor system to ensure that all monitoring and accountability information can be captured and linked without requiring duplication.

Alternatives, Cost-effectiveness and Lessons Learned

Conclusion 14

There are currently no alternative service delivery approaches for the NODF that would produce the same results at a lower cost. Currently, a significant proportion of the project costs are leveraged through other sources. FedNor collaborates extensively with other organizations to ensure that NODF funding complements rather than duplicates the contributions of these other sources of funding. The due diligence exercised by the program officers and program management to ensure that NODF cost-effective investments in Northern Ontario (i.e., the right level of funding and the right leveraging for strategically needed projects) is very comprehensive and effective. Additionally, a large proportion of clients would not proceed with their projects if lower levels of funding were provided by FedNor. It is therefore evident that FedNor is managing NODF costs appropriately. The projects are contributing to the achievement of their intended outcomes. It is therefore evident that the NODF is effective in achieving success. Overall, the program is therefore managed in a cost-effective fashion.

See recommendations 5 to 8 under Success.

Conclusion 15

The critical success factors of the NODF are its flexibility, its staff, the partnerships it helps develop and its local presence.

See recommendations 1 to 8 under Relevance, Staff Support and Success.

Conclusion 16

FedNor's strategic focus is key to its continued relevance. This includes the need for specific strategies to address sectoral needs (including natural resources). Additionally, a means of ensuring that clients are aware of the NODF's strategies would also be beneficial.

See recommendations 1 and 7 under Relevance and Success.

Recommendation 11

The new group responsible for gathering market intelligence for FedNor and helping to develop strategies in light of this market intelligence should take due consideration of this lesson learned.

Conclusion 17

Ongoing staff development is critical to the NODF's continued success. This includes training as required, means of ensuring that best practices (and lessons learned) are shared, clear roles and responsibilities. This will lead to more consistency in dealing with clients.

See recommendation 2 under Staff Support.

Recommendation 12

FedNor should continue to have regular organization-wide or strategically driven events aimed at sharing best practices and lessons learned.

Conclusion 18

The Northern Ontario Community Futures Development Corporations believe they should be more involved in the delivery of some of the NODF programming, in particular, in the trade missions that affect their clients and, possibly in other aspects of projects.

Recommendation 13

While there are currently no alternative service delivery approaches for the NODF, FedNor should further explore the possibilities of involving 3^rd parties more extensively in the delivery of the NODF to ensure that it continues to maximize its cost-efficiency.

Recommendation 14

FedNor should ensure that, to the extent feasible, all Northern Ontario CFDCs are invited to participate in its NODF trade missions, particularly if clients in their delivery region are participating.

Conclusion 19

The Youth Internship Program may benefit from increased flexibility, such as: permitting the overlap of interns where appropriate; longer term internships; as well as, the possibility of just an internship program rather than a youth internship program.

Recommendation 15

Recognizing that the Youth Internship Program has evolved extensively over time, as a result of ongoing monitoring of the effectiveness of this program, FedNor should further explore the implications of expanding the NODF Youth Internship Program.

Conclusion 20

FedNor has been committed to providing services in both official languages. There is also evidence of significant activity by FedNor in consulting with official language minority communities (OLMC) to ensure that their needs are integrated into NODF programming. However, based on stakeholder perceptions as well as some of the program data, FedNor needs to continue to evolve to ensure that it meets the unique needs of OLMCs.

Recommendation 16

FedNor should continue to work towards integrating the needs of the OLMCs into its NODF programming.