The Federal Economic Development Initiative in Northern Ontario (FedNor) is intended to promote economic growth, diversification, job creation and sustainable, self-reliant communities in Northern Ontario. In 1997, an evaluation framework for FedNor was prepared. As a result of changes to FedNor's Program Framework, new Treasury Board Policies and Guidelines, the findings from an audit of FedNor, and FedNor's Management and Accountability Framework, a revised evaluation framework for FedNor was prepared in the late Spring of 2001. In addition, a Results-based Management and Accountability Framework (RMAF) was developed and submitted to Treasury Board in the early Fall of 2001. The RMAF, which was based on the evaluation framework, formed the basis for the FedNor Management Committee (FMC) to approve the conduct of a formative evaluation of the FedNor Program. The evaluation study focused on all elements of FedNor, responding to evaluation issues and research questions of interest to senior management. (See Section 1 of the Final Report for more details.)

Methodology

The evaluation addressed a series of issues categorized according to the Treasury Board Secretariat evaluation guidelines of:

Rationale — Is FedNor meeting the needs of the Northern Ontario communities in improving key conditions and capacities required to support economic growth, diversification, job creation, sustainability, and self-reliance?

Success — To what extent is FedNor meeting its intended objectives without causing unintended negative impacts and effects on its clients and Northern Ontario's economy in general?

Cost-effectiveness — Is FedNor well designed and delivered, or are there significant modifications which need to be made to increase its efficiency and effectiveness?

These issues were addressed using sound evaluation practices therefore ensuring multiple lines of evidence for each issue.

The evaluation involved the use of existing information and included the review of documents and analysis of FedNor databases. In addition, original data was collected through the following approaches:

Two stakeholder focus groups were conducted in Sudbury and Thunder Bay. This included representatives from other communities, either in person or via teleconference. Stakeholders were defined broadly as a group or community partner who have an interest in FedNor and / or in the evaluation.

A total of 20 FedNor staff members were interviewed. Interviewees included both managers and officers, and were chosen to represent a variety of perspectives.

A telephone survey of 164 clients was conducted. For the purpose of the survey, clients were defined as organizations that had received FedNor funding, excluding CFDCs.

Five case studies were carried out on projects selected to represent as wide a range of characteristics as possible.

A total of ten in-depth telephone interviews were conducted with executive directors, managers, and coordinators chosen from the CFDCs in Northern Ontario.

In-depth telephone interviews were also conducted with ten industry experts. For the purpose of this evaluation, experts were defined broadly as individuals who have an expertise in northern economies and / or the sectors of interest to FedNor.

A benchmark study was also conducted. This involved an in-depth review of FedNor compared to the Atlantic Canada Opportunities Agency (ACOA), Canada Economic Development for Quebec (CED-Q), and Western Economic Diversification (WD).

(See Section 2.0 of the Final Report for more details on the issues and the evaluation methodology.)

Findings

The findings are summarized below according to evaluation issue.

Findings - Rationale

Issue 1.1 — What are the needs of communities and targeted groups which are pertinent to the FedNor mandate? Is FedNor meeting these needs through its existing programs and services?

Interviews, case studies, the focus groups and the benchmark study all contribute to demonstrating that, as one of many players, FedNor's programs and services are well aligned to meet the needs of communities and targeted groups for assistance in economic and infrastructure development. These needs are linked to: over-reliance on natural resources and the need for economic diversification; poor access to capital; lack of modern technical infrastructure; insufficient economic opportunities for youth, women, Aboriginal people and Francophones; low level of innovation; and the inability of small communities and rural Northern Ontario to undertake economic development and renewal without outside technical and financial assistance. (See Section 3.1 for more details on Issue 1.1.)

Issue 1.2 — Given its resources and activities, are FedNor objectives plausible?

Based on the benchmark study, FedNor's program design and delivery are consistent with those of the other federal regional development agencies. Also, experts believe that achieving program objectives, at least partially, is plausible. FedNor would need significantly more resources to more completely meet the identified needs of Northern Ontario. The level of objectives achievement for each program area is closely linked to the level of resources being utilized. (See Section 3.2 for more details on Issue 1.2.)

Findings – Progress / Success

Issue 2.1 – To what extent has FedNor achieved its stated objectives?

In general, the FedNor objectives are achieved to varying degrees of success. In summary, the objectives are achieved as shown in Table A below.

Table A: Summary of Objectives Achievement
Objective	Level of Achievement	As demonstrated by
Enhance the participation of financial partners in micro lending and high risk lending through its investment programs and services	Medium	132 loans resulting in $20 million in newly available commercial lending Perceived to be not as extensively used as needed
Improve CFDC capabilities through its investment programs and services	High	Investment Funds add significantly to CFDC operational capabilities
Help create new networks and provide more networking opportunities through its connectedness and community partnerships programs and services	High	More than three-quarters of clients surveyed indicate that this had already occurred on their project and that it was highly due to FedNor assistance
Increase the degree of integrated community-based decision-making and delivery	Medium	Almost two-thirds of clients surveyed indicated that this had occurred and that FedNor's contribution was fairly high (7.6 out of 10)
Enhance access to a more cost-effective telecommunications infrastructure as well as awareness, development, and use of ICT applications including e-commerce through its connectedness programs and services	High impact / small reach	The high need for improved telecommunications infrastructure The high level of success of specific projects, recognizing that FedNor's resources for such projects are limited
Enhance interaction between educational institutions and industry through its innovation capacity building programs and services?	Low	Minimum examples of such projects Low percentage of clients identifying this impact
Help companies implement new processes and bring new products and services to market faster through its innovation programs and services	Low	Low level of readiness and low degree of maturity for undertaking such projects Minimum examples of such projects Only approximately 10% of resources are allocated to these types of projects, yet these are resource intensive projects
Increase the participation of youth, women, Francophone, and Aboriginal entrepreneurs through its investment programs and services?	Varied	1/3 of projects are specifically targeted at youth and 1/4 at Aboriginal people Less than 10% each for women and Francophones although many are reached through projects that do not target them specifically
Increase the number / capabilities of exporting businesses in Ontario through its trade programs and services?	Low	Low level of readiness and low degree of maturity for undertaking such projects Minimal level of resources targeted at this objective

(See Section 4.1 for more details on Issue 2.1.)

Issue 2.2 — Is FedNor reaching its intended target groups?

FedNor's intended target groups are youth, women, Francophones, and Aboriginal people.

The overall profile of these projects is, in general, different than the overall profile of all FedNor projects, indicating that assistance may be strategically adjusted to better meet the needs of the specific target group. For example, Aboriginal people receive a higher proportion of funding for projects related to the connectedness priority. This seems appropriate given the relative telecommunications isolation of Aboriginal communities. (See Section 4.2 for more details on Issue 2.2.)

Issue 2.3 — Have the communications / coordination between partners and with external stakeholders been appropriate?

The evidence from the evaluation research shows that the communications between FedNor and its partners and external stakeholders is appropriate. An improvement in communications in the last four to five years was noted. In addition to being appropriate, strong communication between FedNor and its partners and stakeholders was seen as essential to program success. (See Section 4.3 for more details on Issue 2.3.)

Issue 2.4 — Has FedNor taken the necessary measures to meet its objectives (i.e., planning, training)?

Information from the staff interviews shows that the level of planning carried out by FedNor is appropriate. There is support for providing additional development and training for officers, particularly those with less experience. Staff who were interviewed observed that FedNor, as an organization, is committed and motivated. (See Section 4.4 for more details on Issue 2.4.)

Findings — Cost-Effectiveness

Issue 3.1 — Are there more cost-effective ways of delivering FedNor?

There is almost no evidence that FedNor programs and services are competing with those of private or non governmental organizations. However, as FedNor works with banks and other lending institutions to make high risk loans available to firms, it may appear to the casual observer that they are providing similar services.

FedNor is considered by most people interviewed as being unique within Northern Ontario. Its focus on Northern Ontario needs, wide variety of programs and services, funding assistance, the quality of management and staff, and client friendliness are all deemed to be relatively unique within the region, including in comparison to the Northern Ontario Heritage Fund Corporation. Outside the Northern Ontario region, FedNor is similar in mandate, program design and delivery strategy as the three other regional development agencies for the Atlantic (ACOA), Quebec (CED-Q) and the Western Provinces (WD). (See Section 5.1 for more details on Issue 3.1.)

Issue 3.2 — What changes can be made to improve FedNor's performance and its likelihood or level of success?

According to internal sources, one of the most critical challenges facing FedNor in the future will be the variance of annual funding. This year-to-year uncertainty over the budget levels has had an impact on staffing and is perceived to have a likely impact on priorities. In contrast, the majority of changes suggested by stakeholders and clients to improve FedNor's performance were process oriented (e.g., the proposal process, the approval of proposals and claims processing). These concerns are being addressed by a new risk management approach currently being implemented by FedNor and by a Service Improvement Initiative which is developing a comprehensive plan to enhance client service. (See Section 5.2 for more details on Issue 3.2.)

Issue 3.3 — Are the roles and responsibilities of the various parties clearly understood and agreed to?

For the most part, the roles and responsibilities of FedNor and the various parties appear to be understood. Much of the understanding is based on experience working with FedNor rather than explicitly communicated roles and responsibility. One area that seems to be less clear to clients and stakeholders is the role that FedNor officers can and should play in the development of proposals. (See Section 5.3 for more details on Issue 3.3.)

Issue 3.4 — Have the changes in FedNor terms and conditions been effective?

In 1999, FedNor added new program components to provide funding for capital community economic infrastructure, community recovery for sudden or severe economic downturns, telecommunications infrastructure, ICT applications including e-commerce, innovation capacity building and applied R&D. The stakeholders and experts generally noted that the changes made to FedNor's terms and conditions were appropriate, and to-date, fairly effective. The responses of the Northern Ontario CFDC representatives were more cautious. Most agreed that the changes were appropriate, but that it was difficult to assess their effectiveness at this early stage. (See Section 5.4 for more details on Issue 3.4.)

Issue 3.5 - Has FedNor defined and agreed to key performance targets and objectives, developed approaches to measure and monitor performance, and utilized performance based information effectively to make improvements and report on the extent to which it has been successful?

Information from the benchmark study, staff interviews and focus group discussions all showed that FedNor is at a relatively early stage in the development and implementation of a performance measurement strategy. There is considerable interest and support for improved capability, and additional resources have recently been allocated to the task. FedNor has made working towards developing increased capability in performance measurement and increased use of performance related information for management and reporting a priority. (See Section 5.5 for more details on Issue 3.5.)

Conclusions and Recommendations

Table B below provides the conclusions and recommendations resulting from this evaluation study.

Table B: Conclusions and Recommendations
Conclusions	Recommendations
Rationale
FedNor's programs and services are well aligned to meet the needs of communities, SMEs and entrepreneurs in Northern Ontario for assistance in economic and community development. In many areas, such as improvement to telecommunications infrastructure and facilitating access to capital, FedNor is one of several contributors, and works in partnership with other public and private sector organizations. As such, FedNor's role is to build capacity and clarify solutions so that other programs can contribute. FedNor's broad range of programs and activities is a strategic advantage in being able to respond appropriately to the varying needs of the many actors in Northern Ontario (i.e., "one size does not fit all").	Given the wide range of needs and capabilities of communities, SMEs and entrepreneurs in Northern Ontario, FedNor must continue to offer a wide range of services as well as the flexibility required to assure that all groups that can contribute to FedNor's mandate have the opportunity to participate. Nonetheless, FedNor should continue to monitor the needs of the communities in Northern Ontario to ensure that FedNor's programs and services continue to be relevant to these communities..
Based on FedNor's program design and delivery and current level of resources, at least partial achievement of the program objectives is plausible. However, the total needs of Northern Ontario cannot be met with the present level of resources. The extent of objectives achievement and the plausibility of programs related to each objective is related to the level of resources available.
Progress / Success
It is important for FedNor to have as one of its objectives improved access to capital for small business through financial institutions. However, the loan loss reserve fund was not used as fully as anticipated. It is not possible to confidently conclude on the underlying reasons behind the lower use based on the study results. Particularly through its support to the CFDC Investment Funds, but also through other initiatives, FedNor is contributing to improving CFDC capabilities. FedNor is helping create new networks and providing networking opportunities through both its connectedness and its community partnerships programs and services. FedNor has increased the degree of integrated community-based decision-making and delivery. However, FedNor recognizes that its role is that of facilitator rather than that of driver. FedNor's role in enhancing access to a more cost-effective telecommunications infrastructure, as well as awareness, development, and use of ICT applications including e-commerce is important. However, FedNor's contribution in this area is limited by its available resources. It must therefore seek to leverage funds from other sources. It is, nonetheless, critical that FedNor continue to be involved in providing assistance in these areas, particularly given Northern Ontario's extensive and diverse need for cost-effective telecommunications infrastructure. FedNor is enhancing interaction between educational institutions and industry; however, FedNor's contribution has not been major to date. FedNor has, however, developed strategies to increase its activities in this area. There are a number of projects where companies have taken advantage of the FedNor innovation programs and services to successfully develop and implement new and improved products and processes, and bring them to market faster. However, it is not clear what the demand is in this area, and to what extent FedNor is able to meet it. FedNor has, however, developed strategies to increase its activities in this area, including participation in the Government's overall Innovation Engagement Strategy. FedNor is increasing the participation of youth, women, Aboriginal people, and Francophones either through projects specifically targeted at those groups or through projects that reach those groups while not specifically targeting them. FedNor has not invested much of its resources in initiatives aimed at increasing the number or capabilities of exporting businesses in Northern Ontario. However, this may be due to the lack of readiness or awareness of many businesses. FedNor has, however, prepared a Trade Development Strategy which aims to increase its activities in this area.	FedNor should keep the extensive list of objectives it has identified to help it address its mandate. Without these objectives, FedNor will not be able to address all components of its mandate. In addition, FedNor should continue to prioritize its use of resources to ensure that it can make the greatest difference through the following: financial leveraging on projects by having a wide range of partners; projects that will have an impact on a large number of organizations or individuals in the communities (for example, telecommunications infrastructure); and projects that will have a significant impact on one of the FedNor target groups.
FedNor is reaching its target groups through strategic programming. Target groups are reached by receiving targeted funding in specific priorities or geographic locations. This responds well to the caution provided by various groups that FedNor should ensure that it is flexible to the needs of specific groups, that is, that "one size does not fit all".	FedNor should continue to target specific groups in the community where needs may be different and who therefore require special attention. FedNor should continue to do so through targeted funding that will best address the specific gaps of those groups (e.g., telecommunications infrastructure or bridging the digital divide for Aboriginal communities).
Communications between FedNor and its partners, external stakeholders and clients are essential and are carried out in an appropriate manner. For the most part, stakeholders and partners are satisfied with communications with FedNor.	FedNor should continue to improve the types, quality and quantity of communications with partners, external stakeholders and clients according to the needs of these groups.
Internal communication and strong leadership within FedNor has led to an organization that is committed and highly motivated. Attention has been paid to ensuring open sharing of ideas and project proposals between geographic areas. Based on all feedback received, the continued provision of development and training opportunities for newer officers is key to effective program delivery.	FedNor should continue to offer development and training opportunities for staff, particularly newer officers.
Cost-effectiveness
Based on input from clients, staff and experts, FedNor is providing relevant, largely unique services in a flexible, client-friendly manner. FedNor improves its cost-effectiveness by co-funding projects with other public sector partners as well as the clients. The wide range of programs and services provided by FedNor directly and through partnerships is seen as a strength.	No new recommendation required.
From an internal perspective, it is critical to address the lack of stable funding and the impacts of potential decrease in funding levels. Applications and claims processing are two key areas that FedNor also needs to be improve. However, many of these concerns have been addressed, in principle, by the new Risk Management Strategy adopted by FedNor in late 2001 as well as by the Service Improvement Initiative. It is too early to assess the effectiveness of these strategies in addressing the concerns raised.	Recommendations for these conclusions are provided through the recommendations related to performance measurement below.
FedNor staff see the facilitating role that they play in communities and with proponents in the development of proposals as key. For the most part, this role is understood by Northern Ontario CFDCs and some stakeholders. However, there appears to be some need to better document and clarify FedNor's role with proponents and potential proponents.	FedNor should continue to play a facilitating role and hence help clients improve their level of readiness and their degree of maturity. Where required, the roles and responsibilities of FedNor staff and partners should be clearly documented and communicated to all involved. This could be done through the FedNor website supplemented with various communications activities.
The 1999 changes to the terms and conditions, where FedNor added new program components to provide funding for capital community economic infrastructure, community recovery for sudden or severe economic downturns, telecommunications infrastructure, ICT applications including e-commerce, innovation capacity building and applied R&D, are seen as appropriate but it is too early in their implementation process to assess their effectiveness.	FedNor should seek continued funding under these terms and conditions, recognizing that on-going monitoring via the performance measurement strategy recommended below is required to ensure that information on the effectiveness of the implementation of the terms and conditions is available to FedNor management.
FedNor is at a relatively early stage in the development of performance indicators and targets and, more generally, of a performance measurement, management and reporting culture. FedNor is exploring the use of performance-related information and, as capability and available information increase, will be moving towards a more general implementation of performance information for program management. There are four key initiatives / systems that FedNor can currently use for performance measurement purposes. The first is CMIS, which can provide good information on use of resources and client reach. This evaluation did not involve a thorough review of CMIS and how it is used for performance measurement and management purposes. However, use of the systems revealed that some important data elements are not consistently captured. As FedNor starts using the system more consistently for performance management, this could become an issue. The second initiative, the Project Outcome and Evaluation Score Sheet, is also internal. It captures good results or outcomes information, but from the limited perspective of the program officer managing the project. This information is important but needs to be supplemented with direct feedback on outcomes from the clients. A more recent initiative, which is in its very early stages, is the Service Improvement Initiative. This provides excellent performance information from the perspective of clients, but is generally limited to FedNor processes rather than outcomes. This initiative will definitely be helpful to FedNor for some of its performance information. The fourth type of initiative is the use of special studies. This includes formative or summative evaluations as well as other special studies such as the Northern Ontario Access to Capital Study, the Telecommunications Study, and many others. FedNor has, in many cases, made good use of the results of such studies for improving its existing programs and services as well as for planning for new initiatives. There are two key gaps in FedNor's performance management system. First, the existing tools noted above are not used consistently. Second, there in no consistent on-going information collected from clients on the outcomes of projects.	FedNor must continue to develop its performance measurement system and to maintain the current focus on its development and on its evolution. This system will need to draw upon information from a wide range of sources to ensure that FedNor has the appropriate performance information. The performance information must span the entire range of FedNor's performance from inputs to outputs, reach and outcomes (or impacts). In order to have reliable information in all these areas, FedNor must put in place a performance measurement system that includes internal information as well as information obtained directly from the clients. FedNor should therefore continue to use CMIS for information on inputs, outputs and reach. However, in order to ensure that this information is reliable, it is important to ensure that all performance-related fields are consistently filled out. FedNor should also continue to have program officers complete the Project Outcome and Evaluation Score Sheet. This must continue to be done consistently after project completion. FedNor should also continue its Service Improvement Initiative. This provides important information from clients on various aspects of FedNor activities and services. A mechanism to obtain short-term outcomes information from clients must be put in place. Most importantly, FedNor must develop an organizational-wide culture that ensures that performance information is not only gathered on an on-going basis, but also analyzed and used for decision-making purposes throughout FedNor.

Adobe Acrobat Version (PDF - 337KB - 128 pages)

Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.