This document is the Final Report of the formative evaluation of the federal government's Softwood Industry and Community Economic Adjustment Initiative (SICEAI). It also is intended to help Industry Canada (IC), the Regional Development Agencies (RDAs), and FedNor to help design future adjustment initiatives, especially those directed towards the forest industry and rural communities. This evaluation report is formative in the sense that — although the SICEAI program has already ended — the program existed only for a short time and many of the projects it supported had barely been finished at the time this study was conducted. The findings must also be taken in the context that program officials in federal, regional, and economic development agencies had a relatively short period to "ramp up" activities from essentially nothing to full operations. Thus the program was addressing a very significant adjustment situation (see below) in the affected communities with little lead time and modest resources.

The U.S. Softwood Tariffs

On March 21, 2002, the U.S. Department of Commerce first announced subsidy and dumping tariffs with respect to Canadian softwood lumber. By November 4, 2002, the number of duty-related, direct job losses across in sawmills Canada totalled 3,940, and by September 31, 2004 were estimated at about 5,800. It was expected that production could decline by 8-10% (as high-cost producers dropped out) and that Canadian exports would decline by up to 20%, or up to $1.8 billion.

The SICEAI Program

To mitigate impacts of the tariffs, the government initiated a package of measures, of which the key response was the $110 million SICEAI program, announced October 8, 2002. The program's objective was to create long-term, sustainable economic benefits in regions and communities affected by the tariffs. (Note that the program was not intended to directly provide assistance to laid-off or displaced workers, or to directly replace these forestry-related jobs.) SICEAI was originally intended to expire at the end of the 2003-04 fiscal year. Due to delays in initiating the program, no money was released to proponents in 2002-03 and very little in 2003-04, requiring that the program be extended to the end of the 2004/05 fiscal year.

The program provided both repayable and non-repayable contributions, depending on the region and nature of the project. Two types of projects were eligible (1) Diversification, productivity improvement and new economic opportunities, which would create tangible and sustainable long-term jobs and (2) Community capacity-building, including developing community economic action plans, strengthening local developmental and leadership capacity, and exploiting opportunities in other local industries for sustainable economic development. The program was directed towards hard-hit communities.

SICEAI was funded federally by Industry Canada (IC), and delivered regionally by Regional Development Agencies (RDAs): Western Economic Diversification Canada (WD), Canada Economic Development for Quebec Regions (CED) and the Federal Economic Development Initiative for Northern Ontario (FedNor), which is under Industry Canada ADM Operations, as well as other regional and private sector partners. Delivery in B.C. for non-repayable contributions was through a third-party delivery model WD delivered non-repayables through an agreement with the Community Futures Development Association of BC (CFDABC) and its individual Community Futures Development Corporations (CFDCs).

Evaluation Issues

The study issues were adapted from the SICEAI Results-Based Management and Accountability Framework (RMAF), but are not identical to the RMAF questions. Main issue topics included: (1) Program relevance (2) Design and Delivery (3) Preliminary Results and Success (4) Cost-effectiveness and (5) Lessons Learned.

Methodologies

The main methodologies included: (1) Review of SICEAI program documents (e.g., program guidelines, eligibility and review criteria, performance data, statistical information) (2) Review of project files for case study projects (3) Interviews with program officials and staff in both "headquarters" and in RDAs in the three regions (4) A survey of the 33 participating CFDCs in BC (5) A survey of as many program clients as could be contacted within study resources (n = 252) and (6) Fifteen mini-case studies of projects that were selected to best illustrate lessons learned and/or best practices.

Finding on Program Relevance

Appropriateness. As part of a larger government initiative, the SICEAI program was very appropriate in that it provided the resources required in order to respond quickly to the layoffs in the forestry industry, and to start planning for community adjustment measures. Although certain factors (e.g., which communities were affected) and specific project funding requirements changed over the lifetime of the program, the overall needs addressed by the program did not change. At the time the program was announced, there were no reasonable alternatives to SICEAI. The community-based delivery model was appropriate, because the RDAs and CFDCs could best respond to the needs of their communities, while allowing Industry Canada to ensure consistency between regions and to remain accountable for the program overall. At least 72% of the projects would not have proceeded at all, or only after a significant delay, without the program, meaning SICEAI had good incrementality (also see below).

Program Design and Delivery. The roles, responsibilities and respective spending authorities between IC and the RDAs under SICEAI were unclear to all concerned early in the program's lifetime, and were one cause of the delay in implementing the program. However, these difficulties were eventually resolved.

Project review and selection. A review of the Terms and Conditions indicates that the program eligibility criteria were appropriate and, in general the file review and other data indicate the criteria were appropriately applied to individual projects. All three regions were very flexible in terms of the nature of projects supported, so long as they clearly fit the intent of the program. However, the program's detailed objectives (e.g., community vs. industry focus), rules, and operating procedures were unclear to all concerned at the beginning of the program, and communications from Industry Canada on these topics were often confusing and conflicting.

Targeting. There were three types of marketing and outreach targets set (1) Towards communities — those that had suffered significant impacts because of the tariffs (2) On potential clients and projects — especially forestry companies that were SMEs, First Nations communities, and/or projects were value-added and (3) Strategic targets — for example for outreach to specific regions, or to attempt to generate interest for projects in certain sectors (e.g., tourism). Targeting on First Nations communities or SMEs in value-added wood industries was especially done in BC, based on internal WD strategies.

However, there were no rigidly enforced targets set on the allocation of contributions — project selection was done in an entirely "bottom-up" manner based on the eligibility and quality of the project application. Program targets overall (e.g., jobs by sector, or by region) were not set by IC or by the regions, although an attempt was made in BC to market and encourage client groups to propose projects which fit provincial sector priorities. Targets for job creation and/or maintenance, timelines, deliverables, and matching funding were set for each individual project. These were set in conjunction with proponents, and were generally seen as reasonable. However, the very short timeframe available for project implementation and completion caused problems for many clients.

Client satisfaction. Clients were generally satisfied or very satisfied with all aspects of service delivery, especially from regional offices and their community-based delivery networks (including CFDCs), and very few complaints were noted. Many clients noted that the assistance of the local agencies was essential in allowing them to succeed.

Reach. The program made every effort to target communities affected by the tariffs, although in practice not all were served. Small, remote, and First Nations communities were especially targeted for outreach, as were communities known not to have much economic development capacity. Some aspects of eligibility criteria reduced the program's ability to serve needy communities; e.g., if the timing of layoffs didn't quite match the program's application period, or if communities did not suffer "direct" sawmill job losses, or if local governments could not secure the 10% private sector funding required.

Findings on Preliminary Results and Success.

Program success. The program has had a modest success on mitigating the direct impact on displaced forestry sector workers as a result of the tariffs, although not surprisingly it has not been able to dramatically diminish these effects (nor was the program intended to directly assist these displaced workers or replace these jobs). It has been rather more successful in helping communities diversify their economies, in creating new jobs, and increasing community capacity for responding to future change and the need for adjustment and diversification.

Project success. About 65% of projects are currently complete and operational, with another 11% complete but not yet operational, and 14% still in the construction phase but expected to be completed. Only 1% of the projects are not expected to be completed, and another 2% were never started (the remaining 12% of respondents to the client survey either answered "not applicable" or did not answer at all). Both RDAs and clients noted that the projects have created and/or maintained jobs, and that the projects have often helped communities think "outside the forestry box" in diversifying their economies.

Community Impacts. About 85% of projects were successful in creating jobs or maintaining jobs to date. Just over half the projects did both. A conservative extrapolation of job creation and maintenance is shown below.

	No. jobs expected	No. jobs actual to date	No. jobs estimated in 3 years' time
Direct job creation	3,100	2,600	3,900
Direct job maintenance	2,200	4,000	2,800
Indirect job creation	1,100	850	1,400
Indirect job maintenance	1,100	1,500	750
Total all jobs	7,600	9,000	8,900
(Totals may not add due to rounding errors.)

It was noted that the jobs created through SICEAI are not direct replacements (either in terms of the individuals affected, or in the wages replaced) for those lost in the forestry industry, which tend to be higher paying jobs. (The program was not, of course, intended to directly replace these jobs.)

A conservative estimate is that the program to date has created about $147 million in annual gross revenues, and maintained about $436 million. These figures are expected to rise to about $646 million and $490 million, respectively, in three years' time.

Capacity building projects. Capacity building projects were not common in any of the three regions. It is too soon to judge the impacts of those that were funded.

Increasing Investments. The program was successful in obtaining about $247 million in external (non-program) funding, representing a leverage of about 225% of the $110 million federal funding. The "gross (i.e., total) leveraging" of the average project in our client survey was about $516k, or roughly 230% of the program's contribution to the project. The "net leveraging" for the average project in our client survey was about $208k, or about 157% of the program's contribution. Net leveraging is defined as the additional investment that occurred because of the program's existence; i.e., non-program funding that would not have been committed to these projects in the absence of SICEAI. That the net leveraging figure is relatively large helps confirm that SICEAI was indeed incremental — the program support acted as a critical seed to help project proponents find funding and implement the project.

Unexpected Impacts. Virtually all of the unexpected impacts for clients were positive, and included additional jobs being created, new business opportunities being identified, considerable media attention, revitalization of neglected business areas, and a generally greater sense of community participation and collaboration. Negative impacts were mainly related to the burden of preparing applications and dealing with delays. For RDAs, similar positive impacts were noted. Some additional negative impacts were related to the initial confusion regarding program goals and procedures, and (in BC) the workload associated with the very high number of applications for funding.

Findings on Cost-effectiveness

Cost-effectiveness. First, note that this study was not intended to definitively answer whether the program was cost-effective. Instead, the issue is the standard program evaluation question of whether there are improvements in the program design or delivery that could be made that would increase the cost-effectiveness of this program. A full accounting of cost-effectiveness is one issue that might be addressed in a summative evaluation, which would not be appropriate in this situation.

The general answer is that the program appears to be cost-effective. There is little "hard" data available on this point (although it is always a hard question to answer). Most respondents in the RDAs believe the program was cost-effective because the program could "piggy back" on existing local and regional knowledge and programs. The cost per job to date (i.e., all jobs, either directly or indirectly created, and including both job creation and maintenance) was roughly $12,000. If one only considers direct job creation and maintenance, then the cost per job to date is about $16,000. Both figures will remain relatively constant over the next three years — although the number of newly-created jobs are expected to increase, the number of maintained jobs is expected to diminish. The major mechanism suggested that might improve program efficiency was elimination of the Other Government Department (OGD) suspense accounts mechanism.

Due diligence. The review of Terms and Conditions and the review of program files conducted for the case studies both indicate that extensive due diligence was conducted within this program. External review of the application procedure in BC supports this conclusion.

Findings on Factors that Influenced Success

Key success factors include the local knowledge of CED, FedNor, and the CFDCs of their communities' needs, in some cases supplemented by additional staff hired specifically to help with program outreach and client support. This local expertise also helped to extend the reach of the program to many smaller and more remote communities, develop high quality proposals, and help clients with issues arising (e.g., dealing with Environmental Assessments). Negative factors included the initial delays in implementing the program, and the lack of appropriate promotional materials.

Findings on Lessons Learned for Future Similar Programs

Lessons learned. A longer time frame is needed for adjustment programs to be fully effective. This will allow better outreach and marketing, better quality proposals to be developed, more time for initiation and completion of complex projects, and better retention of community expertise.

Having program goals, objectives, authorities, and procedures worked out between IC and other program partners in advance of the program's public announcement will save considerable time and frustration for all parties.

Terms and Conditions must be flexible enough to respond to individual community circumstances, but there must be an oversight mechanism to ensure the policies regarding eligibility and support levels are applied reasonably consistently and equitably across similar clients in all regions and communities across the country.

A simpler program structure than the Other Government Department (OGD) suspense accounts mechanism used by SICEAI between IC and RDAs would be beneficial.

A mechanism for identifying community and industry needs in advance of crisis situations is preferable, and would allow better identification of exactly what negative impacts need to be addressed by government programs.

Supporting larger projects offers the opportunity for significant collaboration among stakeholders (including those across multiple communities), as well as significant and broadly-spread benefits. There may additionally be long-term "knock-on" benefits not directly associated with the project. Of course, the number of smaller projects that could be supported may diminish, and a very strong business case would have to be made for a large project.

Sharing of lessons learned. Best practices were frequently shared among the RDA offices. Best practices were also shared with at least a third of clients, but it is unclear to what extent these reflect best practices for SICEAI specifically, as opposed to more "general" lessons learned through other economic development initiatives. (Of course, both are valuable.)

Recommendations for Future Similar Programs

The program scope should be widened to include all communities, which are deemed to be heavily dependent on the forestry industry and are being negatively affected by restructuring in the forest sector (i.e., not just those linked directly to the sawmilling industry, as was SICEAI).

A decentralized community-driven model using local delivery networks (including CFDCs) is appropriate for government initiatives intended to benefit broadly-dispersed clients who will use the funding in highly-divergent projects. This is especially true when these clients require a substantial community outreach and assistance program to help all of them develop appropriate proposals, meet complex eligibility criteria (e.g., environmental assessments), and to ensure small, remote, and First Nations communities benefit.

Program Terms and Conditions should be kept flexible. However, an effort should be made to standardize the degree of (necessary) flexibility that is exercised by the regions in interpreting program guidelines.

Set a longer program duration from the beginning. Note that if a longer-term program is employed, the design criteria should consider that it may require additional financial and (perhaps) human resources.

IC needs to be able to anticipate problems within the industry, and begin a process of adjustment sooner, rather than having to wait for a crisis and large numbers of layoffs to be the trigger to initiate a program.

Industry Canada should consult extensively with FedNor and the Regional Development Agencies at the early design stage of such programs. In particular, the details of goals and objectives, outreach and communications, project and applicant eligibility, co-funding, the application and review process (including selection criteria), delivery roles and accountabilities, signing authorities, and implementation guidelines should be carefully thought out ahead of time.

Similar early consultation with the provinces, and increased efforts to engage the provinces by demonstrating the joint aspects of the issues, may result in greater federal/provincial collaboration and an increased ability to obtain co-funding.

While strategic targeting is entirely reasonable with respect to marketing and outreach (e.g., to encourage and/or help specific types of needy clients to apply, or encourage particular types of projects), project selection and funding allocation should be made "bottom-up", entirely on the basis of fit to eligibility criteria and quality of the application.

A simpler model than the OGD (e.g., one that would assign more responsibility and accountability within the RDAs) would offer reduced effort by clients and government alike, more transparency, and easier communications.

A simple, long-term performance monitoring system should be instituted which follows-up with 100% of clients on an annual basis for at least five years following completion of each project to determine actual job creation and maintenance and the degree to which diversification of the local economic base has been achieved and sustained. While an added cost, collecting such long-term data would greatly help in designing future initiatives.

A dedicated cost-effectiveness study of the economic benefits and costs resulting from SICEAI and other similar initiatives should be conducted. The data collected in the monitoring program discussed in recommendation #10 should be of a type to allow such a study to be carried out. Such a study would be of great interest to Industry Canada, RDAs, FedNor, the CFDAs, and other economic development organizations for any individual program. It would also allow far better comparison of the benefits and costs of different program delivery models, so long as the exact same methodologies were used to compare different programs.

Final Report (PDF - 328 KB - 67 pages)

Management Response (PDF - 17 KB - 3 pages)

Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.