Evaluation of Technology Partnerships Canada Program

Audit and Evaluation Branch

Evaluation Report

October 14, 2003
Executive Summary

Study Context (Section 1)

This Technology Partnerships Canada (TPC) summary report is based on the issues and methodologies laid out in the Technology Partnerships Canada Evaluation and Accountability Framework of July 2001. The study covered specific research questions related to:

• program relevance and rationale;
• program governance and management;
• program design and delivery;
• progress in delivering the program and achieving success; and,
• lessons learned.

The evaluation methodologies include a review of program files and documents, an analysis of TPC and IRAP-TPC management information system electronic files, interviews with TPC and IRAP managers and staff, other co-delivery partners and clients, telephone surveys of a sample of 90 of 133 TPC clients and 120 of 323 IRAP-TPC clients with funded projects, and six case studies (four TPC and two IRAP-TPC).

These multiple lines of evidence were used to develop findings and conclusions on the various research questions.

This study was designed as a formative evaluation — focussed on assessing the governance, management, design, delivery and early results of TPC. For this reason the evaluation offers only limited information on overall economic impacts or the industrial policy relevance of the program.

Summary Findings

Relevance (Section 2)

In an era of Federal Government deficit reduction, TPC has been asked to cover a myriad of industrial policy objectives. The program was given a wide reaching mandate to create jobs and foster innovation in three very different sectors. At the same time, TPC’s role in the innovation assistance process was limited by the requirements of investment payback, as well as being subjected to intense international scrutiny. These constraints have led to several logical inconsistencies in its set-up, as compared to previous and other existing contribution programs:

• The program focuses on repayment of investments, yet it must fund (increasingly since the WTO decision of 1998) high risk innovation – often in emerging areas of technology.



• TPC must operate in a consistent, transparent and ‘fiscally responsible’ manner, leading to a lengthy multi-step assistance process, yet it is by definition mandated to serve areas of high technology risk and market uncertainty — areas which require speed and flexibility.



• The program was essentially developed in many respects as a ‘son of DIPP’. The Defence Industries Productivity Program (DIPP) was designed to serve the needs of the mature, cold-war aerospace and defence industry of the 1970s and 1980s. This approach is likely not consistent with a program being asked to assist sectors (even that of the modern aerospace industry) facing a completely different market situation.

In conclusion, TPC has, in theory and rhetoric, been established to serve several publicly stated industrial innovation goals and sectors, while subjecting itself to the constraints of economic development assistance in the modern era. In reality, it would appear that the ‘one-size fits all’ approach that has been taken up to now may not be up to the task.

Governance (Section 3)

While TPC is identified as a Separate Operating Agency, it has, at present few of the flexibilities, such as increased spending authority, often associated with SOA status. Evidence suggests that TPC’s partnerships with NRC’s IRAP and NRCan’s CCAF and TEAM have improved program relevance and success. Also, the TPC Advisory Board, originally identified as an important source of private sector advice for program management, has not met in several years.

Design and Delivery (Section 4)

The TPC assistance process is complex in terms of both its number of steps and the number of groups involved. Some of this complexity is likely unavoidable due to the numerous constraints placed on modern industrial assistance of this magnitude. However, when combined with a low investment in program administration, the effect has been to unfairly handicap the speed, flexibility and ‘learning ability’ of this program.

Success (Section 5)

Within the limits of the methods used for this formative evaluation, it appears that TPC and IRAP-TPC achieved reasonable project incrementality, significant technical and early job creation impacts. Commercial, therefore repayment, results have been limited to date. This can partially be attributed to the long time horizons required for many projects to realize commercial results and partially due to economic circumstances (i.e., a downturn in the markets for the sectors involved).

Case studies and survey responses suggested that there were ‘public good’ types of outcomes which were significant for a number of TPC investments. Unfortunately, a narrow public communications focus on repayment levels, combined with a lack of program investment in intermediate outcomes measurement, have worked against TPC in terms of telling its full performance story.

While TPC funding is important to every firm supported, it appears to play a critical role in support to the aerospace sector, due to the lack of other support programs. On average, since 1996, TPC has supported a large fraction of the major R&D projects and TPC contributions have been about 15 - 20% of the annual R&D expenditures in the aerospace and defence sector. In the other sectors, there are calls for an increased share of TPC funding, as the demand for TPC support is much greater than can be met by available funds. However, there are other government programs and private sector sources to provide funding support, and TPC plays a much less dominant role to the enabling and environmental sectors.

In terms of operational objectives, TPC has recently exceeded the targets of 3% ‘overhead costs’ and 50 Full Time Equivalent (FTE) program staff levels, that were found to be unrealistic for a program of this magnitude and complexity.

Lessons Learned (Section 6)

The use of a single program, essentially based on an assistance model from a previous era, to assist three critically different sectors in high risk innovation appears to have hampered the effectiveness of TPC.

Recommendations

As a formative or mid-course evaluation, the mandate of the study is to address program design and delivery issues. The study is not aimed at making judgments on the fundamental public policy instrument that underpins the TPC program, or to make summative judgements on the success or contribution of this program to the country’s innovation agenda. Nevertheless, there is no evidence to suggest that the government should not remain committed to the “public good” ideals intended with this partnership investment.

It is clear that the objectives at inception of this program were significantly changed to adhere to the “pre-commercialized” dictates of the WTO decision of 1998 and to address the emerging economy of enabling and environmental technologies outside of the aerospace and defence sector. It is also clear that the program design and delivery parameters were never re-aligned to the new realities and objectives of an altered TPC program. These include program criteria, risk assessment, performance and outcome expectations, repayment plans and approval processes.

In order to fully exploit the relevance, effectiveness, transparency and ultimately the success of TPC, a mid-stream course correction is recommended. Accordingly, Industry Canada should undertake a review of TPC to ensure its suitability to the needs of the modern aerospace, enabling and environmental sectors. Such a review should encompass the domains of policy, governance and delivery processes. It is not the recommendation of this study that a “moratorium” be placed on this program but rather that Industry Canada map out a process to re-align this program in accordance with the following recommendations:

Policy

1. In order to improve the relevance and effectiveness of Canada's innovation assistance, Industry Canada should undertake a policy review of TPC with regard to its relevance and suitability to the needs of the modern aerospace, enabling technologies and environmental technologies sectors. Options for consideration may include:

• the establishment of clearly separate program elements under a TPC umbrella;
• the development of completely separate programs; or,
• other structures which will preserve this important element of innovation assistance.

Irrespective of the conclusions of the policy review proposed above, the following recommendations are made with regard to the governance and design and delivery of the program.

Governance

2. TPC should determine what structural flexibilities would assist in delivering the program in a more effective and efficient manner, and seek approval for those changes. Options for consideration may include:

• increasing TPC' s independence from Industry Canada; and / or,
• seeking new opportunities for further delivery and co-funding partnerships.

3. TPC should reinstitute the TPC Advisory Board as soon as possible and involve the Advisory Board in the strategic oversight and management of the program.

Delivery Processes

4. TPC should review and revise the project selection and approval process, with a view to significantly reducing the length of time for decision-making and making funds available. Options for consideration may include:

• negotiating higher levels of spending authority; and /or,
• developing different levels of due diligence and approval processes for differing levels of funding and risk.

5. In order to reduce or eliminate the high level of funds carried forward, TPC should continue to develop more aggressive cash management practices. Options for consideration may include:

• a strategy of over commitment on the number of projects and resources being considered for funding based on experience; and /or,
• reconsidering the present practice of committing funds at the project selection stage, which is a major factor in under-utilization of available funding.

6. TPC should develop a more appropriate performance measurement and reporting approach, that captures operational performance information, as well as a broader range of early, intermediate and longer term technological, economic and societal benefits of projects.

7. Following from the above, TPC should communicate a more complete view of this innovation focused program and its technical and commercial outcomes to stakeholders and the public. This would include the high level of technical success being achieved by projects, as well as the lower level of commercial success achieved to date. TPC should communicate the high risk nature of projects, the fact that not all will be successful and that overall repayments will not likely meet the initial forecasts identified in proposals.

8. In order to achieve the process improvements noted above, TPC should increase the current levels of funding and staffing for policy, planning and program management to a more appropriate level, in line with requirements. The new levels should be determined by a needs analysis and comparison to other related programs.

---

Summary Report - Adobe Acrobat Version (PDF - 248KB - 30 pages)
Management Response - Adobe Acrobat Version (PDF - 123KB - 4 pages)
Management Response - Adobe Acrobat Version, January 2005 (PDF - 103KB - 5 pages)
Detailed Findings Report - Adobe Acrobat Version (PDF - 1,226KB - 124 pages)

Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.